Assise with MPI

[yehonatan123f]

I'm trying to run a BTIO benckmark (link) with Assise over Intel Optane DCPMM to evaluate the performances of Assise and Optane. I have configured Assise only on one single node, and also have run the benchmark on this node, using 4 MPI ranks (processes).
When running, I get this following error from each MPI rank:
mca_fbtl_posix_pwritev: error in writev:Bad file descriptor.
It seems that Assise does not know to intercept MPI read/write calls.
Assuming Assise is under a continuous development, is Assise expected to support MPI I/Os in the future?
Thank you.

[wreda]

To recap our earlier discussion: The benchmark appears to be calling pwrite(), which, as you noted, wasn't supported by Assise. I've now added support to this syscall. Please pull and re-compile LibFS/KernFS.

I haven't used Assise with MPI workloads, but hopefully this patch will help get you there.

[zvikfir]

Hi @wreda,

We have tried to run the benchmark again, but the problem still persists.

This time we ran the benchmark with strace to produce a list of the system calls used in hope to find a system call that is not yet supported in Assise.

We have come up with the following list, containing system calls that the benchmark uses, but do not exist in Assise:

arch_prctl
bind
brk
chdir
clone
connect
epoll_create
epoll_ctl
eventfd2
execve
exit_group
faccessat
futex
getcwd
getegid
geteuid
getgid
getpgrp
getrlimit
getsockname
getuid
ioctl
listen
mknod
mprotect
newfstatat
pipe
poll
readlink
readlinkat
recvfrom
recvmsg
rt_sigaction
rt_sigprocmask
rt_sigreturn
sched_getaffinity
sched_setaffinity
sendto
set_robust_list
set_tid_address
setsockopt
shmat
shmctl
shmdt
shmget
shutdown
socket
socketpair
statfs
uname
wait4

From this list, several system calls came up (there could be more) that are related to file I/O: statfs, newfstatat (similar system calls like lstat, fstat do exist), and perhaps chdir and getcwd.

From the man page of newfstatat:

... The underlying system call employed by the glibc fstatat() wrapper
function is actually called fstatat64() or, on some architectures,
newfstatat(). ... "

These system calls implementation do not exist as well.

Do you think that this could be the reason for the benchmark not to work properly?

[wreda]

Getting to this now.

The use of unsupported file system calls would indeed explain why your benchmark is not working. statfs in particular will not work with Assise since we don't have an implementation of mounts. It appears your benchmark is also using execve. This is typically done after a call to fork, which is also not supported.

Adding support for these calls is currently not in our roadmap. However, you might find it possible to tweak your benchmark to avoid using these particular syscalls with Assise. Alternatively, you could try to add support for some of them yourself. For instance, it might be possible to quickly implement newfstatat by simply re-using our existing implementations of fstat (which can be found here).

To use Assise with fork/execve, one workaround is to disable automated initialization of LibFS (which does not work properly with fork). This can be done by setting the environment variable MLFS_DISABLE_INIT to 1 when running your benchmark. You will then need to call init_fs/shutdown_fs manually for each of your benchmark's processes (these functions are declared here). We applied this same workaround to get Assise working with Filebench.

If you manage to add proper support to any of the aforementioned syscalls, I'd be more than happy to integrate it.

Hope this helps!

[zvikfir]

I followed your suggestion and edited the source code of our program, so that for each process a call to init_fs() and shutdown_fs() will be performed. It seems to work properly in initiating Assise for each process.

I am now experiencing an inconsistent behavior from Assise when performing basic file I/O operations such as read and write.
For instance, echo "some test" > /mlfs/file doesn't seem to produce any output at all (usually the logs by LibFS are shown, but not when running this specific command).

When running our benchmark, I got the following error:

incorrect fd -2: file /mlfs/btio.mpi
mca_fbtl_posix_pwritev: error in writev:Bad file descriptor

I have tried to first create the file because I though it might help, but I am also getting an error when trying to touch a file:

touch: cannot touch '/mlfs/btio.mpi': Bad file descriptor

I have also tried to run 5 python processes in parallel, with each one writing to a different file in /mlfs. At first, only 3 files were created (for processes 2-4). Then, when I tried to run the script again, all of them were created normally.

Have you experienced this type of inconsistent behavior by Assise in your work?
Is the cause of this related to Assise or perhaps to something that is wrong with our setup?

Thanks,
Kfir

[wreda]

I am now experiencing an inconsistent behavior from Assise when performing basic file I/O operations such as read and write.
For instance, echo "some test" > /mlfs/file doesn't seem to produce any output at all

This is likely because LibFS hasn't been preloaded. You can try this command instead (with MLFS_DISABLE_INIT set to 0):
echo "foo" | sudo LD_PRELOAD=../build/libmlfs.so tee -a /mlfs/file

I have tried to first create the file because I though it might help, but I am also getting an error when trying to touch a file:
touch: cannot touch '/mlfs/btio.mpi': Bad file descriptor

The bad file descriptor error is likely due to the touch command using an unsupported syscall to force a certain file descriptor (namely dup2). This could also be why the benchmark fails. I'd recommend creating these files using your own custom script.

I have also tried to run 5 python processes in parallel, with each one writing to a different file in /mlfs. At first, only 3 files were created (for processes 2-4). Then, when I tried to run the script again, all of them were created normally.

If multiple processes are creating files in the same directory, you will need to enable leases in order to observe consistent behavior. This can be done by enabling the flag -USE_LEASE in the LibFS/KernFS makefiles.