Does sisl fork on import? Warning with mpi4py

[jonaslb]

So I occasionally write scripts that are simple to parallelize with mpi4py.
What I have discovered is that I occasionally get MPI warnings when importing sisl like so:

from mpi4py import MPI
import sisl

Which gives the following MPI warning:

--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[22936,1],0] (PID 24519)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------

However, the following two versions don't result in the warning:

import mpi4py
import sisl

import sisl
from mpi4py import MPI

Also importing other libraries doesn't seem to trigger the warning, only sisl.

I can't really figure out why it occurs. It doesn't stop the script from executing, but I still think it would be nice if it could be avoided regardless of import order.

[zerothi]

Hmm. Good question, I have no clue. I don't do any subprocess calls on import.

I'll check around, but I am not sure whether this is on sisl side or some weird combination of imports... :(
If you can gdb or backtrace this it would be really helpful!

[zerothi]

See here h5py/h5py#1079 not only sisl.

[jonaslb]

I found that it was during the import of tqdm the warning was raised. It was already reported in tqdm/tqdm#691 and also discussed in dials/dials#720, where it was mentioned that the possibly "guilty" code was rewritten in a newer version of tqdm. However I checked with tqdm's git master and the problem is the same there (warning triggered with from mpi4py import MPI; import tqdm).

[jonaslb]

I tracked down the issue in tqdm to a platform.system() call. They might be able to replace it with sys.platform instead, but the OP of that issue also mentions that the issue doesn't arise with OpenMPI 4.0, so maybe once dtu hpc updates it won't be relevant anymore. In fact I guess I'll try the new modules now :)

[zerothi]

Actually, the latest dcc-setup @ hpc already uses OpenMPI 4, so if you just do source /dtu/sw... ; module load ... then you should be using 4?

[jonaslb]

I don't just do that, I wanted to continue with the 2019 setup until i found the time myself to update -- I have

source /dtu/sw/dcc/dcc-sw.bash
module purge  # the script above also loads dcc-setup, do it manually instead
module load 2019-jun-dcc-setup
module load .....

instead. Which has OpenMPI 3.
I'll just update I suppose :)

[zerothi]

Ok :)
This is perfectly fine :)

[jonaslb]

I think I will close my issue here, both tqdm and h5py have solutions in their pipelines and it's not relevant with OpenMPI 4 anyway.