New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intel MPI (versions in 2019a and 2019b) segfaults on CentOS 8 #11762
Comments
@hajgato Care to pitch in here? Have we looked at possible workarounds? |
As far as I remember those IMPI libs clashes with glibc. So not many things to do. If it is the known |
In our testing of |
I have found that works on RHEL8.2 and AMD Rome with impi/2018.4.274-iccifort-2019.1.144-GCC-8.2.0-2.31.1 (its not my job to judge things...) just for safety, I copy the relevant stuff here
Compile this file using the following commands:
And apply the generated library at runtime using the following:
|
So I guess we should update the I'm wondering how to detect that this dirty hack is needed though... Just using It would be nice if they provided a bit more context, but I guess we should be happy that there's a workaround. :) |
Should we apply the dirty hack if glibc > some version and impi version < 2019. The linked document says 'Intel® MPI Library 2018 and earlier'. We should be able to determine a suitable glibc version to make the split on - somewhere between what is in CentOS 7 and 8. Do we know if this is seen in Ubuntu 18.04, as that would allow us to further narrow this down. |
I strongly suspect that the same issue is with Ubuntu 18.04. See: https://community.intel.com/t5/Intel-oneAPI-HPC-Toolkit/Intel-MPI-segmentation-fault-bug/td-p/1154073 |
Anyway, I would be more happy if we make a mpirun/mpiexec whatsoever wrapper that preloads the strtok_proxy. (And the reason that I do not like LD_PRELOAD, and if we have to use it, maybe we have to minimize when it is used) |
We briefly discussed this during the EB conf call today. There's probably several options here, and Why create wrappers for Maybe we should reach out to Intel support to get more info on this, and make it clear we're not happy with the It was also mentioned that the latest impi 2019 version (2019 update 5) may no longer have this issue, so only possible workaround could be having a tweaked |
Note that this seem to be a problem only when using mpirun to start things. Using srun inside a batch job doesn't show the problem for me on Ubuntu Focal. |
Just for the record, 2018.5 also have the same problem. |
impi-2018.4.274-iccifort-2019.1.144-GCC-8.2.0-2.31.1.eb
impi-2018.5.288-iccifort-2019.5.281.eb
These Intel MPI versions segfault on CentOS 8.
From https://github.com/easybuilders/easybuild/wiki/Conference-call-notes-20201125#qa
We've decided to follow the same approach and have bumped the
impi
inintel/2019a
andintel/2019b
.The text was updated successfully, but these errors were encountered: