OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already intialized

[nikohunt]

Hi,
Thanks for the amazing project. This was a fresh install on a new python 3.8.3 virtual environment using pip install and launching straight away. The web app launches, but after a few key strokes, the process crashes with the following error logged to the terminal:
OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/. Abort trap: 6

Any advice appreciated here, thanks again.

[PJ-Finlay]

Thanks for the amazing project.

Thanks for using it!

The web app launches

Argos Translate has a PyQt native app, LibreTranslate is a web app/api build on top of Argos Translate which are you using?

I've never seen this before but looking around it looks like this is probably some weird interaction between CTranslate and Intel CPUs on Mac. It looks like export KMP_DUPLICATE_LIB_OK=True may fix the issue but I don't really understand why. It seems like it's a bit of a hack and may have negative side effects. @guillaumekln any idea?

Related issues:

• OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized. dmlc/xgboost#1715
• https://stackoverflow.com/questions/53014306/error-15-initializing-libiomp5-dylib-but-found-libiomp5-dylib-already-initial

[guillaumekln]

libiomp5.dylib is the Intel OpenMP runtime that is used for multithreading. It is required by CTranslate2 and is integrated in the Python package. But Argos Translate also uses Stanza that depends on PyTorch which includes another copy of libiomp5.dylib. This is probably why it complains about multiple copies of the library.

Both PyTorch and CTranslate2 are correct to bundle this library into their Python package, so I'm not sure what should be done here. The workaround suggested in the error message should work and I don't think it will cause issues.

[PJ-Finlay]

Interesting, that's a tricky one. If this is a reoccurring issue maybe Argos Translate should set the KMP_DUPLICATE_LIB_OK environment variable automatically.

[PJ-Finlay]

I have a potential fix for this pushed up. @nikohunt can you try installing from source to see if this fixes the issue? I didn't want to always set the environment variable so I try to detect if Argos Translate is running on an Intel processor with platform.processor(). This isn't foolproof so if this error reoccurs then we can change to always setting the environment variable.

[nikohunt]

Thanks a lot @PJ-Finlay! I pulled down argos just now and built; still the same issue. However, upon checking the fix, I ran something similar on my machine and noticed the following:

>>>import platform  
>>>print(platform.processor().lower())  
i386

[nikohunt]

I'm happy to try and branch and make a pull request for this one - never contributed to a project that's not my own before!

[PJ-Finlay]

That was what I was worried about. It doesn't seem like there is a good way to consistently determine if someone is using an Intel CPU with the Python standard library and I don't want to add a dependency for this. platform.processor().lower() doesn't include the CPU type on my Ubuntu 20.04 setup and it looks like it doesn't on yours either (I'm guessing you're using Mac?). I think the best fallback is going to be to just set the environment variable for everyone, it seems specific enough that it won't cause problems.

I'm happy to try and branch and make a pull request for this one - never contributed to a project that's not my own before!

That would be great! I would look into the Python platform docs and see if there is something that works on Mac/Intel otherwise just remove the if statment.

[nikohunt]

Thanks. Yes, I'm on a macbook. I can use the following to get the "intel" piece out reliably in the terminal:
'sysctl -n machdep.cpu.brand_string'
I would need a way to run this reliably, though, as obviously this would error out in other hardware scenarios - is try/except considered bad form? I could check with platform.system()first to see if darwin...
Thanks again.

[PJ-Finlay]

I think that would work but if it's this involved I think it's better to just remove the if statement and set the environment variable for everyone. I have an Intel CPU and don't have this issue on Linux but this could also happen on Windows so best to be as broad as possible.

Thanks for handling this.

[PJ-Finlay]

Just pushed a patch to PyPI fixing this 1.1.3.