Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load numpy first if available #1604

Merged
merged 4 commits into from
Jan 29, 2021
Merged

Conversation

jrapin
Copy link
Contributor

@jrapin jrapin commented Jan 28, 2021

Torch users often get the following issue:

Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
        Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
srun: error: learnfair5096: task 0: Exited with exit code 1
srun: Terminating job step 26303611.0

This is due to numpy and torch being loaded in a different order when reloading the pickle. We need a way to make sure numpy is loaded first instead.

@jrapin jrapin requested a review from gwenzek January 28, 2021 13:57
@jrapin jrapin self-assigned this Jan 28, 2021
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 28, 2021
submitit/core/submission.py Outdated Show resolved Hide resolved
@jrapin
Copy link
Contributor Author

jrapin commented Jan 28, 2021

Other options:

  • avoid loading if some env variable is present
  • look in an env variable for packages to load dynamically ahead of unpickling

In any case, the current patch does work in my case.

@gwenzek
Copy link
Contributor

gwenzek commented Jan 29, 2021

For now we only had issues with numpy and torch import order, this should fix it.
I think in the general case we could dump sys.modules.keys() which is the list of module in the order in which they have been imported. This would be simillar to #1596 .

@jrapin jrapin changed the title [For discussion] Load numpy first if available Load numpy first if available Jan 29, 2021
@jrapin
Copy link
Contributor Author

jrapin commented Jan 29, 2021

For now we only had issues with numpy and torch import order, this should fix it.
I think in the general case we could dump sys.modules.keys() which is the list of module in the order in which they have been imported. This would be simillar to #1596 .

I am a bit afraid that could become black magic, but we could try eventually if we find a new problem. Should we merge this in the meantime then?

mypy.ini Outdated Show resolved Hide resolved
submitit/core/submission.py Outdated Show resolved Hide resolved
@gwenzek gwenzek merged commit eb8d652 into facebookincubator:master Jan 29, 2021
@jrapin jrapin deleted the numpy_load branch January 29, 2021 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants