Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could not train the model on AWS #173

Open
Li-En-Good opened this issue May 1, 2020 · 2 comments
Open

Could not train the model on AWS #173

Li-En-Good opened this issue May 1, 2020 · 2 comments

Comments

@Li-En-Good
Copy link

I was trying to use the pytorch_fnet on an amazon EC2 instance (I used a g4 instance).
When I run the download_and_train.py, it always gives me:
_mkl-service + Intel(R) MKL_ MKL_THREADING_LAYER=INTEL is incompatible with libgomp-7c85b1e2.so.1 library.Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it_
I tried importing numpy first but it still gave the same error.

I also tried to use release_1 but the pytorch=0.1.8 doesn't seem to fit the EC2 instance so conda env create -f environment.yml failed.
Please let me know if there is a way to fix this.
Thanks a lot!

@fcollman
Copy link
Collaborator

fcollman commented May 2, 2020

This seems related to differences in compilers that were used to create the numpy release you have installed on your environment. This is likely caused by using some combination of conda installations and pip installations of different libraries. Since you are on a linux instance, I would try falling back to using only pip to install and compile packages, rather than relying on pre-compiled binaries from conda.

@fcollman
Copy link
Collaborator

fcollman commented May 2, 2020

You could also try relaxing some of the constraints in the environment.yml file, depending on what the conda failure was. Usually this is some incompatibility of available pre-compiled binaries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants