New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GOMP_4.0 not found #643

Closed
UndeadBlow opened this Issue Jan 30, 2017 · 35 comments

Comments

Projects
None yet
@UndeadBlow
Copy link

UndeadBlow commented Jan 30, 2017

Ubuntu 16.04 LTS
OpenCV 3.2

ImportError: /usr/local/lib/python2.7/dist-packages/torch/lib/libgomp.so.1: version `GOMP_4.0' not found (required by /usr/local/lib/python2.7/dist-packages/torch/lib/libTH.so.1)

Tried to install with pip and from sources, tried to rebuild and reinstall all depencies, like OpenCV. Everything works perfectly, but PyTorch doesn't want to import.

What can I do?

@apaszke

This comment has been minimized.

Copy link
Member

apaszke commented Jan 30, 2017

Did you do a binary install or did you build from source?

@UndeadBlow

This comment has been minimized.

Copy link
Author

UndeadBlow commented Jan 30, 2017

@apaszke I tried via Pip and via source, both the same

@apaszke

This comment has been minimized.

Copy link
Member

apaszke commented Jan 30, 2017

I'd expect it to break because of installing from source. You probably have a new gcc version that builds with OpenMP 4.0, while the lib we ship doesn't support it. But I'm surprised it's broken for binary installs too. Can you try removing (pip uninstall torch) and reinstalling PyTorch from a link on our website?

@apaszke

This comment has been minimized.

Copy link
Member

apaszke commented Jan 30, 2017

Are you using anaconda or just pip?

@apaszke

This comment has been minimized.

Copy link
Member

apaszke commented Jan 30, 2017

Also, do you have libgomp.so in any of the system lib directories (e.g. /usr/lib*, /usr/local/lib*). And what's your compiler?

@ngimel

This comment has been minimized.

Copy link
Contributor

ngimel commented Jan 30, 2017

FWIW, replacing FROM image to say 16.04 in https://github.com/pytorch/pytorch/blob/master/Dockerfile#L1 and building docker image works (I can import torch), so building from source should work - try building using commands from the Dockerfile.

@UndeadBlow

This comment has been minimized.

Copy link
Author

UndeadBlow commented Jan 31, 2017

@apaszke I'm using pip. Tried to re-install everything like on the site but exactly the same error. So, as I said, seems that running pip and building from source giving the same result.
Seems that libgomp located in /usr/lib/x86_64-linux-gnu/
My compiler is GCC 5.4.0

@colesbury

This comment has been minimized.

Copy link
Member

colesbury commented Jan 31, 2017

Can you run a simple script that imports torch under strace and post the output: Something like:

simple.py:

import torch

strace python simple.py 2>trace.txt

@UndeadBlow

This comment has been minimized.

Copy link
Author

UndeadBlow commented Feb 1, 2017

@soumith

This comment has been minimized.

Copy link
Member

soumith commented Feb 2, 2017

@UndeadBlow i see what's happening. libgomp.so.1 is being loaded from the pytorch binary install even though you built from source maybe.

first step: pip uninstall torch

Then, build from source via the commands provided in the Dockerfile if that works for you: https://github.com/pytorch/pytorch/blob/master/Dockerfile#L1

@UndeadBlow UndeadBlow closed this Feb 2, 2017

@UndeadBlow UndeadBlow reopened this Feb 2, 2017

@UndeadBlow

This comment has been minimized.

Copy link
Author

UndeadBlow commented Feb 2, 2017

Ok, I'm sorry, but problem is still here. If I'm installing to cleaned system from pip, I'm getting error from subj.
If I'm installing from sources, I'm getting:

import torch
Traceback (most recent call last):
File "", line 1, in
File "torch/init.py", line 45, in
from torch._C import *
ImportError: No module named _C

@fmassa

This comment has been minimized.

Copy link
Member

fmassa commented Feb 2, 2017

@UndeadBlow could you try opening python from a different folder? This looks related to #7

@UndeadBlow

This comment has been minimized.

Copy link
Author

UndeadBlow commented Feb 2, 2017

@fmassa Gracias, you are right.

@UndeadBlow UndeadBlow closed this Feb 2, 2017

@d1ngn1gefe1

This comment has been minimized.

Copy link

d1ngn1gefe1 commented Mar 1, 2017

I fixed the problem by importing opencv before torch.

@lopuhin

This comment has been minimized.

Copy link
Contributor

lopuhin commented Apr 8, 2017

fwiw, I think this issue is still not completely fixed, at least if you try to import xgboost with pytorch. Here are the steps on Ubuntu 16.04 with default Python 3.5 and pip:

$ python3.5 -m venv pytorch-venv
$ . pytorch-venv/bin/activate
$ pip install http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp35-cp35m-linux_x86_64.whl
Collecting torch==0.1.11.post5 from http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp35-cp35m-linux_x86_64.whl
  Downloading http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp35-cp35m-linux_x86_64.whl (475.1MB)
    100% |████████████████████████████████| 475.1MB 11.3MB/s
Collecting pyyaml (from torch==0.1.11.post5)
Installing collected packages: pyyaml, torch
Successfully installed pyyaml-3.12 torch-0.1.11.post5
$ pip install numpy
Collecting numpy
  Using cached numpy-1.12.1-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.12.1

$ python
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>

$ pip install xgboost
Collecting xgboost
Collecting scikit-learn (from xgboost)
  Using cached scikit_learn-0.18.1-cp35-cp35m-manylinux1_x86_64.whl
Collecting scipy (from xgboost)
  Using cached scipy-0.19.0-cp35-cp35m-manylinux1_x86_64.whl
Requirement already satisfied (use --upgrade to upgrade): numpy in ./pytorch-venv/lib/python3.5/site-packages (from xgboost)
Installing collected packages: scikit-learn, scipy, xgboost
Successfully installed scikit-learn-0.18.1 scipy-0.19.0 xgboost-0.6a2

$ python
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> from xgboost import XGBClassifier
/home/kostia/pytorch-venv/lib/python3.5/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kostia/pytorch-venv/lib/python3.5/site-packages/xgboost/__init__.py", line 11, in <module>
    from .core import DMatrix, Booster
  File "/home/kostia/pytorch-venv/lib/python3.5/site-packages/xgboost/core.py", line 112, in <module>
    _LIB = _load_lib()
  File "/home/kostia/pytorch-venv/lib/python3.5/site-packages/xgboost/core.py", line 106, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/usr/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/kostia/pytorch-venv/lib/python3.5/site-packages/torch/lib/libgomp.so.1: version `GOMP_4.0' not found (required by /home/kostia/pytorch-venv/lib/python3.5/site-packages/xgboost/./lib/libxgboost.so)
@soumith

This comment has been minimized.

Copy link
Member

soumith commented Apr 8, 2017

man, this stuff is so tricky to solve.

If we compile with a new-enough GCC, then we have a problem of our OpenMP being too new that if any other package loads first, and pytorch loads subsequently, then pytorch will fail to load.

If we compile with an old-enough GCC, then if pytorch loads first and other packages load later, then the other package will fail to load.

WHY DOESNT OPENMP HAVE STATIC COMPILATION???

@soumith

This comment has been minimized.

Copy link
Member

soumith commented Apr 8, 2017

and @lopuhin in your case, loading xgboost first and then loading pytorch might solve the error?

@lopuhin

This comment has been minimized.

Copy link
Contributor

lopuhin commented Apr 8, 2017

yeah, tricky stuff indeed :(

and @lopuhin in your case, loading xgboost first and then loading pytorch might solve the error?

yes, it does solve the problem, thanks @soumith !

@amirsaffari

This comment has been minimized.

Copy link

amirsaffari commented Apr 8, 2017

Experiencing similar issues with fresh build from source without anaconda (ubuntu 16.04 with virtualenv). In my case it fails even with python -m torch.

Would this help at all? http://stackoverflow.com/questions/23869981/linking-openmp-statically-with-gcc

Something along the lines of

ln -s `g++ -print-file-name=libpthread.a`
ln -s `g++ -print-file-name=libc.a`
g++ foo.cpp -static-libgcc -static-libstdc++ -L. -o foo -O3 -fopenmp
@soumith

This comment has been minimized.

Copy link
Member

soumith commented Apr 8, 2017

Experiencing similar issues with fresh build from source without anaconda

Wait, source builds SHOULD not have this problem. Source builds will link against the OpenMP given by your compiler.

@amirsaffari

This comment has been minimized.

Copy link

amirsaffari commented Apr 8, 2017

Give me a few min to verify if I did make a clean install. BRB

@amirsaffari

This comment has been minimized.

Copy link

amirsaffari commented Apr 8, 2017

My bad -- it wasn't as clean as I thought, it was linking back to an older version. So ignore :)

@soumith

This comment has been minimized.

Copy link
Member

soumith commented Apr 8, 2017

to your other note, i'll try -static-libgcc. I thought that flag only statically linked libgcc.

@soumith

This comment has been minimized.

Copy link
Member

soumith commented Apr 8, 2017

yea, so -static-libccc wont be sufficient.

soumith@superbox:~/Downloads$ gcc -fopenmp foo.c
soumith@superbox:~/Downloads$ ldd a.out
        linux-vdso.so.1 =>  (0x00007ffdeab3c000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f0dca55e000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0dca341000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0dc9f77000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0dc9d73000)
        /lib64/ld-linux-x86-64.so.2 (0x0000557fecbc0000)
soumith@superbox:~/Downloads$ gcc -fopenmp foo.c -static-libgcc
soumith@superbox:~/Downloads$ ldd a.out
        linux-vdso.so.1 =>  (0x00007ffc9bdda000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f2b2f740000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2b2f377000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2b2f172000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2b2ef55000)
        /lib64/ld-linux-x86-64.so.2 (0x000055a50f829000)
@soumith

This comment has been minimized.

Copy link
Member

soumith commented Apr 8, 2017

I think I did find a solution though:

soumith@superbox:~/Downloads$ gcc -fopenmp foo.c $(g++ -print-file-name=libgomp.a) -ldl
soumith@superbox:~/Downloads$ ldd a.out
        linux-vdso.so.1 =>  (0x00007fff20680000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f161027f000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1610062000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f160fc98000)
        /lib64/ld-linux-x86-64.so.2 (0x000055e8eee7b000)

Thanks for starting the conversation and helping me investigate.

@amirsaffari

This comment has been minimized.

Copy link

amirsaffari commented Apr 8, 2017

Glad it helped :)

@varunagrawal

This comment has been minimized.

Copy link
Contributor

varunagrawal commented Apr 15, 2017

I recently encountered this exact problem when compiling from source. A clean install also didn't help. What worked was removing the libgomp.so.1 library from the torch directory and instead sym-linking it to the one in /usr/lib.

ln -s /usr/lib/libgomp.so.1 path/to/python/site-packages/torch/lib/libgomp.so.1
@acgtyrant

This comment has been minimized.

Copy link
Contributor

acgtyrant commented May 2, 2017

@soumith I find http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp35-cp35m-linux_x86_64.whl is still compiled with libgomp.so.1, so the pytorch will be compiled with the stastic OpenMP library in next release?

@mherkazandjian

This comment has been minimized.

Copy link

mherkazandjian commented May 24, 2017

how about using

LD_PRELOAD=/path/to/libgomp.so  your_commnad_here

this is i think better than also creating a symlink to libgomp.so

@ppwwyyxx

This comment has been minimized.

Copy link
Contributor

ppwwyyxx commented Jun 10, 2017

Found that libgomp.so still exists in latest pytorch (0.1.12.post2), and causing the same problem.

@lzx1413

This comment has been minimized.

Copy link

lzx1413 commented Jul 13, 2017

I remove the libgomp.so.1 in /path/to/python/site-packages/torch/lib/ and the keep the same one in /usr/lib, and the problem is solved

@soumith

This comment has been minimized.

Copy link
Member

soumith commented Jul 18, 2017

to get closure to this thread, sadly as I'm building 0.2 binaries now, I've realized that I cannot link against static OpenMP because libgomp.a shipped by gcc is not compiled with -fPIC.

I'm not willing to maintain my own compiler toolchain for this purpose -- where I modify gcc build scripts to enable libgomp.a with -fPIC and recompile a custom gcc.

@anilsh

This comment has been minimized.

Copy link

anilsh commented Aug 4, 2017

Importing opencv before torch solved the problem.

@etworker

This comment has been minimized.

Copy link

etworker commented Aug 16, 2017

@lzx1413 thanks for your solution, it works on my pc like a charm!

@evanthebouncy

This comment has been minimized.

Copy link

evanthebouncy commented Apr 30, 2018

can anyone explain why @d1ngn1gefe1 solution work? It's such a hack but out of all the crazy shannanigans in this thread it's the simplest fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment