Reduce the amount of TLS used in pytorch

[lxj0276]

when I run the code:https://github.com/bear63/sceneReco
I shows thatImportError:dlopen:cannot load any more object with static TLS

[soumith]

this is a known issue when loading multiple libraries (torch, cv2) that use a lot of TLS. I'm not sure how to fix it.

[lxj0276]

I have fixed it.anyway thx

[soumith]

How did you fix it?

[lxj0276]

the order of import
import cv2,torch,caffe is ok,not import caffe,torch,anyway ,it's import between torch and caffe

[LiangXu123]

Very odd case,I solve it by add one line in the very beginning of my *.py file :

import torch,cv2

wherever it occurs this error.

[lionel3]

Just like @cc786537662. I solved the problem by putting the line " import torch" in the very beginning of my .py file.

[jph00]

There are some options proposed here that claim to fix it: https://stackoverflow.com/questions/19268293/matlab-error-cannot-open-with-static-tls . Not sure if any of these might help?

FYI I (and our students) can reliably replicate this bug by running lesson1 of the new fastai on Paperspace. It's not a high priority since we can just use AWS instead, but I figured you might be interested in a reliable replication environment.

[soumith]

@jph00 i am building binaries for v0.3.0 right now, so i am looking to fix this once and for-all for the new binaries.

I got a paperspace instance and tried to replicate this error. I ran a few notebooks that use pytorch under deeplearning2 but had no luck reproducing so far.

Which notebook is reproducing this error for you?
You said lesson1, but lesson1 doesn't even use pytorch: https://github.com/fastai/courses/blob/master/deeplearning1/nbs/lesson1.ipynb

Am I looking at the wrong location?

[soumith]

nevermind, i see that you're talking about https://github.com/fastai/fastai
I'm working on reproducing that.

[htwaijry]

@soumith I was having a similar issue, then I added these compile flags -fPIC -ftls-model=global-dynamic in setup.py, and recompiled pytorch, and it seems to have resolved my issue.

[soumith]

@htwaijry from what i know the source installs dont have this problem (you dont even need the compile flags). Only the binary installs have this issue. Can you confirm that if you recompiled pytorch from source (without your flags) the problem doesn't exist?

[htwaijry]

@soumith You are right, compiling from source without the flags also works. I just went ahead and added those flags when I ran into the problem without trying to compile from source directly without them.

[xiyaofu]

It is kind of hilarious. When I put cv2 before pytorch I got this error, and when I exchange the position I got the same error as #926 , what can I do?

[soumith]

@frankenstan the workaround is to do this on your machine:

rm $(dirname $(which python))/../lib/*/site-packages/torch/lib/libgomp.so*

[xiyaofu]

@soumith thx for your reply. Sadly I didn't find it in that directory.
I am in anaconda2 and ubuntu14.04. Anything else I can do?

[liruilong940607]

build pytorch from source can avoid this issue

[chenxi-ge]

I solved this issue by separately import torch early in my program. (it could be cv2 issue, as later fastai/imports.py will import cv2 for me)
However, I also saw people say we should import cv2 first.

Nevertheless, worth a try.

[PhotonChow]

Use this instruction to update your pytorch to 0.3.0 and you shall solve this problem.
conda install pytorch torchvision -c pytorch

[teichert]

Fwiw, note that the temporary workaround of putting import torch first source files using torch may be insufficient if functions from those source files are being imported elsewhere.

For example, if a.py imports torch and b.py doesn't import torch but imports functions defined in a.py, then other imports in b.py may happen before the torch import.

In my case, I saw that this was happening by noticing that some source files in the stack trace for the "ImportError:dlopen:cannot load any more object with static TLS" didn't even import torch at all. Adding import torch on the first line of source files lower on the stack trace "fixed" the problem for me.

(btw, I had this error and work-around even after the update to 0.3.0.)

[PkuRainBow]

@cc786537662 Thanks for your solution!

[ayushidalmia]

Do we have any resolution for this yet:

I am using pip install (have to use /usr/bin/python) method to install torch.

I am trying to serve a pytorch application from a docker container using gunicorn. The docker image is having cv2 and therefore when I import torch I am running into this issue.

Now, when I change the order of imports, gunicorn goes into an infinite reboot state. By any chance experienced this?

[zhangpiu]

Similar issue when I import torch in Julia using PyCall.

[jianchao-li]

I just came across this problem when I did

import tensorflow as tf
import mxnet as mx
import torch
import torchvison

Actually it looked a bit scary since a segment fault occurred after executing import torchvison. Anyway, the error disappeared if I changed the order a little bit.

import torch
import torchvision
import tensorflow as tf
import mxnet as mx

[ezyang]

With a more recent version of scikit learn, our gcc5.4 build is affected by this:

Jul 26 05:19:59 Running test_cuda ... [2019-07-26 05:19:59.821119]
Jul 26 05:20:01 Traceback (most recent call last):
Jul 26 05:20:01   File "/opt/conda/lib/python3.6/site-packages/sklearn/__check_build/__init__.py", line 44, in <module>
Jul 26 05:20:01     from ._check_build import check_build  # noqa
Jul 26 05:20:01 ImportError: dlopen: cannot load any more object with static TLS
Jul 26 05:20:01 
Jul 26 05:20:01 During handling of the above exception, another exception occurred:
Jul 26 05:20:01 
Jul 26 05:20:01 Traceback (most recent call last):
Jul 26 05:20:01   File "test_cuda.py", line 20, in <module>
Jul 26 05:20:01     from test_torch import _TestTorchMixin
Jul 26 05:20:01   File "/var/lib/jenkins/workspace/test/test_torch.py", line 46, in <module>
Jul 26 05:20:01     import librosa
Jul 26 05:20:01   File "/opt/conda/lib/python3.6/site-packages/librosa/__init__.py", line 15, in <module>
Jul 26 05:20:01     from . import decompose
Jul 26 05:20:01   File "/opt/conda/lib/python3.6/site-packages/librosa/decompose.py", line 19, in <module>
Jul 26 05:20:01     import sklearn.decomposition
Jul 26 05:20:01   File "/opt/conda/lib/python3.6/site-packages/sklearn/__init__.py", line 75, in <module>
Jul 26 05:20:01     from . import __check_build
Jul 26 05:20:01   File "/opt/conda/lib/python3.6/site-packages/sklearn/__check_build/__init__.py", line 46, in <module>
Jul 26 05:20:01     raise_build_error(e)
Jul 26 05:20:01   File "/opt/conda/lib/python3.6/site-packages/sklearn/__check_build/__init__.py", line 41, in raise_build_error
Jul 26 05:20:01     %s""" % (e, local_dir, ''.join(dir_content).strip(), msg))
Jul 26 05:20:01 ImportError: dlopen: cannot load any more object with static TLS
Jul 26 05:20:01 ___________________________________________________________________________
Jul 26 05:20:01 Contents of /opt/conda/lib/python3.6/site-packages/sklearn/__check_build:
Jul 26 05:20:01 __pycache__               setup.py                  _check_build.cpython-36m-x86_64-linux-gnu.so
Jul 26 05:20:01 __init__.py
Jul 26 05:20:01 ___________________________________________________________________________
Jul 26 05:20:01 It seems that scikit-learn has not been built correctly.
Jul 26 05:20:01 
Jul 26 05:20:01 If you have installed scikit-learn from source, please do not forget
Jul 26 05:20:01 to build the package before using it: run `python setup.py install` or
Jul 26 05:20:01 `make` in the source directory.
Jul 26 05:20:01 
Jul 26 05:20:01 If you have used an installer, please check that it is suited for your
Jul 26 05:20:01 Python version, your operating system and your platform.
Jul 26 05:20:01 Traceback (most recent call last):
Jul 26 05:20:01   File "test/run_test.py", line 408, in <module>
Jul 26 05:20:01     main()
Jul 26 05:20:01   File "test/run_test.py", line 400, in main
Jul 26 05:20:01     raise RuntimeError(message)
Jul 26 05:20:01 RuntimeError: test_cuda failed!

https://circleci.com/gh/pytorch/pytorch/2292470?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

[Arctanxy]

this error appeared when I upgrade torch from 1.1 to 1.2

[ezyang]

To consolidate, I'm closing this as a dupe of #2575

[yiyuzhuang]

This error occurs after I upgrade the version of GCC with some wrong setting.
And I fix this problem through compling the dependency of GCC,

wget https://gcc.gnu.org/pub/gcc/infrastructure/isl-0.18.tar.bz2
tar xvf isl-0.18.tar.bz2 
cd isl-0.18
./configure --prefix=/usr/local/isl --with-gmp-prefix=/usr/local/gmp/
make && make install

Btw, I don't know the reason, and hope for futher disscussion.