torch/_C.so exports a symbol named "state"

[drawoc]

Issue description

Importing both pytorch and hiredis into the same python program often results in a segfault.

After much debugging, I was able to trace this back to the fact that both shared objects export a symbol named state, which causes hiredis to overwrite pytorch's internal variables with its own internal state.

Code example

I can reproduce this consistently with:

$ echo "import torch, hiredis" | python2
Segmentation fault (core dumped)

Note that the segfault happens when the interpreter exits, not during the import

Stack trace:

==1890== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==1890==  Bad permissions for mapped region at address 0x297BA4A4
==1890==    at 0x297BA4A4: ???
==1890==    by 0x1BD2D00E: ~_Function_base (functional:1998)
==1890==    by 0x1BD2D00E: ~function (functional:2142)
==1890==    by 0x1BD2D00E: ~THPCopyInfo (copy_utils.h:8)
==1890==    by 0x1BD2D00E: _Destroy<THPCopyInfo> (stl_construct.h:93)
==1890==    by 0x1BD2D00E: __destroy<THPCopyInfo*> (stl_construct.h:103)
==1890==    by 0x1BD2D00E: _Destroy<THPCopyInfo*> (stl_construct.h:126)
==1890==    by 0x1BD2D00E: _Destroy<THPCopyInfo*, THPCopyInfo> (stl_construct.h:151)
==1890==    by 0x1BD2D00E: std::vector<THPCopyInfo, std::allocator<THPCopyInfo> >::~vector() (stl_vector.h:424)
==1890==    by 0x5090FF7: __run_exit_handlers (exit.c:82)
==1890==    by 0x5091044: exit (exit.c:104)
==1890==    by 0x5077836: (below main) (libc-start.c:325)

System Info

• PyTorch or Caffe2: PyTorch
• How you installed PyTorch: Can reproduce with pip, source, and distro packages
• OS: Ubuntu 16.04 or Arch Linux (up to date 2018-09-07)
• PyTorch version: 0.4.1
• Python version: 2.7
• CUDA/cuDNN version: 9.0/7.0
• GPU models and configuration: N/A
• Versions of any other relevant libraries: hiredis (python) 0.2.0

I think this only happens in builds with GPU support.

Potential Fix

I'd like to suggest that this issue is fixed by compiling all the shared object files with -fvisibility=hidden on gcc/clang, and then defining the *_API macros to set __attribute__((__visibility__("default"))) when compiling. Note that part of the machinery to do this is already in place, but not consistently used (eg. AT_API has the define on master, while THC_API does not. There is build code to detect the flag fvisibility=hidden, but it doesn't appear to be set for most of the libraries)

Fixing it this way should eliminate this entire class of errors.

I can write a PR that does this, if that would help.

[drawoc]

I'll add that you can verify that the symbol state is exported with nm:

$ nm --defined-only -D /usr/lib/python2.7/site-packages/torch/_C.so | grep -e state$
00000000011602f0 B state

[soumith]

yes, we export a global symbol called state, of type THCState.
We no longer need to do this, and probably can get rid of exporting it (or atleast codemod it).

cc: @gchanan is it still exported on master?

[ngimel]

Closing due to age