abort when importing modules in Python 2 inside a virtualenv

[thanatos]

If I attempt to compile a module for Python 2, and import it in a python running out of a virtualenv, I get an abort:

% env/bin/python2 -c 'import _foo'
Fatal Python error: PyThreadState_Get: no current thread
zsh: abort      env/bin/python2 -c 'import _foo'

but the system Python works just fine:

% python2 -c 'import _foo'
%

They should be mostly the same:

% python2
Python 2.7.12 (default, Jun 29 2016, 14:04:44)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
% env/bin/python2
Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin

(this is an virtualenv created with virtualenv env; I don't see these issues in Python 3, with a python3 running inside a pyvenv venv virtual environment.)

[dgrunwald]

What might be happening here is that the extension module calls into the system python library, which aborts because it's not initialized (only the venv python is initialized).

The problem is that python27-sys dynamically links against the system python. That's a bad idea for extension modules, though I'm not sure how to best fix this while using cargo.

[thanatos]

That was my working hunch.

The article mentions running otool -L, and that seems to confirm it:

_foo_2.so:
    $HOME/code/$PROJECT/python/target/debug/deps/lib_foo.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
_foo_3.so:
    $HOME/code/$PROJECT/python/target/debug/deps/lib_foo.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python (compatibility version 3.5.0, current version 3.5.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

I was able to build a dylib successfully by changing two things:

• I built python27-sys/python3-sys without the -l python3.5m in the rustc command, in line w/ what the article says to do. This caused linker errors later, which is good.
• I build my module with -C link-args='-undefined dynamic_lookup', again following the article's directions.

This produced a dylib that I was able to successfully import; it also lacks the reference to Python in the otool -L output. So that seems to work! Unfortunately, I did the above two steps completely manually: I build first with cargo build, and then I re-ran those two rustc calls above, specifically. I'm also unsure how to fix this while using cargo.

[naufraghi]

Is this issue perhaps fized with the extension-module feature addition in e2d7781?

[dgrunwald]

The extension-module feature should fix this on Linux (where keeping symbols unresolved is the default), but I couldn't find a way to get cargo to pass the -undefined dynamic_lookup flag to the linker for OSX.

[naufraghi]

cargo uses the RUSTFLAGS env variable in its appveyor.yml config:

RUSTFLAGS: -Zunstable-options -Ctarget-feature=+crt-static

Can this help?

[fafhrd91]

it is posible with:

cargo rustc --lib --features "cpython/extension-module-2-7 cpython/python27-sys" -- --crate-type cdylib -C link-arg=-undefined -C link-arg=dynamic_lookup

should be added to doc somewhere

[markbt]

The workaround for OSX was added to the docs.