libpython*.so is missing

[dmitriy-serdyuk]

Python installation seems to be strange, there is no libpython.so:

$ docker exec 5319c1eb2c41  /opt/python/cp37-cp37m/bin/python -c "from distutils import sysconfig; print(sysconfig.get_config_var('LIBDIR'))"
/opt/_internal/cpython-3.7.1/lib
$ docker exec 5319c1eb2c41 ls /opt/_internal/cpython-3.7.1/lib
pkgconfig
python3.7
$ docker exec 5319c1eb2c41  find / -name 'libpython3.7*'

Am I doing something wrong?

[rdb]

There isn't supposed to be. You do not need to link extension modules against libpython.so.

[njsmith]

This is intentional. The python interpreter can built in either "shared" mode (which creates libpython*.so files) or "static" mode (which doesn't). Extensions built against a shared-mode interpreter can only be used with shared-mode interpreters, but extensions built against a static-mode interpreter work in any interpreter. Since we want manylinux extensions to work in any interpreter, the manylinux build image contains static-mode interpreters, and has no libpython*.so.

See #30 for more gory details.

[itamarst]

I have the rather obscure use case of wanting to build Python binaries that link against libpython using the manylinux image. So this lack hit me too, but also my use case may be too obscure to be worth your time fixing.

[rdb]

@itamarst See #91 for discussion about that use case.

[itamarst]

Ah, interesting.

In my case however I'd like to actually link against libpython.so, rather than .a, so the Python used matches the patch version on the user's computer. I'm basically building a lightweight Python binary as an alternative to LD_PRELOADing; I'm assuming the user already has a Python install on their computer. Again, somewhat obscure use case :)

[rdb]

I'd say that building things that are decidedly not manylinux-compatible fall outside the scope of the manylinux project. You can make your own Docker image based on manylinux that builds a new copy of Python with shared libraries enabled. You will have to do that anyway, as manylinux builds Python statically, so no libpython.so is created at any point during the process.

Keep in mind that "having a Python install on your computer" is no guarantee of having libpython.so available. Your produced binaries, besides not qualifying for the manylinux tags, will not work with any static distribution of Python.

[dmitriy-serdyuk]

@itamarst you can try to build a new image commenting out this line https://github.com/pypa/manylinux/blob/master/docker/build_scripts/build.sh#L187

[rcoup]

Resurrecting this... would the maintainers be amenable to accepting a PR with a build-time flag that enables building shared cpython alongside the static cpython?

So, conceptually PY_BUILD_SHARED=1 PLATFORM=$(uname -m) POLICY=manylinux2014 COMMIT_SHA=latest ./build.sh which would end up with an image containing /opt/python/cp37-cp37m-shared/ alongside the static /opt/python/cp37-cp37m/, but otherwise compiled identically.

Means people who need it can use the huge work put into manylinux images without maintaining a fork.

(could be PY_BUILD_LIBS and it'd trigger leaving the .a files/etc too for those who need them?)