Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panda3D wheel for linux contain static version of the libpython library #839

Closed
el-dee opened this issue Jan 9, 2020 · 6 comments
Closed

Panda3D wheel for linux contain static version of the libpython library #839

el-dee opened this issue Jan 9, 2020 · 6 comments
Assignees
Labels
Milestone

Comments

@el-dee
Copy link
Contributor

@el-dee el-dee commented Jan 9, 2020

The manylinux wheel of Panda3D contains the static version of libpython, e.g. deploy_libs/libpython3.7m.a in panda3d-1.10.5+opt-cp37-cp37m-manylinux1_x86_64.whl, where macosx version contains deploy_libs/libpython3.7.dylib and win_amd64 contains deploy_libs/python37.dll

This causes problem with CEFPython whose library on Linux, cefpython3.cefpython_py37.so, refers to libpython3.7m.so.1.0

I detected this while writing a CI workflow for my app that now test the generation of the app binaries and test a headless version of it to check that all the dependencies are working correctly.

This went under the radar on my hosts as they are all dev machines with Python installed on them and so the missing library was found in the system.

@el-dee

This comment has been minimized.

Copy link
Contributor Author

@el-dee el-dee commented Jan 9, 2020

What is weird is that, using GitHub Workflow, the error is detected every time, on the other hand on Travis it seems that it sometime find a libpython3.7m.so.1.0 somewhere...

@rdb

This comment has been minimized.

Copy link
Member

@rdb rdb commented Jan 9, 2020

This is a CEFPython build issue. Python does not allow extension modules to link with libpython.so. Instead, the symbols are made available dynamically by the interpreter. This is to ensure that extension modules can be loaded by other interpreters than the one it was linked against, and even static builds of Python (which are common).

There is no shared library for Python available in the manylinux container since the copy of Python was built statically there. This is in accordance with the manylinux specification.

I suppose that it is a mistake that we ship the .a file at all, since deploy-ng has no use for it.

You should file this as an issue upstream; CEFPython needs to use a special linker flag to enable creating the extension module without linking to libpython.

@el-dee

This comment has been minimized.

Copy link
Contributor Author

@el-dee el-dee commented Jan 9, 2020

Ok, and now I understand why cefpython3.cefpython_py37.so on macOS does reference lib libpython3.7.dylib.

I will fill an issue for CEFPython, do you have a reference in the Python documentation that I could use ?

EDIT: Found this information https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1

@rdb

This comment has been minimized.

Copy link
Member

@rdb rdb commented Jan 9, 2020

I think on macOS the practice is also not to link against libpython3.7.dylib. In fact, doing so makes it impossible to use a different distribution of Python, such as Homebrew.

What's a little odd is that the manylinux1 container doesn't even contain libpython3.7.so, so they must have had to go out of their way to modify it. Or are they not using the manylinux1 container, yet calling it manylinux1? You might want to use auditwheel to check that these are in fact valid manylinux1 wheels to begin with; if not, you can report the auditwheel results upstream.

Here is the documentation describing that manylinux1 wheels may not link to libpython3.7.so.1:
https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1

This page shows the linker flags you should use to avoid the need to link to libpython: (I remember it's in more recent docs too, I just can't find it right now)
https://docs.python.org/2.0/ext/link-reqs.html

I wonder if you can work around this bug by shipping a dummy libpython3.7m.so.1.0 containing no symbols.

@el-dee

This comment has been minimized.

Copy link
Contributor Author

@el-dee el-dee commented Jan 9, 2020

I wonder if you can work around this bug by shipping a dummy libpython3.7m.so.1.0 containing no symbols.

Just tested it works fine with an empty shared lib named libpythonX.Y.so.1.0 so I added in my app one for each supported Python version.

@el-dee

This comment has been minimized.

Copy link
Contributor Author

@el-dee el-dee commented Jan 10, 2020

For reference, here is the upstream issue : cztomczak/cefpython#554

@rdb rdb added this to the 1.10.6 milestone Mar 2, 2020
@rdb rdb self-assigned this Mar 2, 2020
@rdb rdb closed this in 8a86ca9 Mar 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.