-
Notifications
You must be signed in to change notification settings - Fork 214
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Missing libpython with Debian default Python install #30
Comments
Same on Ubuntus 14.04, 15.10. |
Where would such a check go? This is really a Debian problem with the way they split upstream python into many subpackages, no? |
It would have to go in the pip check for the validity of manylinux for the platform. It is unfortunate that Debian does this, but I bet that a large majority of our consumers will be on Debian or Ubuntu, and there's not much prospect of Debian / Ubuntu changing this immediately, and even if they do, there will be many systems liable to this problem in the wild. |
But on that system, if you disable |
Because clearly the system doesn't have the python header files ( |
Right - good point. On the other hand, on a Debian-based system, if I try an sdist pip install I first get an error about being unable to compile a C program, and, after installing
That makes it pretty clear what I have to do. On the other hand, if we blanket enable manylinux wheels, on systems without |
Do you know where that SystemError is being thrown from? Is that check in pypa/pip, or something numpy-specific. |
I guess that's a numpy thing: https://github.com/numpy/numpy/blob/master/numpy/core/setup.py#L268-L270 |
Yes, it is a numpy thing, and so a bad example:
Trying with another package,
So - error, arising from gcc not very helpful, but easily solved with a web search. I guess it would make the transition easier, if we could give a more helpful message than that. |
@matthew-brett: Here's a wacky question. What happens if you do Reason I ask: For a regular interpreter where the executable's linked against libpython2.7.so.1, it shouldn't matter whether multiarray.so has a DT_NEEDED for libpython2.7.so.1, b/c it will be loaded into the ELF namespace regardless. And for Debian, /usr/bin/python2.7 itself seems to be exporting all the standard symbols... in fact, the OTOH debian doesn't seem to be consistent -- some python-foo packages depend on libpython, and some don't. And I don't know what they've done to distutils so that it doesn't link to -lpython2.7. Something pretty weird going on here :-/ (The one place where failing to link to libpython will break, AFAICT, is if you have a python interpreter that's embedded, but not via DT_NEEDED linking to an executable like gnucash or blender or whatever, but instead the python interpreter itself is dlopen'ed (or linked to another .so that gets dlopen'ed, etc.). In this case the python symbols don't make it into the global ELF namespace, so that won't be visible to extension modules. So AFAICT it really is better if python extensions link to libpython explicitly, and libpython gets included on distros out-of-the-box -- that's how fedora does it. But maybe we can't fix this anytime soon :-/.) |
Yup, doing this:
allows the test to run without any missing libraries. Are you suggesting that we do this for manylinux wheels in general? |
I'm really not sure... |
Ah, here's an example Debian-compiled extension module that does link to libpython:
Why? I don't know. |
How about floating the idea of editing out the libpython dependency on distutils-sig? |
I'm very nervous about editing it out like that, especially in an automatic tool like |
I guess the idea would be to have another tool or a command line flag to auditwheel to do this extra processing. I think our options are either to add a useful warning and instruction in pip, or to do this hack - what do y'all think? |
I bet the qt libpython dependency is an accident of the complexity of building the package. |
The PEP text says that it's permissible for manylinux wheels to link against libpython. So adding a check to the pip platform detection PR that ensures the library is available would be my vote. |
If we just do that we get the following awful UX:
So if we do want to stick with allowing manylinux wheels to link against libpython, then my vote would be that we add a special case to pip to detect missing libpython, and if that's the only reason it can't install a manylinux wheel then it should error out with some useful error message like "you should install libpython". Anyway, as explained in the email the full details are actually rather complicated and subtle... I guess we can see what people say. Maybe just not linking to libpython is a good solution, and it doesn't necessarily have to require auditwheel to do the fixups. |
I'll respond there. |
I guess that pip would first have to detect whether there was a manylinux wheel available to install, before raising the error, otherwise the user will lose the ability to do installs of pure python packages from sdists. That in turn would mean that if the author of a pure Python package made a manylinux wheel, then users would start getting an error instead of the installed package, on these systems. |
No, that's the key point: the way Debian/Ubuntu have their packages set up, any situation in which there is a manylinux wheel and it won't work, is also a situation where if you tried installing the sdist instead, that wouldn't work either. If you're missing libpython, and your only options are sdist or manylinux-that-depends-on-libpython, then you're doomed no matter what, so we might as well give you an error message that leads you to the simple solution (install libpython so you can use the wheel). That said, after further investigation I think I am leaning towards saying that manylinux1 wheels should not link to libpython (at least for current python versions, and maybe we can fix it for 3.6) -- I will send a more detailed email to distutils-sig later. |
If I see things correctly, there are a couple options. None of them are great.
|
Nathaniel - to make sure we understand each other: With current Debian / Works:
Fails:
We're proposing to raise an error from pip if Presumably we don't want to raise this error for The question then is, how to know when to raise an error. We could ask pip to try and work out whether the sdist needs compilation or not (is there an easy way)? Or we could ask pip to look for a manylinux wheel, and fail in that case, on the basis that this is likely to be a package that pip cannot install without the manylinux wheel. But that does have the disadvantage that if the packager of EDIT: of course (I realized when sitting in a church pew today) you can't build a 'manylinux1' wheel for |
Maybe pip or Debian could be persuaded to include the logic to give a better error message if pip tries and fails to compile package from an sdist - something like "Cannot compile package with extension code; you may need to install python-dev". |
We'd still be leaving other people that use statically-compiled Pythons either on a different distro, or compiling from source, in the lurch. |
Robert - you mean, if we use the pip error, then people with statically compiled Pythons will not be able to use manylinux wheels? But, if we remove the libpython stated dependency from the manylinux wheels, I think these people will be able to use the wheels, except in weird cases like this : https://bugs.python.org/msg18810 |
I'm not totally sure about this one way or the other, but I think my position is that
|
The fact that Debian and Ubuntu has been packaging most extensions without a libpython dependency for a long time, makes me think that we are very likely to get away with it for manylinux. |
Right, so are we agreeing here, that the PEP text should be changed to disallow linking to libpython? |
Yes, I think we're all agreeing now. But I thought you said over on distutils-sig that you didn't think that was acceptable? |
I checked two programs that have plugins where the plugins embed python -- libreoffice and xchat. In both cases the program ends up using RTLD_GLOBAL to load libpython. This is terrible and not what you should do, but it does mean that they will with correctly even with extensions that don't link against libpython. (I guess this is probably why they do this terrible thing.) And manylinux is a relentlessly practical design, so... (You can check by running the program with |
Matthew, sort for being unclear. These issues as confusing, and typing on my phone doesn't help. After we agree on the necessary changes in the policy and the PEP text (to remove libpythonx.y.so from the list of permitted external shared libraries), the next question is what changes are going to be made to the docker images and auditwheel. One implementation route is to change the docker image to use a static builds of python. This will mean that all compiled python extension won't link libpythonx.y.so, since the build box won't even contain a libpythonx.y.so. I think this is the right route. I may have misunderstood the discussion, but my impression was that a second option was also being floated, which was to leave the docker image unchanged, but add code to 'auditwheel repair' that would use patchelf to remove the DT_NEEDED dependencies in libpythonx.y.so after the wheel has been compiled. This was the route that I don't think is satisfactory. |
I guess I was assuming we might want to do both. :-) (auditwheel should at the least check for linking to libpython and yell at people if found, right?) |
Yes, I agree definitely that auditwheel should check for linking to libpython, and give a special error message if found -- this is clearly the minimum required behavior to implement the PEP-as-modified. I think we're in agreement here, so we can defer the discussion about what extra features auditwheel should or should not add for later. (Wearing my |
Perhaps, for example, we could add a feature to But anyways, we can discuss this later -- it's not the first thing that needs to be implemented. |
Okay, #31 has proposed text for mandating that no-one link to libpython; review would be great. Re: That said, it might be good in general to change auditwheel's behavior so that instead of just trying to fix whatever it finds, it has one mode that says "here's the fixes needed" and then another mode where the user explicitly names (maybe in a config file) which fixes they want -- e.g. they have to explicitly list which libraries they want to vendor, etc. This should also help avoid all kinds of surprises -- e.g., you don't want to accidentally start distributing libraries without thinking about the licenses, maybe because the new version of upstream added a new dependency you didn't notice and auditwheel silently swept it up... |
Here's the situation I've been considering. Imagine a wheel that contains two In this case, my understanding (although I haven't tested it) is using patchelf to remove |
No, that case should be fine. As long as Actually, this is easy to demonstrate --
then we see a bunch of output, including:
which means that when resolving each symbol that occurs inside |
Oh, okay. My mistake then. Perhaps this use of |
This seems to be resolved, so closing... re-open if I missed something :-) |
Hey @njsmith @matthew-brett @ncoghlan We are currently working on making a manylinux wheel for a project that embeds python. The project is xeus-python, which is a Jupyter kernel for the Python programming language written in C++ which embeds the interpreter. (The work on the wheel is happenning here). We are trying to figure out what is the best way to achieve that given that libpython is missing. We are even considering building the Python interpreter that we will statically link in our executable as part of the wheel building. Are you aware of another project that would be doing something similar? |
@SylvainCorlay At Panda3D we also have a need to embed the interpreter. We solve this by just building a fork of manylinux1 which has as only difference that it doesn't delete libpython.a from the container: See #91, which talks about this use case. |
Wow, that is exactly what I was hoping for. It would be really great if manylinux what like this already. |
It seems that the base image for the manylinux 2010 is not pullable. Is this meant to be private? @njsmith @matthew-brett @ncoghlan would it be possible to make your base image public? We are making the same change as @rdb but for the manylinux2010 docker image. Note that I would also support upstreaming this change into the manylinux docker image. |
I am testing manylinux numpy wheels.
In particular, I am testing this guy: http://nipy.bic.berkeley.edu/manylinux/numpy-1.10.4-cp27-none-linux_x86_64.whl
With a default install, test of Python, starting with either Wheezy or Jessie:
and running this script:
I get this:
Sure enough, in order for the wheel to work, I need:
Maybe we need to add a check / informative error message for the presence of libpython?
The text was updated successfully, but these errors were encountered: