Handling shared libraries

[jakirkham]

Am still learning how this tool works. So am not sure what it is already done to address this issue. Namely how do shared libraries (outside of the Python package) get handled? Alternatively if these are open questions, how should they be handled?

1. What shared libraries are included within the wheel? How are these named to avoid clashing?
2. What libraries are expected to come from the system? How are these managed?
3. How are shared libraries specific to the Python package/wheel handled w.r.t. 1 & 2?

[scopatz]

OK so the basic strategy is that shared libraries go into site-packages/lib. This is because the wheel format dictates that all files, by default, go into site-packages. We don't really want to move these around to avoid issues of breaking the links.

What shared libraries are included within the wheel? How are these named to avoid clashing?

All of them. Name clashes are avoided because very few things go into site-package/lib

What libraries are expected to come from the system? How are these managed?

The same as with conda, so very few + all of the CDTs

How are shared libraries specific to the Python package/wheel handled w.r.t. 1 & 2?

Yeah, this is a fun one. So I believe that the easiest thing to do is to have a separate copy of Python (obtained via the Python wheel) within site-packages. This works because we provide a wrapper script that points the python command to the proper location within site-packages. Maybe this wrapper script needs some extra environment variables set to point it to the proper site-packages? And maybe the pip package would need to be a dummy. Some of the details here haven't been worked out yet, but that's the basic strategy.

[thewtex]

OK so the basic strategy is that shared libraries go into site-packages/lib.

Really interesting idea! 💡

This works because we provide a wrapper script that points the python command to the proper location within site-packages.

If this experiment proves successful, it would be interesting if CPython could add support at startup for loading libraries, by default, from site-packages/lib to prevent the need for a wrapper script...

[scopatz]

Ultimately, That'd be great!

[SylvainCorlay]

OK so the basic strategy is that shared libraries go into site-packages/lib. This is because the wheel format dictates that all files, by default, go into site-packages. We don't really want to move these around to avoid issues of breaking the links.

It does not seem to be a correct approach to the issue, especially for packages that link with libpython because the relative path of libpython with respect to PREFIX/site-packages/lib is not the same as with respect to PREFIX/lib. Besides, depending on the distutils scheme that is used, site-packages may be in a number of places in the filesystem...

How about making use of data_files for most of the wheel content instead, and only special case the content of Python-specific directories?