Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When a single wheel contains multiple top-level packages, auditwheel creates separate .libs dirs for each of them #89

Closed
njsmith opened this issue Feb 7, 2018 · 4 comments · Fixed by #90
Closed

When a single wheel contains multiple top-level packages, auditwheel creates separate .libs dirs for each of them #89

njsmith opened this issue Feb 7, 2018 · 4 comments · Fixed by #90
Labels
enhancement has PR

Comments

@njsmith
Copy link
Member

njsmith commented Feb 7, 2018

Bug originally filed by @lpsinger at pypa/manylinux#156:

If you have a package that contains multiple C extensions that depend on the same shared libraries, then auditwheel repair saves duplicate copies of all of the shared libraries. An example is shown below.

This makes for packages that are several times larger than necessary. Could auditwheel repair sweep up all of the dependencies and put them in a single .libs directory, rather than creating duplicates?

Example

$ tar -tf lalsuite-6.48-cp27-cp27m-manylinux1_x86_64.whl 
lalxml/git_version.py
lalxml/_lalxml.so
lalxml/__init__.py
lalxml/lalxml.py
lalxml/.libs/libgsl-b885fd68.so.0.14.0
lalxml/.libs/libhdf5_hl-04af5206.so.100.0.1
lalxml/.libs/libhdf5-db4e48f9.so.101.0.0
lalxml/.libs/liblalsupport-bd9338fc.so.11.0.0
lalxml/.libs/liblalxml-a864f854.so.2.0.1
lalxml/.libs/libxml2-8ed442f7.so.2.6.26
lalxml/.libs/libz-a147dcb0.so.1.2.3
lalxml/.libs/libfftw3-c8dcb815.so.3.2.4
lalxml/.libs/liblal-8b0d5843.so.14.0.0
lalxml/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalxml/.libs/libfftw3f-044fdcc5.so.3.2.4
lalmetaio/git_version.py
lalmetaio/lalmetaio.py
lalmetaio/_lalmetaio.so
lalmetaio/__init__.py
lalmetaio/.libs/libgsl-b885fd68.so.0.14.0
lalmetaio/.libs/libhdf5_hl-04af5206.so.100.0.1
lalmetaio/.libs/libhdf5-db4e48f9.so.101.0.0
lalmetaio/.libs/liblalsupport-bd9338fc.so.11.0.0
lalmetaio/.libs/liblalmetaio-f4e58690.so.5.0.1
lalmetaio/.libs/libz-a147dcb0.so.1.2.3
lalmetaio/.libs/libfftw3-c8dcb815.so.3.2.4
lalmetaio/.libs/liblal-8b0d5843.so.14.0.0
lalmetaio/.libs/libmetaio-c02fec4d.so.1.1.0
lalmetaio/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalmetaio/.libs/libfftw3f-044fdcc5.so.3.2.4
lal/_lal.so
lal/gpstime.py
lal/lal.py
lal/git_version.py
lal/__init__.py
lal/rate.py
lal/series.py
lal/.libs/libgsl-b885fd68.so.0.14.0
lal/.libs/libhdf5_hl-04af5206.so.100.0.1
lal/.libs/libhdf5-db4e48f9.so.101.0.0
lal/.libs/liblalsupport-bd9338fc.so.11.0.0
lal/.libs/libz-a147dcb0.so.1.2.3
lal/.libs/libfftw3-c8dcb815.so.3.2.4
lal/.libs/liblal-8b0d5843.so.14.0.0
lal/.libs/libgslcblas-15b2d0fe.so.0.0.0
lal/.libs/libfftw3f-044fdcc5.so.3.2.4
lal/utils/cache.py
lal/utils/__init__.py
lal/utils/series.py
lal/spectrum/distributions.py
lal/spectrum/averagespectrum.py
lal/spectrum/__init__.py
lalsuite-6.48.dist-info/WHEEL
lalsuite-6.48.dist-info/METADATA
lalsuite-6.48.dist-info/DESCRIPTION.rst
lalsuite-6.48.dist-info/metadata.json
lalsuite-6.48.dist-info/RECORD
lalsuite-6.48.dist-info/top_level.txt
lalsimulation/git_version.py
lalsimulation/__init__.py
lalsimulation/_lalsimulation.so
lalsimulation/lalsimulation.py
lalsimulation/.libs/libgsl-b885fd68.so.0.14.0
lalsimulation/.libs/libhdf5_hl-04af5206.so.100.0.1
lalsimulation/.libs/liblalsimulation-f35d2299.so.16.0.3
lalsimulation/.libs/libhdf5-db4e48f9.so.101.0.0
lalsimulation/.libs/liblalsupport-bd9338fc.so.11.0.0
lalsimulation/.libs/libz-a147dcb0.so.1.2.3
lalsimulation/.libs/libfftw3-c8dcb815.so.3.2.4
lalsimulation/.libs/liblal-8b0d5843.so.14.0.0
lalsimulation/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalsimulation/.libs/libfftw3f-044fdcc5.so.3.2.4
lalframe/git_version.py
lalframe/frread.py
lalframe/__init__.py
lalframe/lalframe.py
lalframe/_lalframe.so
lalframe/.libs/libgsl-b885fd68.so.0.14.0
lalframe/.libs/libhdf5_hl-04af5206.so.100.0.1
lalframe/.libs/libhdf5-db4e48f9.so.101.0.0
lalframe/.libs/liblalsupport-bd9338fc.so.11.0.0
lalframe/.libs/libFrame-0c783110.so.1.5.0
lalframe/.libs/liblalframe-2600de1c.so.10.0.2
lalframe/.libs/libz-a147dcb0.so.1.2.3
lalframe/.libs/libfftw3-c8dcb815.so.3.2.4
lalframe/.libs/liblal-8b0d5843.so.14.0.0
lalframe/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalframe/.libs/libfftw3f-044fdcc5.so.3.2.4
lalframe/utils/frtools.py
lalframe/utils/__init__.py
@njsmith njsmith mentioned this issue Feb 7, 2018
Closed
@njsmith
Copy link
Member Author

njsmith commented Feb 7, 2018

Maybe we should put the shared libs in $WHEELNAME.libs, next to the $WHEELNAME-$VERSION.dist-info directory? Do wheels even let us do that?

Or we could arbitrarily pick one of the subpackages to contain the .libs directory, and point everything at it.

@lpsinger
Copy link
Contributor

lpsinger commented Feb 7, 2018

Maybe we should put the shared libs in $WHEELNAME.libs, next to the $WHEELNAME-$VERSION.dist-info directory? Do wheels even let us do that?

If it was called ${WHEELNAME}_libs then it could go into platlib like any other module.

@njsmith
Copy link
Member Author

njsmith commented Feb 7, 2018

Yes, the advantage to using a . in the name is exactly that it guarantees we won't accidentally collide with a real python package :-). (Also, in python 3, creating ${WHEELNAME}_libs would actually make import ${WHEELNAME}_libs work, because any directory on the PYTHONPATH is automatically a package, even if it doesn't have a __init__.py.)

@lpsinger
Copy link
Contributor

lpsinger commented Feb 7, 2018

Hmm, well, I guess the only way to find out is to try. How would one get the name of the wheel inside repair_wheel?

@lpsinger lpsinger mentioned this issue Feb 7, 2018
Merged
@matthew-brett matthew-brett mentioned this issue Feb 7, 2018
Closed
lpsinger added a commit to lpsinger/delocate that referenced this issue Feb 7, 2018
@lpsinger
Install one copy of libs used by multiple extension modules
e69a946 
If you have a package that contains multiple C extensions that
depend on the same shared libraries, then `delocate-wheel` saves
duplicate copies of all of the shared libraries. An example is
shown below.

This makes for packages that are several times larger than
necessary. This patch causes `delocate-wheel` to sweep up all
of the dependencies and put them in a single directory, rather
than creating duplicates.

The convention is based on the proposal in pypa/auditwheel#89
and pypa/auditwheel#90.
lpsinger added a commit to lpsinger/delocate that referenced this issue Feb 7, 2018
@lpsinger
Install one copy of libs used by multiple extension modules
8bb70dc 
If you have a package that contains multiple C extensions that
depend on the same shared libraries, then `delocate-wheel` saves
duplicate copies of all of the shared libraries.

This makes for packages that are several times larger than
necessary. This patch causes `delocate-wheel` to sweep up all
of the dependencies and put them in a single directory, rather
than creating duplicates.

The convention is based on the proposal in pypa/auditwheel#89
and pypa/auditwheel#90.
@lpsinger lpsinger mentioned this issue Feb 7, 2018
Closed
lpsinger added a commit to lpsinger/delocate that referenced this issue Oct 2, 2018
@lpsinger
Install one copy of libs used by multiple extension modules
80433a9 
If you have a package that contains multiple C extensions that
depend on the same shared libraries, then `delocate-wheel` saves
duplicate copies of all of the shared libraries.

This makes for packages that are several times larger than
necessary. This patch causes `delocate-wheel` to sweep up all
of the dependencies and put them in a single directory, rather
than creating duplicates.

The convention is based on the proposal in pypa/auditwheel#89
and pypa/auditwheel#90.
@daa daa mentioned this issue Dec 12, 2018
Closed
lpsinger added a commit to lpsinger/delocate that referenced this issue Feb 7, 2019
@lpsinger
Install one copy of libs used by multiple extension modules
5193592 
If you have a package that contains multiple C extensions that
depend on the same shared libraries, then `delocate-wheel` saves
duplicate copies of all of the shared libraries.

This makes for packages that are several times larger than
necessary. This patch causes `delocate-wheel` to sweep up all
of the dependencies and put them in a single directory, rather
than creating duplicates.

The convention is based on the proposal in pypa/auditwheel#89
and pypa/auditwheel#90.
lpsinger added a commit to lpsinger/delocate that referenced this issue Feb 7, 2019
@lpsinger
Install one copy of libs used by multiple extension modules
fcef653 
If you have a package that contains multiple C extensions that
depend on the same shared libraries, then `delocate-wheel` saves
duplicate copies of all of the shared libraries.

This makes for packages that are several times larger than
necessary. This patch causes `delocate-wheel` to sweep up all
of the dependencies and put them in a single directory, rather
than creating duplicates.

The convention is based on the proposal in pypa/auditwheel#89
and pypa/auditwheel#90.
lpsinger added a commit to lpsinger/delocate that referenced this issue Feb 25, 2019
@lpsinger
Install one copy of libs used by multiple extension modules
bc0a8c5 
If you have a package that contains multiple C extensions that
depend on the same shared libraries, then `delocate-wheel` saves
duplicate copies of all of the shared libraries.

This makes for packages that are several times larger than
necessary. This patch causes `delocate-wheel` to sweep up all
of the dependencies and put them in a single directory, rather
than creating duplicates.

The convention is based on the proposal in pypa/auditwheel#89
and pypa/auditwheel#90.
@mayeut mayeut mentioned this issue Sep 29, 2019
Closed
@PhilipDeegan PhilipDeegan mentioned this issue Sep 29, 2019
Open
lkollar pushed a commit that referenced this issue Jan 23, 2020
@lpsinger @lkollar
Put libraries in $WHEELNAME.libs (#90)
f0b52b3 
Fixes #89.
@mattip mattip mentioned this issue Jun 21, 2020
Closed
@mattip mattip mentioned this issue Jul 25, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement has PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Put libraries in $WHEELNAME.libs
3 participants
@njsmith @lpsinger @ehashman