Python C++ module not properly repaired, retains dependency on symlink to bundled library #159

madig · 2019-05-04T12:21:50Z

I have a Python C++ module that links to libcrypto from OpenSSL and libcurl. I build the package on the manylinux2010 container (python3 setup.py bdist_wheel) and run auditwheel repair on the .whl. The library shared objects (libcrypto, libz and libcurl) are correctly packed in and ldd-ing the Python module shared object correctly finds and shows them as dependencies, yet it retains a dependency on "libcrypto.so.10", which is named differently on e.g. Debian-based containers, making import fail. What am I missing?

The text was updated successfully, but these errors were encountered:

madig · 2019-05-06T09:56:38Z

Ugh. I can't even run the resulting wheel on the manylinux2010 container, same eror:

ImportError: /tmp/venv/lib/python3.7/site-packages/dsign/.libs/libcrypto-a94305d7.so.1.0.1e: version `libcrypto.so.10' not found (required by /tmp/venv/lib/python3.7/site-packages/dsign/_dsign.cpython-37m-x86_64-linux-gnu.so)

Edit: for completeness, here's the output of show and repair after building the wheel initially:

(venv) [root@a381fbecf6ea dsign]# auditwheel show dist/dsign-1.4.4-cp37-cp37m-linux_x86_64.whl


dsign-1.4.4-cp37-cp37m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0'},
libpthread.so.0 with versions {'GLIBC_2.2.5'}, libc.so.6 with versions
{'GLIBC_2.2.5'}, libcrypto.so.10 with versions {'libcrypto.so.10'},
libstdc++.so.6 with versions {'GLIBCXX_3.4', 'CXXABI_1.3'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib64/libc-2.12.so",
    "libcrypto.so.10": "/usr/lib64/libcrypto.so.1.0.1e",
    "libcurl.so.4": "/usr/local/lib/libcurl.so.4.5.0",
    "libdl.so.2": "/lib64/libdl-2.12.so",
    "libgcc_s.so.1": "/lib64/libgcc_s-4.4.7-20120601.so.1",
    "libm.so.6": "/lib64/libm-2.12.so",
    "libpthread.so.0": "/lib64/libpthread-2.12.so",
    "librt.so.1": "/lib64/librt-2.12.so",
    "libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.13",
    "libz.so.1": "/lib64/libz.so.1.2.3"
}

In order to achieve the tag platform tag "manylinux2010_x86_64" the
following shared library dependencies will need to be eliminated:

libcrypto.so.10, libcurl.so.4, libz.so.1

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libcrypto.so.10, libcurl.so.4, libz.so.1

(venv) [root@a381fbecf6ea dsign]# auditwheel repair dist/dsign-1.4.4-cp37-cp37m-linux_x86_64.whl
INFO:auditwheel.main_repair:Repairing dsign-1.4.4-cp37-cp37m-linux_x86_64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
INFO:auditwheel.wheeltools:New filename tags: manylinux1_x86_64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp37-cp37m-linux_x86_64
INFO:auditwheel.wheeltools:New WHEEL info tags: cp37-cp37m-manylinux1_x86_64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /tmp/dsign/wheelhouse/dsign-1.4.4-cp37-cp37m-manylinux1_x86_64.whl

Here's ldd and auditwheel lddtree (which doesn't complain about the "missing library"):

(venv) [root@a381fbecf6ea dsign]# ldd /tmp/venv/lib/python3.7/site-packages/dsign/_dsign.cpython-37m-x86_64-linux-gnu.so
/tmp/venv/lib/python3.7/site-packages/dsign/_dsign.cpython-37m-x86_64-linux-gnu.so: /tmp/venv/lib/python3.7/site-packages/dsign/.libs/libcrypto-a94305d7.so.1.0.1e: version `libcrypto.so.10'
not found (required by /tmp/venv/lib/python3.7/site-packages/dsign/_dsign.cpython-37m-x86_64-linux-gnu.so)
        linux-vdso.so.1 =>  (0x00007ffd4d7d6000)
        libcurl-a77dbfac.so.4.5.0 => /tmp/venv/lib/python3.7/site-packages/dsign/.libs/libcurl-a77dbfac.so.4.5.0 (0x00007f054eea4000)
        libcrypto-a94305d7.so.1.0.1e => /tmp/venv/lib/python3.7/site-packages/dsign/.libs/libcrypto-a94305d7.so.1.0.1e (0x00007f054ea8e000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f054e788000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f054e504000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f054e2ee000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f054e0d1000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f054dd3d000)
        libz-eb09ad1d.so.1.2.3 => /tmp/venv/lib/python3.7/site-packages/dsign/.libs/libz-eb09ad1d.so.1.2.3 (0x00007f054db26000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f054d91e000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f054d71a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f054f58f000)

(venv) [root@a381fbecf6ea dsign]# auditwheel lddtree /tmp/venv/lib/python3.7/site-packages/dsign/_dsign.cpython-37m-x86_64-linux-gnu.so
INFO:auditwheel.main_lddtree:{
    "interp": null,
    "path": "/tmp/venv/lib/python3.7/site-packages/dsign/_dsign.cpython-37m-x86_64-linux-gnu.so",
    "realpath": "/tmp/venv/lib/python3.7/site-packages/dsign/_dsign.cpython-37m-x86_64-linux-gnu.so",
    "needed": [
        "libcurl-a77dbfac.so.4.5.0",
        "libcrypto-a94305d7.so.1.0.1e",
        "libstdc++.so.6",
        "libm.so.6",
        "libgcc_s.so.1",
        "libpthread.so.0",
        "libc.so.6"
    ],
    "rpath": [
        "/tmp/venv/lib/python3.7/site-packages/dsign/.libs"
    ],
    "runpath": [],
    "libs": {
        "libcurl-a77dbfac.so.4.5.0": {
            "realpath": "/tmp/venv/lib/python3.7/site-packages/dsign/.libs/libcurl-a77dbfac.so.4.5.0",
            "path": "/tmp/venv/lib/python3.7/site-packages/dsign/.libs/libcurl-a77dbfac.so.4.5.0",
            "needed": [
                "libz-eb09ad1d.so.1.2.3",
                "librt.so.1",
                "libdl.so.2",
                "libpthread.so.0",
                "libc.so.6"
            ]
        },
        "libz-eb09ad1d.so.1.2.3": {
            "realpath": "/tmp/venv/lib/python3.7/site-packages/dsign/.libs/libz-eb09ad1d.so.1.2.3",
            "path": "/tmp/venv/lib/python3.7/site-packages/dsign/.libs/libz-eb09ad1d.so.1.2.3",            "needed": [
                "libc.so.6"
            ]
        },
        "libc.so.6": {
            "realpath": "/lib64/libc-2.12.so",
            "path": "/lib64/libc.so.6",
            "needed": [
                "ld-linux-x86-64.so.2"
            ]
        },
        "ld-linux-x86-64.so.2": {
            "realpath": "/lib64/ld-2.12.so",
            "path": "/lib64/ld-linux-x86-64.so.2",
            "needed": []
        },
        "librt.so.1": {
            "realpath": "/lib64/librt-2.12.so",
            "path": "/lib64/librt.so.1",
            "needed": [
                "libc.so.6",
                "libpthread.so.0",
                "ld-linux-x86-64.so.2"
            ]
        },
        "libpthread.so.0": {
            "realpath": "/lib64/libpthread-2.12.so",
            "path": "/lib64/libpthread.so.0",
            "needed": [
                "libc.so.6",
                "ld-linux-x86-64.so.2"
            ]
        },
        "libdl.so.2": {
            "realpath": "/lib64/libdl-2.12.so",
            "path": "/lib64/libdl.so.2",
            "needed": [
                "libc.so.6",
                "ld-linux-x86-64.so.2"
            ]
        },
        "libcrypto-a94305d7.so.1.0.1e": {
            "realpath": "/tmp/venv/lib/python3.7/site-packages/dsign/.libs/libcrypto-a94305d7.so.1.0.1e",
            "path": "/tmp/venv/lib/python3.7/site-packages/dsign/.libs/libcrypto-a94305d7.so.1.0.1e",
            "needed": [
                "libdl.so.2",
                "libz-eb09ad1d.so.1.2.3",
                "libc.so.6"
            ]
        },
        "libstdc++.so.6": {
            "realpath": "/usr/lib64/libstdc++.so.6.0.13",
            "path": "/usr/lib64/libstdc++.so.6",
            "needed": [
                "libm.so.6",
                "libc.so.6",
                "ld-linux-x86-64.so.2",
                "libgcc_s.so.1"
            ]
        },
        "libm.so.6": {
            "realpath": "/lib64/libm-2.12.so",
            "path": "/lib64/libm.so.6",
            "needed": [
                "libc.so.6"
            ]
        },
        "libgcc_s.so.1": {
            "realpath": "/lib64/libgcc_s-4.4.7-20120601.so.1",
            "path": "/lib64/libgcc_s.so.1",
            "needed": [
                "libc.so.6"
            ]
        }
    }
}

madig · 2019-05-06T10:44:18Z

I tried linking the library to libcrypto statically and it works, but:

(venv) [root@a381fbecf6ea dsign]# auditwheel show dist/dsign-1.4.5.dev0+gf7b8133.d20190506-cp37-cp37m-linux_x86_64.whl

dsign-1.4.5.dev0+gf7b8133.d20190506-cp37-cp37m-linux_x86_64.whl is
consistent with the following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0'},
libpthread.so.0 with versions {'GLIBC_2.2.5'}, libc.so.6 with versions
{'GLIBC_2.3.4', 'GLIBC_2.7', 'GLIBC_2.4', 'GLIBC_2.3', 'GLIBC_2.2.5'},
libstdc++.so.6 with versions {'CXXABI_1.3', 'GLIBCXX_3.4'}

This constrains the platform tag to "manylinux2010_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

(venv) [root@a381fbecf6ea dsign]# auditwheel repair dist/dsign-1.4.5.dev0+gf7b8133.d20190506-cp37-cp37m-linux_x86_64.whl
INFO:auditwheel.main_repair:Repairing dsign-1.4.5.dev0+gf7b8133.d20190506-cp37-cp37m-linux_x86_64.whl
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: cannot repair "dist/dsign-1.4.5.dev0+gf7b8133.d20190506-cp37-cp37m-linux_x86_64.whl" to "manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

Wat.

lkollar · 2019-05-08T15:00:13Z

Could you upload the wheel somewhere? If possible the original version before you ran repair on it.

madig · 2019-05-13T07:40:58Z

(Sorry for the late reply, I can't share that wheel but will do a minimal example when I get the time...)

bakatrouble · 2019-05-14T02:07:33Z

@lkollar
I have a "repaired" wheel here: https://files.pythonhosted.org/packages/8a/12/9db1f67e2747c32491edbfaccd00e8c6379dba2589da0ba22ff4ec747c19/pytgvoip-0.0.2.3-cp37-cp37m-manylinux2010_x86_64.whl

Same issue (I hope, even though it comes from side dependency shipped with the wheel):
ImportError: /home/bakatrouble/.virtualenvs/sandbox/lib/python3.7/site-packages/.libs_tgvoip/libcrypto-a94305d7.so.1.0.1e: version `libcrypto.so.10' not found (required by /home/bakatrouble/.virtualenvs/sandbox/lib/python3.7/site-packages/.libs_tgvoip/libtgvoip-d66a5fc7.so.0.0.0)

bakatrouble · 2019-05-14T02:34:54Z

https://drop.bakatrouble.me/pytgvoip.whl/
Here are newly built non-repaired and repaired wheels

lkollar · 2019-05-14T05:58:40Z

Thanks, I’ll try to have a look later this week.

lkollar · 2019-05-15T08:21:48Z

I think this is due to a patchelf bug @njsmith has already reported and fixed in NixOS/patchelf#84 but hasn't been released. patchelf does not update the .gnu_version_r section with the vendored library so the linker rejects our .so as it thinks it's not the correct version:

386: /tmp/voip/lib/python3.7/site-packages/.libs_tgvoip/libcrypto-a94305d7.so.1.0.1e: error: version lookup error: version `libcrypto.so.10' not found (required by /tmp/voip/lib/python3.7/site-packages/.libs_tgvoip/libtgvoip-d66a5fc7.so.0.0.0) (fatal)

Relevant .gnu_version_r section in .libs_tgvoip/libtgvoip-d66a5fc7.so.0.0.0:

0x0030: Version: 1  File: libcrypto-a94305d7.so.1.0.1e  Cnt: 1
 0x0040:   Name: libcrypto.so.10  Flags: none  Version: 8

I'll try to confirm with the patched patchelf version.

lkollar · 2019-05-15T08:23:29Z

Small correction: there is a patchelf release which should contain the fix: 0.10. I had 0.9 installed while I was testing this. Will check with 0.10.

lkollar · 2019-05-15T19:57:14Z

The version information appears to be correct after installing patchelf 0.10 and running auditwheel repair. I can't produce a libtgvoip.so version which would have matching symbols needed by the wheel attached by @bakatrouble though. Can you try to repair the wheel again with the newer patchelf?

I think we should add a version check on patchelf 0.10 and display a warning if users attempt to repair wheels with lower versions.

bakatrouble · 2019-05-15T23:16:15Z

Same error when importing extension (version ``libcrypto.so.10' not found)
Added v0.0.2.7 wheels at the same location: https://drop.bakatrouble.me/pytgvoip.whl/

If it'll help, Dockerfile used for building is here.
Run using docker run -v ${PWD}/dist:/dist $IMAGEID cp37-cp37m, source and repaired wheels will output to the dist directory

bakatrouble · 2019-05-15T23:33:40Z

Here's the output of auditwheel lddtree for the extension .so and libtgvoip.so.0.0.0

lkollar · 2019-05-16T08:17:39Z

Thanks for the Dockerfile. I can confirm that even with patchelf 0.10 the resulting .so file does not appear to have the correct version information:

  0x0030: Version: 1  File: libcrypto-a94305d7.so.1.0.1e  Cnt: 1
  0x0040:   Name: libcrypto.so.10  Flags: none  Version: 8

This needs more investigation.

lkollar · 2019-05-28T21:26:43Z

The above version information is actually correct. This particular version oflibcrypto.so uses the libcrypto.so.10 version string on its exported symbols. The .gnu.version_d section in the original .so displays the correct version strings:

Version definition section '.gnu.version_d' contains 4 entries:
  Addr: 0x00000000000371d8  Offset: 0x0371d8  Link: 4 (.dynstr)
  000000: Rev: 1  Flags: BASE  Index: 1  Cnt: 1  Name: libcrypto.so.10
  0x0014: Rev: 1  Flags: none  Index: 2  Cnt: 1  Name: libcrypto.so.10
  0x0030: Rev: 1  Flags: none  Index: 3  Cnt: 1  Name: OPENSSL_1.0.1
  0x004c: Rev: 1  Flags: none  Index: 4  Cnt: 1  Name: OPENSSL_1.0.1_EC

However, the vendored .so has incorrect version information in this section:

Version definition section '.gnu.version_d' contains 4 entries:
  Addr: 0x00000000000371d8  Offset: 0x0371d8  Link: 27 (.dynstr)
  000000: Rev: 1  Flags: BASE  Index: 1  Cnt: 1  Name: XXXXXXXXXXXXXXX
  0x0014: Rev: 1  Flags: none  Index: 2  Cnt: 1  Name: XXXXXXXXXXXXXXX
  0x0030: Rev: 1  Flags: none  Index: 3  Cnt: 1  Name: OPENSSL_1.0.1
  0x004c: Rev: 1  Flags: none  Index: 4  Cnt: 1  Name: OPENSSL_1.0.1_E

This happens because patchelf replaces the original DT_SONAME entry in .dynstr with a "XXXXXXXXXXXXXXX" string. Since the .gnu.version_r section in libcrypto.so still points to the index of the original entry in .dynstr now the required version is "XXXXXXXXXXXXXXX".

Version strings in the dnyamic string table:

readelf -p .dynstr libcrypto-a94305d7.so.1.0.1e

  [ 1327f]  XXXXXXXXXXXXXXX
  [ 1328f]  OPENSSL_1.0.1
  [ 1329d]  OPENSSL_1.0.1_EC
  [ 132ae]  GLIBC_2.2.5
  [ 132ba]  GLIBC_2.4
  [ 132c4]  GLIBC_2.3
  [ 132ce]  GLIBC_2.7
  [ 132d8]  GLIBC_2.3.4
  [ 132e4]  libcrypto-a94305d7.so.1.0.1e
  [ 13301]  libz-eb09ad1d.so.1.2.3

The new SONAME is added to the end as a new value and the original version is "zeroed out". Unofortunately in the case of libcrypto.so, this string is used as the exported version as well.

The original .dynstr section for reference:

readelf -p .dynstr /usr/lib64/libcrypto.so

  [ 1327f]  libcrypto.so.10
  [ 1328f]  OPENSSL_1.0.1
  [ 1329d]  OPENSSL_1.0.1_EC
  [ 132ae]  GLIBC_2.2.5
  [ 132ba]  GLIBC_2.4
  [ 132c4]  GLIBC_2.3
  [ 132ce]  GLIBC_2.7
  [ 132d8]  GLIBC_2.3.4

The code to zero out the original SONAME in patchelf is here: https://github.com/NixOS/patchelf/blob/1c95784c028d214c475d4f34ea6c3faabb5765ce/src/patchelf.cc#L1077-L1082. I will file an issue to better understand why this is necessary and maybe remove it. I don't see other options at the moment to work around this issue.

lkollar · 2019-05-28T21:28:42Z

This has already been reported in patchelf: NixOS/patchelf#155.

Removes the code mentioned here: pypa/auditwheel#159 (comment)

Split Linux Python packages to avoid timeout issues. Remove ITKPythonGitTag from .yml variable. It should be defined in the pipeline variables section when configuring a new pipeline. Use ITKv5.0.1 in setup.py until ITKPythonBuilds for v5.1.0 are ready. Setup self hosted pipeline to build with CUDA. Building on manylinux image requires CUDA_NVCC_FLAGS="-std=c++11". WARNING: CUDA packages for Linux build successfully but fail at runtime because of a known issue with auditwheel. See the following for reference: pypa/auditwheel#159 The docker image used to build packages should be updated with a custom version of patchelf to work around the issue.

jfolz · 2020-01-31T08:49:45Z

Has there been any movement on this? As far as I can tell the only way to publish wheels that require, e.g. CUDA, is to lie about them being manylinux.

lkollar · 2020-02-04T09:20:53Z

There has been no movement on applying the fix in patchelf. We could apply the patch in the version of patchelf we ship with the manylinux images as a workaround though.

jfolz · 2020-02-07T09:34:23Z

That might be necessary, though the current situation is overall undesirable.
I ran into strange issues recently that I tracked down to Pytorch from PyPI loading its own version of the CUDA runtime, which was different from the system version. My extension was built again the system CUDA and they were interfering with each other. Basically, my code would change some global state (probably the CUDA context) that confused Pytorch and it crashed profusely.

jfolz · 2020-02-07T09:39:58Z

What I mean by this is: I would very much like to publish wheels to PyPI, but I doubt it would bring any good. Since my package is meant to interact closely with Pytorch and other similar frameworks, there are bound to be issues where users update either package, resulting in such a version mismatch with dependencies. So, I don't know if publishing manylinux wheels is useful at all without users knowing exactly what's going on in the background.

jfolz · 2020-04-28T15:18:53Z

Can confirm that pypa/manylinux#545 fixes auditwheel with CUDA libraries. Thanks a bunch!

bakatrouble mentioned this issue May 15, 2019

please help with version `libcrypto.so.10' not found bakatrouble/pytgvoip#1

Closed

lkollar mentioned this issue May 28, 2019

--set-soname corrupts binary if the original soname string is also used for symbol versioning NixOS/patchelf#155

Closed

lkollar mentioned this issue Jun 26, 2019

Fix running 'wheelhouse' test script on Travis CI #168

Closed

bakatrouble mentioned this issue Jul 1, 2019

Can't install bakatrouble/pytgvoip_pyrogram#3

Open

lkollar mentioned this issue Aug 7, 2019

Find a replacement for patchelf #174

Closed

nvictus added a commit to nvictus/patchelf that referenced this issue Nov 25, 2019

Patch patchelf for auditwheel

d7483d9

Removes the code mentioned here: pypa/auditwheel#159 (comment)

dralley mentioned this issue Feb 15, 2020

Provide more insight into auditwheel repair failures? RalfG/python-wheels-manylinux-build#6

Closed

IaroslavR mentioned this issue Apr 3, 2020

Auditwheel for manylinux2014 struggles with libcrypto / libssl #217

Closed

lkollar mentioned this issue Apr 18, 2020

Apply patch in patchelf to avoid zeroing SONAME pypa/manylinux#545

Merged

lkollar closed this as completed in pypa/manylinux#545 Apr 24, 2020

IaroslavR mentioned this issue Apr 26, 2020

set up a build system for pytype binary wheels google/pytype#562

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python C++ module not properly repaired, retains dependency on symlink to bundled library #159

Python C++ module not properly repaired, retains dependency on symlink to bundled library #159

madig commented May 4, 2019

madig commented May 6, 2019 •

edited

Loading

madig commented May 6, 2019

lkollar commented May 8, 2019

madig commented May 13, 2019

bakatrouble commented May 14, 2019 •

edited

Loading

bakatrouble commented May 14, 2019

lkollar commented May 14, 2019

lkollar commented May 15, 2019

lkollar commented May 15, 2019

lkollar commented May 15, 2019

bakatrouble commented May 15, 2019 •

edited

Loading

bakatrouble commented May 15, 2019

lkollar commented May 16, 2019

lkollar commented May 28, 2019

lkollar commented May 28, 2019

jfolz commented Jan 31, 2020

lkollar commented Feb 4, 2020

jfolz commented Feb 7, 2020

jfolz commented Feb 7, 2020

jfolz commented Apr 28, 2020

Python C++ module not properly repaired, retains dependency on symlink to bundled library #159

Python C++ module not properly repaired, retains dependency on symlink to bundled library #159

Comments

madig commented May 4, 2019

madig commented May 6, 2019 • edited Loading

madig commented May 6, 2019

lkollar commented May 8, 2019

madig commented May 13, 2019

bakatrouble commented May 14, 2019 • edited Loading

bakatrouble commented May 14, 2019

lkollar commented May 14, 2019

lkollar commented May 15, 2019

lkollar commented May 15, 2019

lkollar commented May 15, 2019

bakatrouble commented May 15, 2019 • edited Loading

bakatrouble commented May 15, 2019

lkollar commented May 16, 2019

lkollar commented May 28, 2019

lkollar commented May 28, 2019

jfolz commented Jan 31, 2020

lkollar commented Feb 4, 2020

jfolz commented Feb 7, 2020

jfolz commented Feb 7, 2020

jfolz commented Apr 28, 2020

madig commented May 6, 2019 •

edited

Loading

bakatrouble commented May 14, 2019 •

edited

Loading

bakatrouble commented May 15, 2019 •

edited

Loading