Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use `--no-enabled-shared` for Linux wheel-builder interpreter so that the released PEX works with statically built interpreters #7794

Merged

Conversation

Projects
None yet
2 participants
@Eric-Arellano
Copy link
Contributor

commented May 23, 2019

Problem

A downstream user (Twitter) is having issues consuming the Python 3 PEX because it is complaining ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory. This is because they build a Python interpreter from source and do so statistically, meaning that they do not use --enable-shared when compiling so cannot use shared libraries like libpython3.6.m.so.1.0.

Specifically, the issue boils down to several of our wheels not being manylinux1 compatible, which means that they use native libraries outside of the accepted set for compatibility across Linux environments. PEP 513 states that it is a manylinux1 wheel cannot rely on libpython.X.Y.so.1, as several of ours are doing, because statically built interpreters will fail to work.

Ideally users will be able to use our PEX regardless of whether their interpreter is built statically or not.

Solution

No longer use --enable-shared for our build, which means that our wheels will not be able to depend on shared libraries, so downstream users will no longer be hit by this issue.

@Eric-Arellano Eric-Arellano changed the title WIP: Build Linux wheels with `--no-enabled-shared` so that the released PEX works with statically built interpreters WIP: Build Linux wheel-builder Python with `--no-enabled-shared` so that the released PEX works with statically built interpreters May 23, 2019

@Eric-Arellano Eric-Arellano changed the title WIP: Build Linux wheel-builder Python with `--no-enabled-shared` so that the released PEX works with statically built interpreters WIP: Use `--no-enabled-shared` for Linux wheel-builder interpreter so that the released PEX works with statically built interpreters May 23, 2019

@Eric-Arellano Eric-Arellano force-pushed the Eric-Arellano:fix-libpython-so-issues branch from 97ece24 to 05ebdd1 Jun 2, 2019

Eric-Arellano added some commits Jun 2, 2019

Revert using $PY for bootstrap_rust
Issue is probably the Travis cache...
Revert set -x
Think the issue was Travis caches, and either way we found the problematic line.
Try ci.sh -x, which runs clean.sh
Danny pointed out that this is how he got Python 2.7 shard working when it complained about libpython missing. Maybe this will get it working.
@stuhood

stuhood approved these changes Jun 3, 2019

Copy link
Member

left a comment

Thanks @Eric-Arellano!

Shipping this one, although it sounds like a separate commit to get the docker image changes published will be necessary first.

No longer inline centos6 image
I think CI will still pass without this. Once master gets built, then the image will update automatically.

@Eric-Arellano Eric-Arellano changed the title WIP: Use `--no-enabled-shared` for Linux wheel-builder interpreter so that the released PEX works with statically built interpreters Use `--no-enabled-shared` for Linux wheel-builder interpreter so that the released PEX works with statically built interpreters Jun 3, 2019

@Eric-Arellano Eric-Arellano added this to the 1.16.x milestone Jun 3, 2019

@Eric-Arellano Eric-Arellano merged commit cf95c0a into pantsbuild:master Jun 3, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:fix-libpython-so-issues branch Jun 3, 2019

Eric-Arellano added a commit that referenced this pull request Jun 3, 2019

Use `--no-enabled-shared` for Linux wheel-builder interpreter so that…
… the released PEX works with statically built interpreters (#7794)

### Problem
A downstream user (Twitter) is having issues consuming the Python 3 PEX because it is complaining `ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory`. This is because they build a Python interpreter from source and do so statistically, meaning that they do not use `--enable-shared` when compiling so cannot use shared libraries like `libpython3.6.m.so.1.0`.

Specifically, the issue boils down to several of our wheels not being `manylinux1` compatible, which means that they use native libraries outside of the accepted set for compatibility across Linux environments. [PEP 513](https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1) states that it is a `manylinux1` wheel cannot rely on `libpython.X.Y.so.1`, as several of ours are doing, because statically built interpreters will fail to work.

Ideally users will be able to use our PEX regardless of whether their interpreter is built statically or not.

### Solution
No longer use `--enable-shared` for our build, which means that our wheels will not be able to depend on shared libraries, so downstream users will no longer be hit by this issue.

@Eric-Arellano Eric-Arellano modified the milestones: 1.16.x, 1.15.x, 1.17.x Jun 4, 2019

Eric-Arellano added a commit that referenced this pull request Jun 4, 2019

Use `--no-enabled-shared` for Linux wheel-builder interpreter so that…
… the released PEX works with statically built interpreters (#7794)

### Problem
A downstream user (Twitter) is having issues consuming the Python 3 PEX because it is complaining `ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory`. This is because they build a Python interpreter from source and do so statistically, meaning that they do not use `--enable-shared` when compiling so cannot use shared libraries like `libpython3.6.m.so.1.0`.

Specifically, the issue boils down to several of our wheels not being `manylinux1` compatible, which means that they use native libraries outside of the accepted set for compatibility across Linux environments. [PEP 513](https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1) states that it is a `manylinux1` wheel cannot rely on `libpython.X.Y.so.1`, as several of ours are doing, because statically built interpreters will fail to work.

Ideally users will be able to use our PEX regardless of whether their interpreter is built statically or not.

### Solution
No longer use `--enable-shared` for our build, which means that our wheels will not be able to depend on shared libraries, so downstream users will no longer be hit by this issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.