Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide manylinux1 wheels on PyPI #5033

Closed
vmarkovtsev opened this Issue Oct 18, 2016 · 12 comments

Comments

Projects
None yet
5 participants
@vmarkovtsev
Copy link
Contributor

commented Oct 18, 2016

Context: what is manylinux1 tag.

Currently, Tensorflow wheels on PyPI suck. As many know, we cannot upload specific Linux wheels on PyPI because the cheese shop prohibits different flavors (though Windows and macOS are OK). However, one can upload universal Linux wheels aka manylinux1 (see the link from above). There is a tool which helps making manylinux1 wheels called auditwheel.

auditwheel show tensorflow-0.11.0rc0-cp35-cp35m-linux_x86_64.whl 

tensorflow-0.11.0rc0-cp35-cp35m-linux_x86_64.whl is consistent with
the following platform tag: "manylinux1_x86_64".

The wheel references no external versioned symbols from system-
provided shared libraries.

The wheel requires no external shared libraries! :)

This basically means that we can execute auditwheel repair and obtain the manylinux1 wheel.

Thus I propose to upload CPU-only wheels for Linux for Python 2/3 to PyPI with manylinux1 tag.

@asimshankar

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2016

@martinwicke has a better understanding of PyPi and wheels than I do, perhaps he can comment or redirect to a more appropriate person.

@martinwicke

This comment has been minimized.

Copy link
Member

commented Oct 29, 2016

I did not know about this tool. @yifeif can you look into this one and see if this is something that would work for us? It may actually make the pypi story simpler.

@yifeif

This comment has been minimized.

Copy link
Member

commented Oct 29, 2016

Will look into this.

@vmarkovtsev

This comment has been minimized.

Copy link
Contributor Author

commented Nov 22, 2016

So?..

@yifeif

This comment has been minimized.

Copy link
Member

commented Nov 29, 2016

This is now available with 0.12rc0 release!

@yifeif yifeif closed this Nov 29, 2016

@jjhelmus

This comment has been minimized.

Copy link

commented Nov 29, 2016

The Linux 0.12rc0 whl files available on PyPI do not conform to the manylinux1 specification provided in PEP531. Specifically, they contain shared libraries which reference versioned symbols which do not meet the GLIBC <= 2.5 and GLIBCXX <= 3.4.9 requirements specified in the PEP.

$ auditwheel show tensorflow-0.12.0rc0-cp35-cp35m-manylinux1_x86_64.whl 

tensorflow-0.12.0rc0-cp35-cp35m-manylinux1_x86_64.whl is consistent
with the following platform tag: "manylinux1_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.16',
'GLIBC_2.9', 'GLIBC_2.17', 'GLIBC_2.3.4', 'GLIBC_2.10', 'GLIBC_2.3',
'GLIBC_2.2.5', 'GLIBC_2.4', 'GLIBC_2.14', 'GLIBC_2.11', 'GLIBC_2.6',
'GLIBC_2.7', 'GLIBC_2.3.2'}, libstdc++.so.6 with versions
{'GLIBCXX_3.4.9', 'GLIBCXX_3.4.11', 'CXXABI_1.3', 'GLIBCXX_3.4',
'CXXABI_1.3.5', 'GLIBCXX_3.4.14', 'GLIBCXX_3.4.19', 'GLIBCXX_3.4.18',
'GLIBCXX_3.4.15'}, libdl.so.2 with versions {'GLIBC_2.2.5'},
libpthread.so.0 with versions {'GLIBC_2.3.2', 'GLIBC_2.2.5'},
libm.so.6 with versions {'GLIBC_2.2.5'}, libgcc_s.so.1 with versions
{'GCC_3.0'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.23.so",
    "libdl.so.2": "/lib/x86_64-linux-gnu/libdl-2.23.so",
    "libgcc_s.so.1": "/lib/x86_64-linux-gnu/libgcc_s.so.1",
    "libm.so.6": "/lib/x86_64-linux-gnu/libm-2.23.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.23.so",
    "libstdc++.so.6": "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21"
}

The pypa/manylinux repository includes Docker images which can be used to build manylinux1 compatible wheels.

@vmarkovtsev

This comment has been minimized.

Copy link
Contributor Author

commented Nov 29, 2016

@yifeif I guess auditwheel repair should be enough to fix this. Just curious, how do you make wheels currently?

@jjhelmus

This comment has been minimized.

Copy link

commented Nov 29, 2016

auditwheel repair cannot correct versioned symbol outside of the spec. The command only fixes wheel files which are missing external shared libraries.

@vmarkovtsev

This comment has been minimized.

Copy link
Contributor Author

commented Nov 29, 2016

Mmm, building TF on CentOS 5 looks like a joke. Besides, nvidia/cuda:8.0 is already used, it is still Ubuntu 14.04 - the perfect ABI balance IMHO. I would explicitly write in the docs that TF's "manylinux1" is actually not manylinux1 in the strict definition and it should be enough.

@yifeif

This comment has been minimized.

Copy link
Member

commented Nov 29, 2016

Thanks for bring this up guys. We do run our whl files through auditwheel repair, but as @jjhelmus mentioned, auditwheel does not fix symbols when they are from a too-recent version. @vmarkovtsev I will add a note on pypi for next release.

@jjhelmus

This comment has been minimized.

Copy link

commented Nov 29, 2016

@yifeif Thanks for update and I think the note sounds like the best solution.

Any chance you could share a bit about how the released whl files are built? I help maintain the conda-forge tensorflow package and we have been running into some issues on Linux. We uses CentOS 6 as our Linux target which uses an older GLIBC than the whl files support. I've tried to build tensorflow from source using our build setup but this has proved challenging. Any hints you could provide would be most appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.