Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for manylinux2010 rollout #179

Closed
ncoghlan opened this issue Apr 14, 2018 · 146 comments
Closed

Tracking issue for manylinux2010 rollout #179

ncoghlan opened this issue Apr 14, 2018 · 146 comments

Comments

@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented Apr 14, 2018

I've accepted the manylinux2010 PEP, so that's now an active interoperability standard: https://www.python.org/dev/peps/pep-0571/

However, there are a number of further steps needed before folks can actually make use of that new baseline, and the order of operations matters (since building manylinux2010 wheels isn't very useful if installers won't install them):

Essential client support:

  • Update pip 19.0 to install manylinux2010 wheels (PR: pypa/pip#5410)
  • Update pip 9.x to install manylinux2010 wheels (needed to avoid the manylinux2010 rollout being delayed by the Python API break between pip 9.x and pip 10.x) (@ncoghlan: skip for now, wait and see if we get complaints about needing this)
  • Update pipenv to install manylinux2010 wheels (needed due to pipenv vendoring pip, so the support depends on the version of pipenv being used, rather than the version of pip in the host or target environments)

Enable publication of manylinux2010 wheel archives:

Management of transition from manylinux1:

Additional projects to consider once core capability support has rolled out:

Tagging @takluyver @markrwilliams as the PEP 571 authors, and @di and @brainwane from the PyPI/PSF side of things.

@ncoghlan ncoghlan mentioned this issue Apr 14, 2018
@takluyver
Copy link
Member

@takluyver takluyver commented Apr 15, 2018

Updating the spec page on PyPUG:

pypa/packaging.python.org#470

mayeut added a commit to mayeut/warehouse that referenced this issue May 12, 2018
manylinux2010 platform tag is official per PEP571

see also pypa/manylinux#179
@mayeut
Copy link
Contributor

@mayeut mayeut commented May 12, 2018

Confirm that twine will upload manylinux2010 wheel archives

Confirmed

Update warehouse to accept manylinux2010 wheel uploads

PR: pypa/warehouse#3951

di added a commit to pypa/warehouse that referenced this issue May 14, 2018
manylinux2010 platform tag is official per PEP571

see also pypa/manylinux#179
@di
Copy link
Member

@di di commented May 14, 2018

pypa/warehouse#3951 has been merged.

@wtolson
Copy link

@wtolson wtolson commented May 15, 2018

pip merge request has been updated here: pypa/pip#5410

@wtolson
Copy link

@wtolson wtolson commented May 15, 2018

auditwheel merge request has been updated here: pypa/auditwheel#92

@armudgal
Copy link
Contributor

@armudgal armudgal commented May 17, 2018

Hi,
First of all great job with this, I just wanted to know is there an estimate when the manylinux2010 will roll-out. Thanks :)

@takluyver
Copy link
Member

@takluyver takluyver commented May 17, 2018

There's no one roll-out date, because support needs to be added to various different tools in the ecosystem (see the list at the top of the issue). I imagine that it's probably a matter of a few weeks to a few months before you can use it and expect it to work.

@armudgal
Copy link
Contributor

@armudgal armudgal commented May 20, 2018

Oh, thanks for the information

@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 23, 2018

@ncoghlan Pipenv will have pip 10 support after we merge pypa/pipenv#2255 which un-vendors pip9 completely, and bundles a patched version of pip10 which can be re-vendored and patched when updates land (this was possible because pip-tools accepted my PR which makes them pip10 compatible as well)

so whenever the pr lands in pip we can make the call about whether to actually go ahead and pull that into our patched version before it's released (technically we could do this now, even) and we will be set

@westurner
Copy link

@westurner westurner commented Sep 17, 2018

"SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register"
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/4BGE226DB5EWIAT5VCJ75QD5ASOVJZCM/

@safijari
Copy link

@safijari safijari commented Oct 13, 2018

Other than the open pull requests needing to get merged, is there any more left to do for this effort? How can I help?

@ehashman
Copy link
Member

@ehashman ehashman commented Nov 18, 2018

I have merged support for manylinux2010 in auditwheel. Will be cutting a release candidate today.

Note that auditwheel will default to assigning the strictest possible tag, hence it is possible for a binary built inside the manylinux2010 image to receive a manylinux1 tag given that the wheel's symbols match the stricter manylinux1 policy.

@takluyver
Copy link
Member

@takluyver takluyver commented Oct 7, 2019

Thanks, I've ticked it.

@di
Copy link
Member

@di di commented Oct 12, 2019

The last remaining item here is:

Nominate a timeline and/or "percentage of PyPI downloads by manylinux2010 compatible installers" for switching the default build environment and auditwheel output to manylinux2010

Given @ehashman's comment:

Note that auditwheel will default to assigning the strictest possible tag, hence it is possible for a binary built inside the manylinux2010 image to receive a manylinux1 tag given that the wheel's symbols match the stricter manylinux1 policy.

I think we actually want to retain the current behavior, and this item is invalid.

If no-one disagrees, I'm going to call this issue resolved and manylinux2010 officially "rolled out".

@ehashman
Copy link
Member

@ehashman ehashman commented Oct 14, 2019

I think I agree this is done, but for a slightly different reason...

Nominate a timeline ... for switching the default build environment ... to manylinux2010

I think this is already our recommendation due to the security issues with building in the manylinux1 docker environment. We don't really need to worry about a percentage of PyPI downloads for this item.

@zackw
Copy link

@zackw zackw commented Oct 14, 2019

@di I agree that auditwheel's behavior of assigning the strictest possible tag is correct.

I hesitate to call manylinux2010 "rolled out" until we have some evidence that the people building wheels really are doing so in the manylinux2010 (or 2014) build environment. This is almost entirely a messaging issue: for instance, do the top five google, bing, duckduckgo, weibo, etc. search results still tell people to use the manylinux1 build environment? if so, we still have some work to do.

@JonasVautherin
Copy link

@JonasVautherin JonasVautherin commented Oct 14, 2019

One thing that is not clear to me is whether I should move to manylinux2010 or not. Intuitively, it sounds like if my project builds with manylinux1, then it is "more compatible" than if it only built with manylinux2010. So anyway I still want to keep manylinux1 to support as many linux flavours as possible. Is that wrong?

@mayeut
Copy link
Contributor

@mayeut mayeut commented Oct 14, 2019

@JonasVautherin,

Intuitively, it sounds like if my project builds with manylinux1, then it is "more compatible" than if it only built with manylinux2010.

That's true, however

So anyway I still want to keep manylinux1 to support as many linux flavours as possible. Is that wrong?

It's probably wrong. I'd say most of linux flavors that came out before 2010 are now EOL meaning they do not receive any security updates. Time to move on. (manylinux2010 base OS, which is centos6 will reach end of life in November of 2020)

@mayeut
Copy link
Contributor

@mayeut mayeut commented Oct 14, 2019

A partial overview of manylinux2010 adoption through quay.io stats (yes, it's partial but can give an estimate which might not be too biased):

@snowman2
Copy link

@snowman2 snowman2 commented Nov 7, 2019

Needing pip>=19.0.0 is causing some troubles in pyproj in this issue pyproj4/pyproj#477. Thoughts?

@di
Copy link
Member

@di di commented Nov 7, 2019

@snowman2 Unfortunately we can't go back in time and add manylinux2010 support to earlier pip versions. Can the project target manylinux1 instead? It's not clear to me from that issue what the problem is.

@snowman2
Copy link

@snowman2 snowman2 commented Nov 7, 2019

Thanks for the information!

It's not clear to me from that issue what the problem is.

Mostly just relaying the issues so y'all are aware.

Can the project target manylinux1

Are there issues doing both?

@veblush
Copy link
Contributor

@veblush veblush commented Nov 7, 2019

Since gRPC Python started distributing both manylinux1 and manylinux2010, it might be useful to see download stats for gRPC 1.25 as of today:

  • manylinux2010: 83%
  • manylinux1: 17%

Definitely, manylinux2010 is a primary platform now for gRPC users. For manylinux1 case, it's not because their environment are not compliant with manylinux2010 but because their pip is older than 19.

@veblush
Copy link
Contributor

@veblush veblush commented Jan 29, 2020

Please update the bullet point for "Distinguish manylinux2010 and manylinux1 images in dockercross" since they have manylinux1, manylinux2010, and manylinux2014 now.

@di
Copy link
Member

@di di commented Jan 29, 2020

Please update the bullet point for "Distinguish manylinux2010 and manylinux1 images in dockercross" since they have manylinux1, manylinux2010, and manylinux2014 now.

Thanks! Done.

@ehashman
Copy link
Member

@ehashman ehashman commented Feb 8, 2020

Will this issue ever die? Surely we can call the manylinux2010 rollout complete, now.

@di
Copy link
Member

@di di commented Feb 8, 2020

I agree!

@di di closed this Feb 8, 2020
@jbarlow83
Copy link

@jbarlow83 jbarlow83 commented Feb 8, 2020

@brainwane
Copy link
Member

@brainwane brainwane commented Feb 9, 2020

Thanks all!

Discussed some further documentation/message-spreading opportunities on Discourse.

brainwane added a commit to brainwane/peps that referenced this issue Apr 23, 2020
Per pypa/manylinux#179 , manylinux2010 is
now implemented in its entirety. Per PEP 1, "When the reference
implementation is complete and incorporated into the main source code
repository, the status will be changed to 'Final'."

Signed-off-by: Sumana Harihareswara <sh@changeset.nyc>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.