Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for manylinux2010 rollout #179

Closed
12 of 14 tasks
ncoghlan opened this issue Apr 14, 2018 · 146 comments
Closed
12 of 14 tasks

Tracking issue for manylinux2010 rollout #179

ncoghlan opened this issue Apr 14, 2018 · 146 comments

Comments

@ncoghlan
Copy link
Member

ncoghlan commented Apr 14, 2018

I've accepted the manylinux2010 PEP, so that's now an active interoperability standard: https://www.python.org/dev/peps/pep-0571/

However, there are a number of further steps needed before folks can actually make use of that new baseline, and the order of operations matters (since building manylinux2010 wheels isn't very useful if installers won't install them):

Essential client support:

  • Update pip 19.0 to install manylinux2010 wheels (PR: Manylinux2010 pip#5410)
  • Update pip 9.x to install manylinux2010 wheels (needed to avoid the manylinux2010 rollout being delayed by the Python API break between pip 9.x and pip 10.x) (@ncoghlan: skip for now, wait and see if we get complaints about needing this)
  • Update pipenv to install manylinux2010 wheels (needed due to pipenv vendoring pip, so the support depends on the version of pipenv being used, rather than the version of pip in the host or target environments)

Enable publication of manylinux2010 wheel archives:

Management of transition from manylinux1:

Additional projects to consider once core capability support has rolled out:

Tagging @takluyver @markrwilliams as the PEP 571 authors, and @di and @brainwane from the PyPI/PSF side of things.

@ncoghlan ncoghlan mentioned this issue Apr 14, 2018
@markrwilliams
Copy link

See:

pypa/auditwheel#88
pypa/pip#5008

@takluyver
Copy link
Member

Updating the spec page on PyPUG:

pypa/packaging.python.org#470

@mayeut
Copy link
Member

mayeut commented May 12, 2018

Confirm that twine will upload manylinux2010 wheel archives

Confirmed

Update warehouse to accept manylinux2010 wheel uploads

PR: pypi/warehouse#3951

di pushed a commit to pypi/warehouse that referenced this issue May 14, 2018
manylinux2010 platform tag is official per PEP571

see also pypa/manylinux#179
@di
Copy link
Sponsor Member

di commented May 14, 2018

pypi/warehouse#3951 has been merged.

@wtolson
Copy link

wtolson commented May 15, 2018

pip merge request has been updated here: pypa/pip#5410

@wtolson
Copy link

wtolson commented May 15, 2018

auditwheel merge request has been updated here: pypa/auditwheel#92

@armudgal
Copy link
Contributor

Hi,
First of all great job with this, I just wanted to know is there an estimate when the manylinux2010 will roll-out. Thanks :)

@takluyver
Copy link
Member

There's no one roll-out date, because support needs to be added to various different tools in the ecosystem (see the list at the top of the issue). I imagine that it's probably a matter of a few weeks to a few months before you can use it and expect it to work.

@armudgal
Copy link
Contributor

Oh, thanks for the information

@techalchemy
Copy link
Member

@ncoghlan Pipenv will have pip 10 support after we merge pypa/pipenv#2255 which un-vendors pip9 completely, and bundles a patched version of pip10 which can be re-vendored and patched when updates land (this was possible because pip-tools accepted my PR which makes them pip10 compatible as well)

so whenever the pr lands in pip we can make the call about whether to actually go ahead and pull that into our patched version before it's released (technically we could do this now, even) and we will be set

@westurner
Copy link

"SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register"
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/4BGE226DB5EWIAT5VCJ75QD5ASOVJZCM/

@safijari
Copy link

Other than the open pull requests needing to get merged, is there any more left to do for this effort? How can I help?

@ehashman
Copy link
Member

I have merged support for manylinux2010 in auditwheel. Will be cutting a release candidate today.

Note that auditwheel will default to assigning the strictest possible tag, hence it is possible for a binary built inside the manylinux2010 image to receive a manylinux1 tag given that the wheel's symbols match the stricter manylinux1 policy.

@di
Copy link
Sponsor Member

di commented Oct 12, 2019

The last remaining item here is:

Nominate a timeline and/or "percentage of PyPI downloads by manylinux2010 compatible installers" for switching the default build environment and auditwheel output to manylinux2010

Given @ehashman's comment:

Note that auditwheel will default to assigning the strictest possible tag, hence it is possible for a binary built inside the manylinux2010 image to receive a manylinux1 tag given that the wheel's symbols match the stricter manylinux1 policy.

I think we actually want to retain the current behavior, and this item is invalid.

If no-one disagrees, I'm going to call this issue resolved and manylinux2010 officially "rolled out".

@ehashman
Copy link
Member

I think I agree this is done, but for a slightly different reason...

Nominate a timeline ... for switching the default build environment ... to manylinux2010

I think this is already our recommendation due to the security issues with building in the manylinux1 docker environment. We don't really need to worry about a percentage of PyPI downloads for this item.

@zackw
Copy link

zackw commented Oct 14, 2019

@di I agree that auditwheel's behavior of assigning the strictest possible tag is correct.

I hesitate to call manylinux2010 "rolled out" until we have some evidence that the people building wheels really are doing so in the manylinux2010 (or 2014) build environment. This is almost entirely a messaging issue: for instance, do the top five google, bing, duckduckgo, weibo, etc. search results still tell people to use the manylinux1 build environment? if so, we still have some work to do.

@JonasVautherin
Copy link

One thing that is not clear to me is whether I should move to manylinux2010 or not. Intuitively, it sounds like if my project builds with manylinux1, then it is "more compatible" than if it only built with manylinux2010. So anyway I still want to keep manylinux1 to support as many linux flavours as possible. Is that wrong?

@mayeut
Copy link
Member

mayeut commented Oct 14, 2019

@JonasVautherin,

Intuitively, it sounds like if my project builds with manylinux1, then it is "more compatible" than if it only built with manylinux2010.

That's true, however

So anyway I still want to keep manylinux1 to support as many linux flavours as possible. Is that wrong?

It's probably wrong. I'd say most of linux flavors that came out before 2010 are now EOL meaning they do not receive any security updates. Time to move on. (manylinux2010 base OS, which is centos6 will reach end of life in November of 2020)

@mayeut
Copy link
Member

mayeut commented Oct 14, 2019

A partial overview of manylinux2010 adoption through quay.io stats (yes, it's partial but can give an estimate which might not be too biased):

@snowman2
Copy link

snowman2 commented Nov 7, 2019

Needing pip>=19.0.0 is causing some troubles in pyproj in this issue pyproj4/pyproj#477. Thoughts?

@di
Copy link
Sponsor Member

di commented Nov 7, 2019

@snowman2 Unfortunately we can't go back in time and add manylinux2010 support to earlier pip versions. Can the project target manylinux1 instead? It's not clear to me from that issue what the problem is.

@snowman2
Copy link

snowman2 commented Nov 7, 2019

Thanks for the information!

It's not clear to me from that issue what the problem is.

Mostly just relaying the issues so y'all are aware.

Can the project target manylinux1

Are there issues doing both?

@veblush
Copy link
Contributor

veblush commented Nov 7, 2019

Since gRPC Python started distributing both manylinux1 and manylinux2010, it might be useful to see download stats for gRPC 1.25 as of today:

  • manylinux2010: 83%
  • manylinux1: 17%

Definitely, manylinux2010 is a primary platform now for gRPC users. For manylinux1 case, it's not because their environment are not compliant with manylinux2010 but because their pip is older than 19.

@veblush
Copy link
Contributor

veblush commented Jan 29, 2020

Please update the bullet point for "Distinguish manylinux2010 and manylinux1 images in dockercross" since they have manylinux1, manylinux2010, and manylinux2014 now.

@di
Copy link
Sponsor Member

di commented Jan 29, 2020

Please update the bullet point for "Distinguish manylinux2010 and manylinux1 images in dockercross" since they have manylinux1, manylinux2010, and manylinux2014 now.

Thanks! Done.

@ehashman
Copy link
Member

ehashman commented Feb 8, 2020

Will this issue ever die? Surely we can call the manylinux2010 rollout complete, now.

@di
Copy link
Sponsor Member

di commented Feb 8, 2020

I agree!

@di di closed this as completed Feb 8, 2020
@jbarlow83
Copy link

jbarlow83 commented Feb 8, 2020 via email

@brainwane
Copy link
Member

Thanks all!

Discussed some further documentation/message-spreading opportunities on Discourse.

brainwane added a commit to brainwane/peps that referenced this issue Apr 23, 2020
Per pypa/manylinux#179 , manylinux2010 is
now implemented in its entirety. Per PEP 1, "When the reference
implementation is complete and incorporated into the main source code
repository, the status will be changed to 'Final'."

Signed-off-by: Sumana Harihareswara <sh@changeset.nyc>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests