gh-114965: Updated bundled pip to 24.0

[sbidoul]

• Issue: Update bundled pip to 24.0 #114965

[sethmlarson]

SBOM updates LGTM, hope the experience was straightforward :)

[pfmoore]

@sethmlarson I thought the idea of the new SBOM stuff was that we didn't have to change the SBOM data for a pip release?

I'm going to assume you've reviewed that as I don't know what I'd check. Also, won't the SBOM data impact the automatic backports? I'll add the "needs backport" labels once this is merged, but if backports are going to neeed manual intervention on an ongoing basis, that's a bit awkward 🙁

[miss-islington-app]

Thanks @sbidoul for the PR, and @pfmoore for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

[miss-islington-app]

Thanks @sbidoul for the PR, and @pfmoore for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @sbidoul and @pfmoore, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker a4c298c1494b602a9650b597aad50b48e3fa1f41 3.11

[miss-islington-app]

Sorry, @sbidoul and @pfmoore, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker a4c298c1494b602a9650b597aad50b48e3fa1f41 3.12

[sbidoul]

@sethmlarson the SBOM procedure went well, although the CI had to remind me to do it, of course :) That's a small additional burden in the pip release process, especially the manual entry of the checksums. Any chance to automate that further?

Regarding the backport, I'm stuck because cherry_picker says I'm not in a cpython repo. I do have the upstream remote pointing to the right place, it's the same cpython local clone I used before. Any idea what where I could look to resolve this?

[bedevere-app]

GH-114971 is a backport of this pull request to the 3.12 branch.

[sbidoul]

Regarding the backport, I'm stuck because cherry_picker says I'm not in a cpython repo

I could make it work with an older version of cherry_picker.

[bedevere-app]

GH-114973 is a backport of this pull request to the 3.11 branch.

[hugovk]

Regarding the backport, I'm stuck because cherry_picker says I'm not in a cpython repo. I do have the upstream remote pointing to the right place, it's the same cpython local clone I used before. Any idea what where I could look to resolve this?

Sounds like this bug, Cherry Picker is swallowing warnings and issuing a misleading one:

python/cherry-picker#99

[pfmoore]

@sethmlarson gentle ping on the SBOM question

[sethmlarson]

Thanks for the ping @pfmoore, responding to you and @sbidoul's comments:

I thought the idea of the new SBOM stuff was that we didn't have to change the SBOM data for a pip release?

My thought process for the SBOM tracking "checked in" dependencies was to have the review from experts on the dependency being updated to acknowledge the changes because all non-pip dependencies exist outside a packaging ecosystem and thus have no way to fetch any metadata. Perhaps I can move the pip SBOM generation into the release-tools repository instead since the metadata is mostly discoverable.

This is also does a double-check that what gets checked into the repository actually is the artifact on PyPI, but moving this step to the Python release stage won't add any additional risk, more likely will only catch mistakes much later in the process (but I don't think this really matters).

That's a small additional burden in the pip release process, especially the manual entry of the checksums.

You mention "checksums" here, do you mean more than the checksum for the wheel itself? That checksum update could also likely be automated, since whatever appears on PyPI for the pip version being claimed should be correct.

[pfmoore]

have the review from experts on the dependency being updated to acknowledge the changes because all non-pip dependencies exist outside a packaging ecosystem

I think the problem here is that the pip maintainers likely won’t check the SBOM data (I know I won’t) but instead will simply trust that the automation worked, so no added value is gained. Add to that the fact that the SBOM breaks automated backports, and the current approach doesn’t add any real value while increasing effort.

I can also confirm that, like @sbidoul, I wouldn’t remember to do the SBOM process until prompted by CI. And I’m concerned about the checksums - I’m on Windows, and I don’t immediately know how to reliably calculate a checksum for a file. I’m sure there are utilities (or I could write something in Python) but I’d need to go and look for docs each time. Would I need to manually download all of pip’s dependencies to calculate the checksums?

[sethmlarson]

@pfmoore @sbidoul I've opened an issue that would move pip's automated discovery into the CPython release process, rather than the source repository, since pip is a special case as I noted above. Let me know if this proposal makes sense to you.

Apologies to you both for the extra burden the past two releases have been. I've tried to optimize for as little manual work as possible, in many cases a bit is unavoidable but for pip it should be.