Ongoing CI issues on main

[pradyunsg]

Filing an issue to track the issues we're seeing with CI on main.

[pradyunsg]

The tests are failing due to git/git@7b70e9e, which was released in (latest) git 2.45.1.

actions/runner-images@e11ab1f#diff-5c04a529d3c8adf7a5f23afe544071dad1853e281c9c7b44cd8d626b6c57444dR51 is the commit/change bumping the docs about the Git version in GitHub Actions' CI.

[pradyunsg]

We should be able to set GIT_NO_LAZY_FETCH=0 to fix our CI, but it's unclear whether we should expect our users to do so for being able to use pip.

/cc @sbidoul since this is a VCS-specific issue and he's one of the our resident experts.

[pfmoore]

I'm not completely clear what the precise behaviour change in git is, and what their intended resolution for users is. From what I can understand, it sounds like the issue is that if you have a partial clone, then a git fetch on that partial clone will no longer work. But that seems like a pretty major breakage, so I sort of assume I've misunderstood somehow.

As this seems to be realed to a CVE, I'm not convinced we should be setting the "ignore the risks" flag in our CI, much less expect our users to do so.

But I'm definitely not a specialist here, so while I'm interested in what the issue is about, I'll defer to @sbidoul over how we fix it.

[sbidoul]

he's one of the our resident experts

Expert = the one still around who has fiddled the most with pip's VCS code base 😅

Thanks for the investigation @pradyunsg.

I can reproduce the test failures locally with git 2.45.1.

I just tried pip install git+https://github.com/pypa/pip-test-package and this works fine. And many of our git tests do succeed. Which is good news, and probably why no user has complained yet.

At first sight, and if I interpret the hints from git/git@7b70e9e and the CVE, it might be the problem is with local git repos that are themselves partial clone.

This can be reproduced with

$ git clone --filter=blob:none https://github.com/pypa/pip-test-package
$ pip install git+file://$PWD/pip-test-package
...
remote: fatal: could not fetch 4487a60e3ea0fa99473a0b056b40a59241cc290d from promisor remote.
...

An editable install such as pip install -e git+https://github.com/pypa/pip-test-package@0.1#egg=pip-test-package does work.

So the issue may not impact many pip users in practice. Time will tell.

Setting GIT_NO_LAZY_FETCH=0 in our CI seems like a valid short term solution to me, while we investigate if a better approach exists (maybe declaring the local git repos in our tests as safe or something).

[pradyunsg]

From what I can understand, it sounds like the issue is that if you have a partial clone, then a git fetch on that partial clone will no longer work.

Not quite, if I'm understanding the CVE correctly. It's rather that, when you clone with --filter, git used to enable lazy fetching so that a git checkout would realise "oh, I don't have that" and reach out to the server to fetch automatically. And, now, it doesn't (by default).

[pradyunsg]

Expert = the one still around who has fiddled the most with pip's VCS code base 😅

True.

[pradyunsg]

it might be the problem is with local git repos that are themselves partial clone.

I think the local repository is used as an example and, based on 2d35b80, it seems to me that git has effectively disabled-by-default the reason that we'd adopted --filter in the first place.

That seems like a thing to consider for us, and to evaluate if this feature should continue to be a thing.

[sbidoul]

it seems to me that git has effectively disabled-by-default the reason that we'd adopted --filter in the first place.

I'm not sure about that yet. According to my preliminary tests (see #12719 (comment)), a local partial clone is necessary to reproduce the issue. Installing from a remote repo with --filter=blob:none continues to work fine.