Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: pip cache failures in workflows #293

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Conversation

xarvex
Copy link
Member

@xarvex xarvex commented Jun 14, 2024

From #255:

Ocassionally, a pip install will fail because of mismatched hashes, which from my research I believe this to be related to a cache error. This cannot easily be tested due to its seemingly random nature, but my hope is that my fix in using the action/setup-python workflow will avoid it.

Unfortunately the fix in 82037e5 seems to have just caused registered bad cache to be reused over and over. So, this serves as a sort of "opposite" to that and removes using the cache altogether.

Again, due to random nature, this cannot easily be tested and will show results over time.

@xarvex xarvex added Type: Bug Something isn't working as intended Type: CI Continuous Integration / workflows Status: Review Needed A review of this is needed labels Jun 14, 2024
@yedpodtrzitko
Copy link
Collaborator

This is most likely caused by requirements-dev.txt which is not taken into account when creating the cache key, only requirements.txt is considered by default.

This can be fixed by specifying cache-dependency-path

I'm not sure if this wouldn't exclude requirements.txt for a change:

cache-dependency-path: '**/requirements-dev.txt'

so using this should be safe:

cache-dependency-path: '**/requirements*.txt'

@xarvex
Copy link
Member Author

xarvex commented Jun 14, 2024

This is most likely caused by requirements-dev.txt which is not taken into account when creating the cache key, only requirements.txt is considered by default.

This can be fixed by specifying cache-dependency-path

I'm not sure if this wouldn't exclude requirements.txt for a change:

cache-dependency-path: '**/requirements-dev.txt'

so using this should be safe:

cache-dependency-path: '**/requirements*.txt'

I would think this if it weren't for the fact that packages in requirements.txt are affected by this too, and actually happens with only those packages as far as I have been able to tell. For example, the release workflow which only uses requirements.txt experienced this error recently.

@yedpodtrzitko
Copy link
Collaborator

yedpodtrzitko commented Jun 15, 2024

So this means the error happens during download, which means the problem isnt in the cached files. I tried to search for the error message, and it seems pip doesnt know the connection was interrupted, considering the download is complete which causes the hash mismatch.

In that case not using cache wont help, it might make the problem worse as we'll be downloading each dependency every time, increasing the chance for the mismatch.

Is there any occurrence of this problem for the pytest workflow? In that one there's upgrade to latest version of pip (python -m pip install --upgrade pip) which we can hope has the problem fixed.

Other than that I found that uv has also dealt with this problem, but they actually fixed it ( astral-sh/uv#1912 ), so using uv instead of pip could be worth trying:

pip install uv
uv pip install -r requirements.txt

@xarvex xarvex added Status: Help Wanted Extra attention is needed and removed Status: Review Needed A review of this is needed labels Jun 15, 2024
@xarvex xarvex marked this pull request as draft June 15, 2024 22:55
@xarvex
Copy link
Member Author

xarvex commented Jun 15, 2024

Going back to the drawing board for this.

@xarvex xarvex changed the title fix: avoid using pip cache in workflow fix: pip cache failures in workflows Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Help Wanted Extra attention is needed Type: Bug Something isn't working as intended Type: CI Continuous Integration / workflows
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants