Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache the .tox directory on GitHub Actions #1

Closed
wants to merge 1 commit into from
Closed

Conversation

seanh
Copy link
Contributor

@seanh seanh commented Jul 9, 2022

This can speed up CI on pull requests significantly as we avoid re-installing all the Python packages.

It also means that new releases of our Python dependencies won't cause CI to break because CI won't be using them: it'll be using its cached versions. So this effectively pins our Python dependencies on CI. Our scheduled nightly CI run bypasses this pinning and updates the cache with the latest versions of all dependencies so that our packages do get tested against new versions of their dependencies.

The cached .tox directory will not be used under certain circumstances:

  • If your PR has changed tox.ini (compared to main) then tox will automatically recreate the venvs

  • If your PR has changed setup.cfg (compared to main) then our tox-recreate plugin will automatically recreate the venvs

  • When CI is run on a schedule (every night at midnight) rather than on a PR an rm -rf .tox/<ENV> command is run to delete the venv and therefore force its recreation.

    The recreated venv will have the latest versions of all the packages installed in it and will be cached so that future PRs retrieve the latest versions from the cache. The cache will only be updated if CI passed, GHA won't update the cache if the scheduled nightly run failed due to a problem with a new version of a dependency.

    An alternative implementation of this could be if the branch is main (or the default branch), rather than if we're running on a schedule.

  • If your PR's branch name ends with -rm-tox-dir that will trigger the rm -rf .tox/<ENV> command. This allows you to send PRs that won't use the cached .tox dirs (for example if fixing a bug with a new version of a dependency). Such PRs will not update the cache for other PRs or for scheduled nightly runs due to how GHA's caching works: workflows running on PR branches see files cached on that branch or on main but they don't see files cached on other PR branches (and workflows running on main only see files cached on main)

  • When running CI manually you can check a checkbox to enable the rm -rf .tox/<ENV> command. If you do this on a PR branch you won't update the cache for other PRs or scheduled nightly runs. If you do it on main you will.

Cache the .tox directory on GitHub Actions.
@seanh seanh changed the title Cache the .tox directory Cache the .tox directory on GitHub Actions Jul 9, 2022
@seanh
Copy link
Contributor Author

seanh commented Jul 9, 2022

I think restore-keys should be removed, for packages we only want to restore the cache from exact key matches

@seanh
Copy link
Contributor Author

seanh commented Jul 11, 2022

Closing in favour of hypothesis/cookiecutters#34

@seanh seanh closed this Jul 11, 2022
@seanh seanh deleted the cache-tox branch July 11, 2022 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant