CI: omit repeated slow tests #1131
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Here we mark a few particularly slow tests, and edit tox.ini so that these tests are not run (by default) in the "variant" testenvs (e.g. py311-tzdata). The tests marked with `slowtest` are particularly slow on the macOS (and probably the Windows runners). Omitting them saves about a minute per testenv in those environments.
There have been occasional CI test failures (I think) do to this wait being too short.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are a few of our pytest tests that are particularly slow (especially on the macOS and Windows runners).
We run several testenvs on each runner:
pyXX: the default set of testspyXX-mistune0: run with amistune<1.0pin to test under mistune 0.xpyXX-notuils: discovery of external utilities (e.g. ffmpeg, git) via PATH disabled to test that things degrade gracefully when to utitilies are not available (currently only run on the python 3.11 runners)pyXX-pytz: run withpytzinstalled (currently only run on the python 3.11 runners)pyXX-tzdata: run withtzdataexplicitly installed (currently only run on the python 3.11 runners)The slow tests are slow because due to internet access, or spawning of new processes. We don't really get any benefit by repeating them on all of the above testenv variants.
This PR adds a
slowtestpytest marker and applies it to those particularly slow tests which may be safely skipped in the variant testenvs.Tox.iniis adjusted so that those tests are skipped for all but the mainpyXXtestenvs.This appears to speed the macOS job by a minute or more per testenv.
Issue(s) Resolved
CI tests are very slow