Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify CI test workflow #279

Closed
ian-r-rose opened this issue Aug 25, 2022 · 0 comments · Fixed by #370
Closed

Simplify CI test workflow #279

ian-r-rose opened this issue Aug 25, 2022 · 0 comments · Fixed by #370
Assignees
Labels
dx Developer experience

Comments

@ian-r-rose
Copy link
Contributor

We currently run the submodules in the tests/ directory in separate jobs:

  • benchmarks
  • runtime
  • stability

Each of those jobs has its own test matrix, so we actually wind up having a large number of sub-jobs for every workflow run.

I'm not convinced that is serving us well today:

  1. Each job is parallelized using pytest-xdist with ten workers. That makes a lot of sense, since the vast majority of the work is done by coiled clusters, and the CI runner is mostly sitting around waiting for results. However, we aren't getting the full value out of pytest-xdist because we are running it with loadscope, so each test module is run as a single unit. This basically means we don't have enough modules to actually saturate the xdist workers.
  2. The different jobs have different test matrices. In particular, "stability" has the largest matrix, testing a few combinations that are not tested in others. I think this is because, at the time of writing, "stability" was the fastest submodule to run, so we could afford the extra matrices (is this right @jrbourbeau?). But since Integration tests for spilling #229 the stability tests are quite slow. @hendrikmakait may have ideas to speed it up a bit, but I still am not convinced it's a good idea to have different test matrices for different submodules.
  3. There is a lot of code duplication in the workflow yaml for each submodule, which adds cognitive overhead.

I propose we consolidate the above jobs into a single "test" job, which is itself parameterized over a consistent test matrix. I doubt that would actually change the runtime too much, and might even speed it up, as the xdist workers would be better able to balance work. It would also simplify maintenance going forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dx Developer experience
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants