Speeding up CI using pytest-xdist

[chrisiacovella]

Pull Request Summary

The CI is taking quite a long time to run (about 45 minutes). This aims to speed this up using pytest-xdist, which will allow concurrent execution of tests. To do this though, we need to make sure our tests will not be in conflict (make sure any files being written out/read have unique names, making sure all unique test files are at minimum using their own unique temp directories).

Edit: actual numbers from the last merge into main:

• ubuntu 3.11: 44m 16s
• ubtuntu 3.10: 44m 23s
• macOS 3.11: 53m 30s
• macOS 3.10: 48m 42s

Associated Issue(s)

• Speeding up CI #277

Pull Request Checklist

• Issue(s) raised/addressed and linked
• Includes appropriate unit test(s)
• Appropriate docstring(s) added/updated
• Appropriate .rst doc file(s) added/updated
• PR is ready for review

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.21%. Comparing base (80e4f43) to head (425352c).

Additional details and impacted files

[chrisiacovella]

using auto which sets the number of concurrent process to the number of logical cores (which in this case I think is 2 for ubuntu):

• ubuntu 3.11: 23m 49s
• ubtuntu 3.10: 23m 8s
• macOS 3.11: 45m 32s
• macOS 3.10: 59m 22s

Clearly this is not effective on macOS, but that is okay, since we are primarily focused on the ubuntu tests, so as long as those can complete faster so we can get more immediate feedback that would be helpful.

[chrisiacovella]

Setting -n 8 seemed to speed things up substantially, but ran out of memory as the test_training_with_lightning tests started to run; setting -n4 seems to make no difference compared to auto for ubuntu. So I will switch back to auto. Since it is slower on macOS using parallel, this will be set to 0 which is the default pytest. Again, the point is mostly to speed up ubuntu so we can get faster turn around time as to whether things are passing.

One way to speed this up would be to profile the training with lightning tests to figure out which can be safely run in parallel and do separate tests instead of using parametrize to loop over implemented networks.

[chrisiacovella]

Codecov randomly took 17 minutes on Ubuntu 3.10. Ignore that (usually takes <10 s) both ubuntu CI timings are about 24 minutes.

[wiederm]

That is super usefull!

And it's great that you fixed the output location for model saving and dataset caching. That is something that we should write in the style guide: only output to the tmp_dir