CI: add pytest-randomly workflow. #4553
Conversation
|
This is really interesting! +1 on having this run as a cron-job or something on the master branch. |
|
Okay, I moved the workflow from being triggered on PR/push events to a scheduled one. The only problem is that the GH workflow docs make it seem like I will be the only one who sees notifications about failures (since I'm creating the action). It would be nice if there were a way for other people to subscribe to such notifications, but if I read these docs right it seems like it only goes to the person who last modified the scheduled workflow. |
|
We may have to add a manual notification step a-la https://github.com/marketplace/actions/send-email |
MridulS
force-pushed
the
ci/add-pytest-randomly
branch
from
d2eaa3c
to
4a9f5ea
Compare
MridulS
force-pushed
the
ci/add-pytest-randomly
branch
from
c96146b
to
4fd37aa
Compare
MridulS
force-pushed
the
ci/add-pytest-randomly
branch
from
4fd37aa
to
7512452
Compare
MridulS
force-pushed
the
ci/add-pytest-randomly
branch
from
7512452
to
3891c3a
Compare
|
I'll go ahead and merge this and get the cron job up and running on github actions. Let's see if we find more test failures :) |
* Move random test order to scheduled workflow. * Add missing jobs keyword. * bump to py3.9 and retrigger build * run the workflow in the PR * fix path to requirements * remove action run from PRs, just run on a schedule Co-authored-by: Mridul Seth <mail@mriduls.com> Co-authored-by: Mridul Seth <seth.mridul@gmail.com>
* Move random test order to scheduled workflow. * Add missing jobs keyword. * bump to py3.9 and retrigger build * run the workflow in the PR * fix path to requirements * remove action run from PRs, just run on a schedule Co-authored-by: Mridul Seth <mail@mriduls.com> Co-authored-by: Mridul Seth <seth.mridul@gmail.com>
Adds a single workflow that runs the test suite with the tests executed in random order. This can help identify problems with poorly-designed tests.
On the one hand, adding this to CI can help improve the robustness of the test suite. OTOH, it can also introduce failures to PRs that are not related to the submitted changes.
One alternative could be to de-couple this from submitted PRs and just have a time or status-based workflow (e.g. run the workflow once a day). LMK if you think this is worthwhile to pursue!