New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run fuzzer daily #2634
Run fuzzer daily #2634
Conversation
c32d36e
to
453c5d2
Compare
Pull Request Test Coverage Report for Build 8440434973Details
💛 - Coveralls |
concept ACK. More often than not, when the fuzzer fails on a PR, it's not related to the PR itself. It looks to me like the current script runs for 30 seconds, not 30 minutes. And I would expect 30 consecutive minutes of fuzzing to be better than 30 non-consecutive minutes. |
Ha! I had a feeling it was something obvious. Thanks, will run it for an hour I rekon - for no particular reason other than we can. That will have the effect of blocking anyone running any other CI jobs though if we go over 20 jobs inside that hour. |
Looks like we currently have 18 fuzzing jobs, perhaps we want to keep that in mind and leave a couple of slots open for that poor dev who wants to run CI during the fuzz window? |
Waiting for the fuzzer slows down the dev feedback loop because it makes CI slow. We still want fuzz coverage but not in a way that slows down devs. Instead of running the fuzzer on every PR just run it once a nightly. As we do for other daily jobs, rename the yaml file to make explicit what it does. Open questions: - This should run for 30 minutes but I can't work out why the current set up only runs for a shorter time than that? - Is this less fuzzing i.e., is 30 minutes once a day better or worse that 1 minute 30 times?
453c5d2
to
6ab0110
Compare
Now runs for an hour. Note that the internet told me GitHub infrastructure uses UTC time so I've set the time to hopefully by not too annoying for @apoelstra and @Kixunil (and doxed our timezones but all our github accounts already do that). Its a bit annoying for me, falling in the last hour of my normal work day, lets give it a go and see how it performs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 6ab0110 though if this proves too onerous we should try 30 or even 15 minutes
Sweet, this is CI only, can go in via the one-ack carve out. No real rush but I'm really liking #2633 (except the wasm thing). |
Yeah, agreed, I'll one-ACK merge this. It is easy to revert if somebody complains (but I think this actually results in significantly more fuzzing than we had before, and it should speed up CI on all other PRs, so I'm not expecting complaints). |
Waiting for the fuzzer slows down the dev feedback loop because it makes CI slow. We still want fuzz coverage but not in a way that slows down devs.
Instead of running the fuzzer on every PR just run it once a nightly.
As we do for other daily jobs, rename the yaml file to make explicit what it does.
Note also we only get 20 jobs, currently there are 18 fuzzing jobs. This means for this hour any other CI runs will only have access to 2 jobs (if i understand GitHub resource usage correctly).
Open questions
Docs: https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration
What is the definition of "account" - is that a user account, or a repository? Which account is running the cron job?