Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run fuzzer daily #2634

Merged
merged 1 commit into from Mar 27, 2024
Merged

Run fuzzer daily #2634

merged 1 commit into from Mar 27, 2024

Conversation

tcharding
Copy link
Member

@tcharding tcharding commented Mar 26, 2024

Waiting for the fuzzer slows down the dev feedback loop because it makes CI slow. We still want fuzz coverage but not in a way that slows down devs.

Instead of running the fuzzer on every PR just run it once a nightly.

As we do for other daily jobs, rename the yaml file to make explicit what it does.

Note also we only get 20 jobs, currently there are 18 fuzzing jobs. This means for this hour any other CI runs will only have access to 2 jobs (if i understand GitHub resource usage correctly).

Open questions

Docs: https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration

Concurrent jobs - The number of concurrent jobs you can run in your account depends on your GitHub plan,

What is the definition of "account" - is that a user account, or a repository? Which account is running the cron job?

@coveralls
Copy link

coveralls commented Mar 26, 2024

Pull Request Test Coverage Report for Build 8440434973

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 83.225%

Totals Coverage Status
Change from base Build 8427789763: 0.0%
Covered Lines: 19066
Relevant Lines: 22909

💛 - Coveralls

@apoelstra
Copy link
Member

concept ACK. More often than not, when the fuzzer fails on a PR, it's not related to the PR itself.

It looks to me like the current script runs for 30 seconds, not 30 minutes.

And I would expect 30 consecutive minutes of fuzzing to be better than 30 non-consecutive minutes.

@tcharding
Copy link
Member Author

It looks to me like the current script runs for 30 seconds, not 30 minutes

Ha! I had a feeling it was something obvious. Thanks, will run it for an hour I rekon - for no particular reason other than we can. That will have the effect of blocking anyone running any other CI jobs though if we go over 20 jobs inside that hour.

@tcharding
Copy link
Member Author

Looks like we currently have 18 fuzzing jobs, perhaps we want to keep that in mind and leave a couple of slots open for that poor dev who wants to run CI during the fuzz window?

Waiting for the fuzzer slows down the dev feedback loop because it makes
CI slow. We still want fuzz coverage but not in a way that slows down
devs.

Instead of running the fuzzer on every PR just run it once a nightly.

As we do for other daily jobs, rename the yaml file to make explicit
what it does.

Open questions:

- This should run for 30 minutes but I can't work out why the current
  set up only runs for a shorter time than that?
- Is this less fuzzing i.e., is 30 minutes once a day better or worse that 1
  minute 30 times?
@tcharding
Copy link
Member Author

tcharding commented Mar 26, 2024

Now runs for an hour. Note that the internet told me GitHub infrastructure uses UTC time so I've set the time to hopefully by not too annoying for @apoelstra and @Kixunil (and doxed our timezones but all our github accounts already do that). Its a bit annoying for me, falling in the last hour of my normal work day, lets give it a go and see how it performs.

Copy link
Member

@apoelstra apoelstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 6ab0110 though if this proves too onerous we should try 30 or even 15 minutes

@tcharding
Copy link
Member Author

Sweet, this is CI only, can go in via the one-ack carve out. No real rush but I'm really liking #2633 (except the wasm thing).

@apoelstra
Copy link
Member

Yeah, agreed, I'll one-ACK merge this. It is easy to revert if somebody complains (but I think this actually results in significantly more fuzzing than we had before, and it should speed up CI on all other PRs, so I'm not expecting complaints).

@apoelstra apoelstra merged commit ebcddfc into rust-bitcoin:master Mar 27, 2024
168 checks passed
@tcharding tcharding deleted the 03-24-fuzz-daily branch March 27, 2024 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants