-
-
Notifications
You must be signed in to change notification settings - Fork 3.1k
Closed
Labels
enhancementSolving this issue will likely involve adding new logic or components to the codebase.Solving this issue will likely involve adding new logic or components to the codebase.fuzzing
Milestone
Description
There should be two modes of fuzzing, infinite mode, and finite mode. This issue is for finite mode.
Introduce --fuzz=<limit>
, where the limit can be one of these:
- time - a value such as
1s
,2m
,3h
,4d
- iteration count - a value such as
100
,100K
,100M
,100G
Time-based fuzzing makes all fuzz tests finish successfully after the specified interval has passed. The time is global; adding more fuzz tests and keeping the time the same means less time spent on each fuzz test.
Iteration count based fuzzing makes a given fuzz test finish successfully after that many different inputs have been tried. The number of iterations is per test, so adding more fuzz tests while keeping the same iteration count means it will take longer to finish successfully, since the new tests will be also given the iterations.
Open questions:
- what about fuzz tests that might want more iterations or more time spent on them relative to each other?
- should
zig build
default to--fuzz=1K
or similar when fuzz tests are found? If this is set to a reasonably low value, it could be quite valuable in finding shallow bugs quickly. It also opens the door for reporting the seed rather than input and having developers reproduce the failure easily since the iteration count was low.
squeek502
Metadata
Metadata
Assignees
Labels
enhancementSolving this issue will likely involve adding new logic or components to the codebase.Solving this issue will likely involve adding new logic or components to the codebase.fuzzing