Skip to content

introduce --fuzz=<limit> to make tests pass after a certain quantity of fuzz testing has completed #22900

@andrewrk

Description

@andrewrk

There should be two modes of fuzzing, infinite mode, and finite mode. This issue is for finite mode.

Introduce --fuzz=<limit>, where the limit can be one of these:

  • time - a value such as 1s, 2m, 3h, 4d
  • iteration count - a value such as 100, 100K, 100M, 100G

Time-based fuzzing makes all fuzz tests finish successfully after the specified interval has passed. The time is global; adding more fuzz tests and keeping the time the same means less time spent on each fuzz test.

Iteration count based fuzzing makes a given fuzz test finish successfully after that many different inputs have been tried. The number of iterations is per test, so adding more fuzz tests while keeping the same iteration count means it will take longer to finish successfully, since the new tests will be also given the iterations.

Open questions:

  • what about fuzz tests that might want more iterations or more time spent on them relative to each other?
  • should zig build default to --fuzz=1K or similar when fuzz tests are found? If this is set to a reasonably low value, it could be quite valuable in finding shallow bugs quickly. It also opens the door for reporting the seed rather than input and having developers reproduce the failure easily since the iteration count was low.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementSolving this issue will likely involve adding new logic or components to the codebase.fuzzing

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions