Sulley: Optimize Session.fuzz_single_case()

[jtpereyda]

If you use fuzz_single_case() to fuzz all of the test cases, it will take longer and longer as you get into higher numbers. Each time the method is called, it iterates up to the desired point in the list. This can add non-trivial delay with a big enough test.

Options:

• Cache cases within fuzz_case_iterator. This would be snazzy.
• Cache the iterator within fuzz_single_case(), and don't reset unless needed. Less flexible.

[sawilson2009]

This affects me.

I'm distributing a boofuzz load in parallel across multiple computers (in a kinda dumb way, somewhat necessary due to my constraints). I have a "server" that calls session.num_mutations() and then waits for clients to call in; when they do, the server uses the number of clients ("boofuzz host machines") and the return from num_mutations() to tell each client what chunk of the tests it should run. Then the server is then no longer involved. The clients then each spin up multiple isolated boofuzz+target instances in parallel, each of them re-generating the same session graph (hopefully) with the same random.seed, each running a single test against an isolated target copy. Once each boofuzz+target instance finishes, the client kills that instance and then starts another boofuzz+target instance on the next mutation from the pool that the client was assigned from the server.

Each boofuzz instance fuzzes a single case via session.fuzz_single_case().

[jtpereyda]

@sawilson2009 Thank you for explaining your use case. The two options I listed in the first comment wouldn't help in that case, since the Python script is completely killed and restarted.

The challenge here is that the interface for creating new fuzz primitives expects an interator-like behavior rather than list-like behavior. (This is done to allow for optimal performance when running through many many test cases)

A couple other options I can think of:

1. Add a fuzz-single-case method to Request, use the len method to check the number of primitive mutations for each primitive, and skip primitives until coming to the target primitive. From there, iterate mutations until coming to the exact number.
2. Add a fuzz-single-case method to IFuzzable, implementing the behavior described above and also implementing behavior for individual primitives.