max_replays>1 causes difference in output, yet all bugs are triggered in first round?

[colin-scott]

This problem could be indicitive of a deeper problem in STS:

Very strange to me that the behavior of interreplay’s differs depending on whether max_replays_per_subsequence=5 or 1, yet the runtime_stats show that all violations are triggered on the first run. Are new subdirectories created for each unsuccessful replay?

Experiment directory:

fuzz_pyretic_mesh_proactive_firewall_no_close_check_loop_mcs

is with max_replays=1

and with max_replays_per_subsequence=5:

fuzz_pyretic_mesh_proactive_firewall_no_close_check_loop_mcs_with_max_replays_5

---

I question the validity of runtime_stats: they say that all violations are triggered in the first run, yet the execution seems to be quite different.

I wonder if I should question my assumptions, and go back and read the delta debugging papers thoroughly. The biggest difference in our work is that they assume that the test case is injected as a single flat file.