Adding option to merge all inputs at once

[chkeita]

No description provided.

[bmc-msft]

Can 'merge' be it's on template, rather than part of the existing libfuzzer template?

[tunz]

What if I want to minimize corpus using "merge", what would be the best approach?

I'm running these commands to minimize corpus.

mkdir empty_dir
azcopy sync {container} ./corpus
./fuzzer -merge=1 ./empty_dir ./corpus
azcopy sync ./empty_dir {container} --deleteDestination=true

I'm wondering if I can use this new merge task for this purpose. I imagine this might be one way:

1. Create a new container
2. Run merge with inputs=old container and unique_inputs=new container
3. Remove the old container
4. Copy corpus from the new container to the old container
5. Remove the new container

But then, some other process should wait for the task to be completed and run the copy step.

[chkeita]

What if I want to minimize corpus using "merge", what would be the best approach?

I'm running these commands to minimize corpus.

mkdir empty_dir
azcopy sync {container} ./corpus
./fuzzer -merge=1 ./empty_dir ./corpus
azcopy sync ./empty_dir {container} --deleteDestination=true

I'm wondering if I can use this new merge task for this purpose. I imagine this might be one way:

1. Create a new container
2. Run merge with inputs=old container and unique_inputs=new container
3. Remove the old container
4. Copy corpus from the new container to the old container
5. Remove the new container

But then, some other process should wait for the task to be completed and run the copy step.

@tunz you will be able to use this task for that purpose. i am adding a new option to the libfuzzer template that will allow you to do this with this command.

onefuzz template libfuzzer merge  <project> <name> <build> <pool_name> --inputs <input_dir>

[tunz]

@chkeita Thanks for the new option :) This will helps us a lot.
But, I was also wondering if I can minimize the corpus within the same container (i.e. pushing results to the same container as the "inputs" container) because I want to run continuous fuzzing keeping the same container. If I understood correctly, I still have to copy the results to the inputs container back.

[chkeita]

@tunz you are correct. You will need to manually copy the result of the merge from the unique_inputs container.
if we output the result of the merge to the inputs container, it can result in some files being deleted. This will prevent other tasks like coverage from functioning properly.

[bmc-msft]

I think it's reasonable to optionally support merging into the inputs container, especially if we're talking about performing in a bulk fashion.

[bmc-msft]

Something important to remember, merge provides a minimum set of inputs that provide the same coverage. By default, we should save the results to an minimized output container.

However, it's common enough to say "ok, this is a known good binary, minimize my corpus such that I can use the minimized set moving forwards" without having manual effort

[bmc-msft]

Standard integration tests pass, though I've not tested the new merge job.