Take benchmarks to know the performance of each fuzzer and improve rezzuf

[potetisensei]

Relation to existing issues

Relation to existing roadmaps

This issue is important to complete rezzuf because we've never taken enough benchmarks of fuzzers implemented in fuzzuf and it is unknown whether rezzuf actually has a good performance compared to other fuzzers (including ones not in fuzzuf).

By taking benchmarks, we can know if there's something wrong that should be fixed to complete rezzuf, or other possible improvement to enhance rezzuf.

Motivation

Written above.

Description

Because we have an internal benchmark system, running fuzzers on it is enough.
First, we should compare 4 fuzzers: the original AFL++, fuzzuf aflplusplus, fuzzuf aflplusplus --slopt, fuzzuf rezzuf.
rezzuf should be the best or at least comparable to the others.

Alternatives (optional)

Additional context (optional)

---

Authors of this issue must fill the checklist below when they submit it

Mandatory entries

• The title of this issue clearly summarizes the feature request
• The relations to existing issues and roadmaps are stated if they exist
• Each entry above has been filled

---

A repository maintainer will set up an asignee for each issue.
Asignee must check this issue from the perspective of the checklist below before going through a full-fledged feature addition, and leave the initial comment.
Copy to your comment and fill out the checklist on that occasion.

Optional entries

• If the feature request is going to be accepted, it has been commented
• If it is not going to be accepted and this issue is closed, it has been commented

Mandatory entries

• Have checked relationships to existing issues and roadmaps

---

If an assignee is going to solve the task for this issue, consider the policy and divide the procedures into the checklist.
Once divided, write down the checklist to the comment.
Each step for the procedures must correspond to one single pull request. Once the PR is made, update the checklist.
If the procedures are remained undecided, or need additional reviews, update the checklist appropriately with the discussion here.

Procedure checklist

• Deploy fuzzuf on the internal benchmark system.
• Take benchmarks.

---

If an assignee finishes solving the task for this issue, copy the following checklist to the comment and fill it out.

Optional entries

• If you have remaining tasks, relating issues have been created or updated
• If the changes against this issue affect the milestone (e.g. TODO.md), the necessity of version upgrading has already been discussed
• If new tests are required to guarantee the changes against this issue, they have been split to the checklist
• If entries in the checklist are hard to understand, that has been pointed out
• If the discussion concludes that the design concept should be documented appropriately, an assignee has checked additional documents and source codes
• If there were uncertainties during the discussion, all of them have been resolved

Mandatory entries

• The procedure checklist is not required to update any longer
• The PRs correspond to each entries of the checklist have adopted the best way to solve the problems as long as an assignee can come up with
• All the PRs correspond to each entries of the checklist have already been merged
• This issue can be closed

--

If an assignee agrees to close the issue without solving the tasks relating it, describe the reasons in the comment.
Then, copy the following checklist to the comment and fill them out.

Mandatory entries

• The reason to close the issue has been agreed among assignees
• This issue can be closed

[potetisensei]

Phase 1 is done in #95

[potetisensei]

As written in the description, I compared 4 fuzzers: the original AFL++ (the latest version), fuzzuf aflplusplus, fuzzuf aflplusplus --slopt, fuzzuf rezzuf for 3 days against 5 programs.
I launched 10 instances of each fuzzer on the workstation with 64 cores of "Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz".

As we can see, SLOPT successfully made code coverage grow faster at least in these 5 programs.
But fuzzuf aflplusplus is too inefficient compared to the original AFL++.
We don't know why and it should be investigated.

While intuitively we can say SLOPT generally improves the performance, some people may think 5 programs are not enough to be sure. I will add 5 more programs and take another benchmark.

[potetisensei]

Sooo good.
I think this result is enough to close this issue.