DGFuzz Integration

[DanBlackwell]

This is a new fuzzer that I'd like to test out.

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-07-dgfuzz --fuzzers dgfuzz libafl 

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-08-dgfuzz --fuzzers dgfuzz

[DonggeLiu]

Ops:
Would you mind making a trivial modification to service/gcbrun_experiment.py?
This will allow PR experiments. Here is an example to add a dummy comment : )
Thanks!

[DanBlackwell]

Done! Forgot about that sorry

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-08-dgfuzz --fuzzers dgfuzz

[DanBlackwell]

experiment name: 2024-07-08-dgfuzz
Report: here
experiment data: here

[DanBlackwell]

@DonggeLiu Agh, forgot to run make presubmit and the linter had taken issue with my whitespace. I think it needs restarting? The experiment report didn't populate, so I assume it's down to the CI checks failing.

[DonggeLiu]

@DonggeLiu Agh, forgot to run make presubmit and the linter had taken issue with my whitespace. I think it needs restarting?

Hi @DanBlackwell
That experiment has launched: The data directory is not empty.
The report normally takes longer to show after FB finishes building the benchmarks and fuzzers.

I've manually terminated all VM instances of that experiment and will relaunch a new one to use the latest code.

The experiment report didn't populate, so I assume it's down to the CI checks failing.

The CI checks won't block PR experiments, they are separated.
But it's good that you double-checked CIs, as sometimes the fuzzer might not be compatible with some benchmarks : )

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-08-dgfuzz-1 --fuzzers dgfuzz

[DanBlackwell]

Hi @DanBlackwell
That experiment has launched: The data directory is not empty.
The report normally takes longer to show after FB finishes building the benchmarks and fuzzers.

My apologies, I thought the experiment-folders and coverage dirs in the data directory normally showed up after 1-2 hours; then when I saw the CI checks failed I jumped to the conclusion that it only ran the build and bailed.

But it's good that you double-checked CIs, as sometimes the fuzzer might not be compatible with some benchmarks : )

This fuzzer will only work on a subset for now, as DFSan requires all dependencies to be compiled with the -fsanitize=dataflow flag set (including the C++ standard library, OpenSSL etc.). If it looks promising then I'll put in some time to figure out how that can be done.

[DanBlackwell]

experiment name: 2024-07-08-dgfuzz-1

Report: here
experiment data: here

[DanBlackwell]

@DonggeLiu Hmm, seems like it’s not started the experiment again (no experiment-folders created again). Is it possible that because it’s the only fuzzer in the experiment - and some of the targets did not build?

If so, maybe it could be run with just the 8 targets that do build. I've now added dgfuzz to all the unsupported_fuzzers fields for those that do not build, so in theory it can be run with the same gcb command. (from what I saw here, it looks like gcbrun only lets you do one benchmark at a time, is that correct?)

Additionally, I’ve set the merge_with_nonprivate flag to false just in case running less programs causes any weirdness; it would be best if we could also run libafl as a baseline / control given that it won't merge now.

If it helps, these are the 8 that build ok:

harfbuzz_hb-shape-fuzzer lcms_cms_transform_fuzzer libpcap_fuzz_both mbedtls_fuzz_dtlsclient openthread_ot-ip6-send-fuzzer stb_stbi_read_fuzzer vorbis_decode_fuzzer zlib_zlib_uncompress_fuzzer

PS, if the experiment did start at some point (it's been 5 hours without starting at the time of writing), then ignore all this and let it run through :)

[DonggeLiu]

PS, if the experiment did start at some point (it's been 5 hours without starting at the time of writing), then ignore all this and let it run through :)

I think it did : )
https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2024-07-08-dgfuzz-1/index.html

Is it possible that because it’s the only fuzzer in the experiment - and some of the targets did not build?

I don't think this will block the experiment. If this happens, then that benchmark will be excluded from the exp without affecting others.

[DanBlackwell]

Hi @DonggeLiu , I've pushed up some updates - I think that last run was getting killed due to shared memory not getting freed correctly. Can you run the following:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-10-dgfuzz --fuzzers dgfuzz

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-11-dgfuzz --fuzzers dgfuzz

[DanBlackwell]

Experiment 2024-07-11-dgfuzz data and results will be available later at:
The experiment data.
The experiment report(experimental).

[DanBlackwell]

@DonggeLiu Just a note that most of these did not build with some apt errors it seems. The following was taken from libpcap_fuzz_both:

Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.257 Get:240 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-dev amd64 3.8.2-0ubuntu2 [1212 B]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.447 Err:233 http://security.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04.9
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.447   404  Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.555 Err:234 http://security.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04.9
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.555   404  Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.555 Err:239 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04.9
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.555   404  Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/python3.8/libpython3.8_3.8.10-0ubuntu1~20.04.9_amd64.deb  404  Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/python3.8/libpython3.8-dev_3.8.10-0ubuntu1~20.04.9_amd64.deb  404  Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/python3.8/python3.8-dev_3.8.10-0ubuntu1~20.04.9_amd64.deb  404  Not Found [IP: 91.189.91.81 80]
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 7.562 Fetched 124 MB in 4s (28.4 MB/s)
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": #12 ERROR: executor failed running [/bin/sh -c apt-get install -y         build-essential         python3-dev         python3-setuptools         automake         cmake         git         flex         bison         libglib2.0-dev         libpixman-1-dev         cargo         libgtk-3-dev         ninja-build         gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev         libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev]: exit code: 100
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": ------
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate":  > [8/9] RUN apt-get install -y         build-essential         python3-dev         python3-setuptools         automake         cmake         git         flex         bison         libglib2.0-dev         libpixman-1-dev         cargo         libgtk-3-dev         ninja-build         gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev         libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev:
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": ------
Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate": executor failed running [/bin/sh -c apt-get install -y         build-essential         python3-dev         python3-setuptools         automake         cmake         git         flex         bison         libglib2.0-dev         libpixman-1-dev         cargo         libgtk-3-dev         ninja-build         gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev         libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev]: exit code: 100
Finished Step #2 - "dgfuzz-libpcap_fuzz_both-builder-intermediate"
ERROR
ERROR: build step 2 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

From what I see it looks like http://archive.ubuntu.com/ goes through fine but http://security.ubuntu.com fails. Strangely it did work for mbed_fuzz_dtlsclient.

[DonggeLiu]

From what I see it looks like http://archive.ubuntu.com/ goes through fine but http://security.ubuntu.com fails. Strangely it did work for mbed_fuzz_dtlsclient.

Yep, this should be a flaky internet connection failure.
We sometimes see this but cannot do much other than re-run the exp.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-11-dgfuzz-1 --fuzzers dgfuzz

[DanBlackwell]

Experiment 2024-07-11-dgfuzz-1 data and results will be available later at:
The experiment data.
The experiment report(experimental).

[DanBlackwell]

Apt is still not playing ball it seems. It's trying to fetch:

http://security.ubuntu.com/ubuntu/pool/main/p/python3.8/libpython3.8_3.8.10-0ubuntu1~20.04.9_amd64.deb

Looking here it seems there is no 20.04.9, but there is 20.04.10 (python3.8-dbg_3.8.10-0ubuntu1~20.04.10_amd64.deb). As to why it is trying to fetch that particular version, I don't know. The particular command that fails is copied from AFL++ builder.Dockerfile, so maybe that will have the same issue?

EDIT: it was missing apt-get update; I guess so far I'd got lucky that the base image was new enough that nothing had been replaced yet.

[DanBlackwell]

Hi @DonggeLiu , I figured out that the issue was on my end (see comment above). In the meantime I've added a 'control' condition so that I can see how much of the improvement is down to the DFSan guidance. Could you run the following command please:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-12-dgfuzz --fuzzers dgfuzz dgfuzz_control

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-07-13-dgfuzz --fuzzers dgfuzz dgfuzz_control

[DanBlackwell]

Experiment 2024-07-13-dgfuzz data and results will be available later at:
The experiment data.
The experiment report(experimental).