Reproducing evaluation part of paper Healer

[harperchen]

Dear developers,

I recently ran healer, healer-, syzkaller, and moonshine with the following setting and aimed to reproduce the branch coverage growth shown in the paper published on SOSP 2021 "HEALER: Relation Learning Guided Kernel Fuzzing".

• Syzkaller commit: 4285ffa3f
• Moonshine: the same commit as Syzkaller with extra strong_distill.db as initial seed
• Healer commit: ea7e71361
• Healer-: the same commit as Healer, without dynamically updating relation table (comment out line 249-269 in file healer_fuzzer/src/fuzzer.rs) https://github.com/SunHao-0/healer/blob/main/healer_fuzzer/src/fuzzer.rs#L249

The test kernel target is Linux 5.0. I launched 4 virtual machines for all experiments. Each virtual machine is configured to use 2 cores and 4G memory, which is the same as the setting illustrated in the paper. The first figure shows the branch coverage growth of four tools over 24 hours.

To eliminate the influence of throughput, I further drew the branch coverage growth of four tools over the number of executed traces and got the following figure.

I'm curious about the following problems:

1. The branch coverage achieved by healer in my experiments is less than 10w, whereas, the coverage can achieve near 20w (Figure 4) in the paper. To reproduce the results shown in the paper, could you please shed some light on whether my setting is incorrect or not?
2. The coverage doesn't hurt a lot when I disabled the dynamic relation learning part (healer-). May I know how to reproduce the branch coverage improvement (Table 2) as shown in Sec 6.2 when evaluating the effectiveness of the relation learning?

Thank you.

The detailed configurations are shown below.

# Command to run healer and healer-
./bin/healer -d stretch.img --ssh-key stretch.id_rsa -k bzImage_5.0  -S ./ -j 4

# Config of Syzkaller and Moonshine
{
        "name": "",
        "target": "linux/amd64",
        "http": "0.0.0.0:56741",
        "rpc": ":0",
        "workdir": "/path/to/syzkaller/workdir",
        "kernel_obj": "/path/to/linux_5.0",
        "kernel_src": "/path/to/linux_5.0",
        "kernel_build_src": "/path/to/linux_5.0",
        "image": "/path/to/stretch.img",
        "sshkey": "/path/to/stretch.id_rsa",
        "ssh_user": "root",
        "syzkaller": "/path/to/syzkaller",
        "procs": 1,
        "sandbox": "none",
        "cover": true,
        "type": "qemu",
        "vm": {
                "count": 4,
                "kernel": "/path/to/bzImage_5.0",
                "cpu": 2,
                "mem": 4096
        }
}

[SunHao-0]

Hi @harperchen,

I think there're two reasons for your questions.

First, there're many limitations of this open-source version of Healer compared with Syzkaller, and the Healer we used in our experiments is internally maintained in a private repo. For example, the hints based mutations here and the checksum handling here are not provided in this repo. Also, this version of Healer can not use the latest Syzlang description due to this commit a7ce77be27d8e3728b97122a005bc5b23298cfc3 of latest Syzkaller. For me, the motivation behind this repo is to demonstrate the idea of implementing a kernel fuzzer with rust language and show the basic idea of relation, but, again, many important features have not been added here. I can't help much with this due to kinds of reasons, sorry for that. Second, you need a better configuration. Obviously, you didn't compile the Linux kernel with some important features based on the coverage range you got, which should be at least 150,000. Also, the way you disable relations may not be right. See here and here for more information.

Besides, I prefer technical discussions and bug reports here. For anything else, e.g, paper-related stuff, please send an email to my public gmail.

[harperchen]

Thanks for the reply. May I know how to disable relations correctly to learn the effectiveness of relation learning?

[harperchen]

Sorry. I'm new to using syzbot config. Do you use config stable-5.4-kasan to achieve at least 150,000 coverage?

[SunHao-0]

May I know how to disable relations correctly to learn the effectiveness of relation learning?

You need at least modify fuzzer.rs, select.rs, relation.rs. See the source code for more details.

Do you use config stable-5.4-kasan to achieve at least 150,000 coverage?

No, I use the latest config from syzbot and make customization based on it.

This can be closed now. No more basic configuration stuff.