Reproduce Linux ext4 Fuzzer Results

[JohnPMerrill]

Hello,

Awesome project, many thanks for maintaining it.

I'm attempting to reproduce the Linux ext4 fuzzer as described in the kAFL USENIX paper. I'm using the -vm_dir/-vm_ram method to start an Ubuntu 16.10 VM, as follows:

python3 kAFL-Fuzzer/kafl_fuzz.py \           
  -vm_dir snapshot \                         
  -vm_ram snapshot/ram.qcow2 \               
  -mem 512 \                                 
  -seed_dir fs-seed \                        
  -p 8 \                                     
  --purge \                                  
  -abort_time 32 \                           
  -work_dir /dev/shm/kafl-fs-fuzz \          
  -catch_resets \                            
  -i 2048-65535 \  # As the paper mentions, limit the deterministic mutations to the first two KB (and last 4B) of the image                        
  -ip0 0xffffffff812bc000-0xffffffff81357af8 

Some more details on this configuration:

• My machine has a Xeon 4214 (48 cores @ 2.20GHz) and 128GB DDR4 RAM.
• I'm using debootstrap to create the Ubuntu image. The kernel I'm using is a precompiled 4.8.0-22-generic.
• I'm using dd/mkfs.ext4/tune2fs to generate a few 64KB seed images, then I'm appending 4 bytes to the end for the flags to mount. In practice I usually need to pass at least 4 seeds to stop the fuzzer from getting stuck on the import phase.
• To get the -ip0 range I'm looking through /proc/kallsyms for ext4 functions. My first attempt used the entire kernel core range, but that seemed to slow down the fuzzer too much.
• I modified fs-fuzzer.c to add HYPERCALL_KAFL_LOCK and a handshake at the beginning so I wouldn't need to use the loader binary.
• I made a couple of other modifications to kAFL itself, namely changing the machine type from q35 to pc to get loadvm working, and the parallel -vm_ram fix mentioned here: Failed to reuse ram.qcow2 in parallel fuzzing mode. #15 (comment)

So far my fuzzer appears to run in a stable manner, but the performance I'm seeing makes me suspicious that something isn't configured right:

• Execs/s hovers around 400-600 after the first hour or so
• Number of paths gets to about 1000 after the full 32 hours
• No timeouts or panics yet

Any idea if there's an issue in my configuration that could be hampering performance? Any guidance is greatly appreciated.

[il-steffen]

Hi there,

Congrats on getting this to work! Perhaps you can elaborate on the changes you made / issues you encountered? I've seen a couple reports that PT feedback and/or VM loading did not work properly for this use case and it would be great to have a working tutorial again.

Regarding the "lost seeds", I usually see this in connection with noisy coverage output due to non-deterministic target or buggy PT decode. The -funky option may help with this as it spends some more time to validate new findings.

I don't recall what performance we can expect for ext4 fuzzing. If the performance of your corpus is degrading fast, it may help to set a more aggressive timeout initially so slow findings are discarded:

kAFL/kAFL-Fuzzer/common/qemu.py

Line 561 in 8c45d91

ready = select.select([self.control], [], [], 0.25)

There is currently no logic to adapt the timeout dynamically, so you would have to restart fuzzing after some time with higher timeout. Check kafl_plot.py to see the execution time of individual inputs (perf=).

Some stages are also a lot slower than others and you may get 'stuck' in these slow stages when facing a target with many or very long paths. You can run your corpus just using the havoc stage to see what speed you may get in the best case (add -D, and don't use -redqueen -grimoire -radamsa). Grimoire is probably not very useful for ext4. I would also disable Radamsa initially as it does a lot of repetition+length extension which can produce slow inputs. If you then see the fuzzer stuck for a long time in deterministic stages or even redqueen, consider disabling deterministic (-D) or limiting redqueen/deterministic to run only on favorite inputs (in state_logic.py handle_redqueen(), return if len(metadata["fav_bits"] ==0 or <=1).

[JohnPMerrill]

Thanks for the reply. So far disabling deterministic mode is improving perf considerably (up to about 2000 execs/s). Funky mode also does help decrease the frequency of "lost seeds", so I'll use that as well. I still haven't found any panics; my next steps are to look at different kernel configurations (make ext4 loadable, enable KASAN).

As for my VM loading setup, here's a more detailed write-up:

1. Edit targets/linux_x86_64/src/fuzzer/fs_fuzzer.c and add the following to the start of main:

/* Take a VM snapshot */
hprintf("Taking snapshot\n");
kAFL_hypercall(HYPERCALL_KAFL_LOCK, 0);
hprintf("Done\n");

/* Fuzzer handshake */
kAFL_hypercall(HYPERCALL_KAFL_ACQUIRE, 0);
kAFL_hypercall(HYPERCALL_KAFL_RELEASE, 0);

2. Change the following in kAFL-Fuzzer/common/qemu.py:
   a. Change the default QEMU machine type from q35 to pc (I couldn't get q35 to play nicely with savevm/loadvm)
   b. Add console=ttyS0 to the bootparam to get Linux serial working
   c. Change the -hdb option from self.config.argument_values['vm_ram'] to self.config.argument_values['vm_dir'] + "/ram_" + self.qemu_id + ".qcow2". This allows us to load a VM snapshot on multiple threads.

3. Create a basic init script for the guest:

$ cat init
mount -t proc none /proc
mount -t sysfs none /sys

echo 0 > /proc/sys/kernel/randomize_va_space

exec /bin/bash

4. Download or compile Linux v4.8.0.

5. Use debootstrap to create an Ubuntu 16.10 install (could also install from the ISO if you want):

mkdir snapshot && cd snapshot
sudo debootstrap --variant=minbase yakkety yakkety
cp ../targets/linux_x86_64/bin/fuzzer/ext4 yakkety/bin/fuzzer
cp ../targets/linux_x86_64/bin/info/info yakkety/bin/info
cp ../init yakkety/init

mke2fs -L '' -N 0 -d yakkety -m 5 -r 1 -t ext4 "yakkety.ext4" 1G
qemu-img convert -f raw -O qcow2 yakkety.ext4 yakkety.qcow2

6. Setup initial overlays:

qemu-img create -f qcow2 ram.qcow2 512
qemu-img create -b $(realpath yakkety.qcow2) -f qcow2 overlay_0.qcow2
qemu-img create -b $(realpath ram.qcow2) -f qcow2 ram_0.qcow2

7. Start the VM:

../qemu-5.0.0/x86_64-softmmu/qemu-system-x86_64 \
        -enable-kvm \
        -nographic \
        -net none \
        -machine pc \
        -kernel /path/to/vmlinuz-4.8.0-22-generic \
        -hda overlay_0.qcow2 \
  -hdb ram_0.qcow2 \
        -m 512 \
        -append "root=/dev/sda rw console=ttyS0 nokaslr oops=panic nopti mitigations=off init=/init"

8. In the VM, run the fuzzer binary to take a snapshot (and kill QEMU once this is complete):

root@(none):/# fuzzer
Taking snapshot
Done

9. In the snapshot dir, copy overlay_0.qcow2 and ram_0.qcow2 for each thread (overlay_1.qcow2, overlay_2.qcow2 ... overlayN.qcow2).

10. See my above comments on creating a seed and finding ip0 ranges for ext4.

11. Should be ready to launch kafl_fuzz.py.