afl-fuzz seg faults when using a python custom mutator

[r00tus3r]

Compiled AFLplusplus with ASAN_BUILD=1 + AFL_DEBUG=1. Segmentation fault seems to occur during calibration stage 2.

Btacktrace

==3737653==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f9a66109800 at pc 0x7f9a69b9c397 bp 0x7ffcd41b17c0 sp 0x7ffcd41b0f68
READ of size 2014 at 0x7f9a66109800 thread T0
    #0 0x7f9a69b9c396 in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
    #1 0x7f9a6956e825 in PyByteArray_FromStringAndSize (/lib/x86_64-linux-gnu/libpython3.10.so.1.0+0xcd825)
    #2 0x562920c00ccd in post_process_py src/afl-fuzz-python.c:537
    #3 0x562920c1efe3 in write_to_testcase src/afl-fuzz-run.c:87
    #4 0x562920c21c7b in calibrate_case src/afl-fuzz-run.c:481
    #5 0x562920bc240f in perform_dry_run src/afl-fuzz-init.c:894
    #6 0x562920bb241c in main src/afl-fuzz.c:2228
    #7 0x7f9a69268d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #8 0x7f9a69268e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #9 0x562920b9d804 in _start (/usr/local/bin/afl-fuzz+0x24804)

On a high level, the python custom mutator had the following code:

def init(seed):
    pass

def post_process(buf):
    # Modify buf into the format required by the program being fuzzed
    # If buf is not good, return b"" else return the updated buf in out_buf
    return out_buf

def deinit():
    pass

I tried to follow the example code to do this, maybe I'm missing something. I'll be happy to provide any additional information. Thanks!

[vanhauser-thc]

but you have not defined out_buf in your python script? if you do not want to change anything then "return buf".

[r00tus3r]

I have defined it, len(out_buf) > len(buf), data in buf is copied to out_bufwhich is in the format expected by the program.

[vanhauser-thc]

please give a real script that fails, not something that cannot work in the first place.

[vanhauser-thc]

but yes post_process has a bug, I am clueless though why but I have neither knowledge about python, nor did I implement that python function. handing over to dominik :)

[vanhauser-thc]

for dominik:
make ASAN_BUILD=1 afl-fuzz && PYTHONPATH=pwd AFL_PYTHON_MODULE=test ASAN_OPTIONS="disable_coredump=0:unmap_shadow_on_exit=1:abort_on_error=1:detect_leaks=0:symbolize=1:detect_odr_violation=0:print_stacktrace=1" ./afl-fuzz -i in -o out -- ./t
needs a large input file for me in in/

[domenukk]

Tried to fix it in #1701, but not sure if this messes other things up..

[r00tus3r]

I ran my code with your fix, now, after fuzzing for a few seconds, I get the following error:

==4139592==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61d000049080 at pc 0x7f00a753be57 bp 0x7ffca034e730 sp 0x7ffca034dee8
READ of size 142144 at 0x61d000049080 thread T0
    #0 0x7f00a753be56 in __interceptor_write ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1146
    #1 0x557844318733 in save_if_interesting src/afl-fuzz-bitmap.c:532
    #2 0x55784439d3cb in common_fuzz_stuff src/afl-fuzz-run.c:1073
    #3 0x55784435fb52 in fuzz_one_original src/afl-fuzz-one.c:2953
    #4 0x55784437501b in fuzz_one src/afl-fuzz-one.c:5836
    #5 0x55784432b9cb in main src/afl-fuzz.c:2599
    #6 0x7f00a6c02d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #7 0x7f00a6c02e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #8 0x557844314804 in _start (/usr/local/bin/afl-fuzz+0x24804)

0x61d000049080 is located 0 bytes to the right of 2048-byte region [0x61d000048880,0x61d000049080)
allocated by thread T0 here:
    #0 0x7f00a75b0c18 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x557844396082 in afl_realloc include/alloc-inl.h:700
    #2 0x557844396ece in write_with_gap src/afl-fuzz-run.c:231
    #3 0x55784439c243 in trim_case src/afl-fuzz-run.c:923
    #4 0x55784434fe06 in fuzz_one_original src/afl-fuzz-one.c:512
    #5 0x55784437501b in fuzz_one src/afl-fuzz-one.c:5836
    #6 0x55784432b9cb in main src/afl-fuzz.c:2599
    #7 0x7f00a6c02d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

SUMMARY: AddressSanitizer: heap-buffer-overflow ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1146 in __interceptor_write

Does this help or should I provide steps to reproduce?

(Getting the same error when using C or Python custom mutators)

[r00tus3r]

Also, this fix seems to have fixed the stability issue (at least during the few seconds of fuzzing) I had mentioned on discord @vanhauser-thc :)

[vanhauser-thc]

Dominik's fix attempt removed important functionality.
I attempted a fix with f756734 in the dev branch - please test.
sorry guys, for me it had so far worked, I guess how I used it was different to your test cases and hence did not trigger the bug.

[r00tus3r]

Now, the crash is at a different location and stability issue is back

==4155969==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d000016880 at pc 0x55c65a8cf76e bp 0x7ffef4aa1220 sp 0x7ffef4aa1210
READ of size 8 at 0x61d000016880 thread T0
    #0 0x55c65a8cf76d in afl_realloc include/alloc-inl.h:674
    #1 0x55c65a8e1cfc in fuzz_one_original src/afl-fuzz-one.c:2958
    #2 0x55c65a8f719e in fuzz_one src/afl-fuzz-one.c:5836
    #3 0x55c65a8ad93b in main src/afl-fuzz.c:2594
    #4 0x7efc33293d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #5 0x7efc33293e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #6 0x55c65a8967c4 in _start (/usr/local/bin/afl-fuzz+0x247c4)

0x61d000016880 is located 0 bytes inside of 2048-byte region [0x61d000016880,0x61d000017080)
freed by thread T0 here:
    #0 0x7efc33c41c18 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x55c65a917f57 in afl_realloc include/alloc-inl.h:700
    #2 0x55c65a918672 in write_to_testcase src/afl-fuzz-run.c:138
    #3 0x55c65a91f70a in common_fuzz_stuff src/afl-fuzz-run.c:1040
    #4 0x55c65a8e1cd5 in fuzz_one_original src/afl-fuzz-one.c:2953
    #5 0x55c65a8f719e in fuzz_one src/afl-fuzz-one.c:5836
    #6 0x55c65a8ad93b in main src/afl-fuzz.c:2594
    #7 0x7efc33293d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

previously allocated by thread T0 here:
    #0 0x7efc33c41c18 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x55c65a8cf809 in afl_realloc include/alloc-inl.h:700
    #2 0x55c65a8d18f5 in fuzz_one_original src/afl-fuzz-one.c:464
    #3 0x55c65a8f719e in fuzz_one src/afl-fuzz-one.c:5836
    #4 0x55c65a8ad93b in main src/afl-fuzz.c:2594
    #5 0x7efc33293d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

SUMMARY: AddressSanitizer: heap-use-after-free include/alloc-inl.h:674 in afl_realloc

[vanhauser-thc]

can you please share a custom mutator where this happens?

[vanhauser-thc]

we want to do a new release and this should be fixed then. and we have no own custom mutator that shows this issue, at least for me it is working fine, hence I cannot debug this.

[r00tus3r]

sent you an email!

[vanhauser-thc]

this should be fixed now, please test

[r00tus3r]

Thanks a lot! It seems to be fixed now but I'm seeing a mem leak, would you be able to verify:

==1342537==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 22600896 byte(s) in 159 object(s) allocated from:
#0 0x7f12b19c5867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x564fee717cc2 in queue_testcase_store_mem src/afl-fuzz-queue.c:1394
#2 0x564fee6abdf8 in save_if_interesting src/afl-fuzz-bitmap.c:601
#3 0x564fee730bc4 in common_fuzz_stuff src/afl-fuzz-run.c:1081
#4 0x564fee6f2cd5 in fuzz_one_original src/afl-fuzz-one.c:2953
#5 0x564fee70819e in fuzz_one src/afl-fuzz-one.c:5836
#6 0x564fee6be93b in main src/afl-fuzz.c:2594
#7 0x7f12b1017d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)