'std::out_of_range' crash in I2P fuzz test

[murchandamus]

Is there an existing issue for this?

• I have searched the existing issues

Current behaviour

I got a std::out_of_range crash during merging fuzz outputs in the i2p target (see log below.) I was not able to reproduce the crash when re-running the seed with the fuzz executable in the regular build, but I figured I’d share it here if someone else wants to take a look. The binaries used for the merge and the reproduction were both built from the latest master: 738ef44abb6895dad016d8f32f7d7fa1c251b354.

Expected behaviour

If this issue can be reproduced, it may point at a bug in the I2P fuzzer or the I2P code.

Steps to reproduce

You can recreate the seed with:

echo "wIA9ID0gUkVTVUxUPU9LClBSSVY9gD0gPSBSRVNVTFQ9T0sKUFJJVj0CAAD//13/GhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoAEBoaGhoaGhoaGhoaGhouGhoaGhoaGhoaGhoaGn4aGhoaGhoaGgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaABoaGhoaGhpXGhoAGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGurq6mrqUFBQUFBQUFBQUFBQUOrq6gAAAABbAAAAAAAAAAAAAAAAAAAAAgAAeHh4eHh4eHgpeHh4eHh4eHh4eHgaGhoaGhoaGhoaGho=" | base64 -d crash-946784c8f03d9aeeef70e22b346a069e6940e186

Relevant log output

Run i2p with args /home/murch/Workspace/qa-merge/src/test/fuzz/fuzz -set_cover_merge=1 -shuffle=0 -prefer_small=1 -use_value_profile=0 /tmp/merge-all/i2p ../qa-assets/fuzz_seed_corpus/i2p ../qa-assets-active-fuzzing/fuzz_seed_corpus/i2p
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2619897554
INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x55a029467ca0, 0x55a0294c4a37), 
INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x55a0294c4a38,0x55a029a923a8), 
MERGE-OUTER: 14141 files, 0 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3047975919
INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x563efb209ca0, 0x563efb266a37), 
INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x563efb266a38,0x563efb8343a8), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge17284.txt'
MERGE-INNER: 14141 total files; 0 processed earlier; will process 14141 files now
#1	pulse  cov: 244 exec/s: 0 rss: 88Mb
#2	pulse  cov: 245 exec/s: 0 rss: 88Mb
#4	pulse  cov: 245 exec/s: 0 rss: 88Mb
#8	pulse  cov: 245 exec/s: 0 rss: 88Mb
#16	pulse  cov: 263 exec/s: 0 rss: 88Mb
#32	pulse  cov: 300 exec/s: 0 rss: 88Mb
#64	pulse  cov: 313 exec/s: 0 rss: 88Mb
#128	pulse  cov: 336 exec/s: 0 rss: 88Mb
#256	pulse  cov: 417 exec/s: 0 rss: 88Mb
#512	pulse  cov: 435 exec/s: 0 rss: 88Mb
#1024	pulse  cov: 455 exec/s: 1024 rss: 88Mb
#2048	pulse  cov: 485 exec/s: 1024 rss: 88Mb
#4096	pulse  cov: 538 exec/s: 682 rss: 88Mb
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 385) >= this->size() (which is 0)
==17287== ERROR: libFuzzer: deadly signal
    #0 0x563ef9f7eda4 in __sanitizer_print_stack_trace (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x9bfda4) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
    #1 0x563ef9f56248 in fuzzer::PrintStackTrace() (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x997248) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
    #2 0x563ef9f3c2d3 in fuzzer::Fuzzer::CrashCallback() (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x97d2d3) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
    #3 0x7fa19a83c45f  (/lib/x86_64-linux-gnu/libc.so.6+0x3c45f) (BuildId: ff2d8e707625b73b293961a4bc168e373d14a44a)
    #4 0x7fa19a89152a in __pthread_kill_implementation nptl/pthread_kill.c:43:17
    #5 0x7fa19a89152a in __pthread_kill_internal nptl/pthread_kill.c:78:10
    #6 0x7fa19a89152a in pthread_kill nptl/pthread_kill.c:89:10
    #7 0x7fa19a83c3b5 in raise signal/../sysdeps/posix/raise.c:26:13
    #8 0x7fa19a82287b in abort stdlib/abort.c:79:7
    #9 0x7fa19aca4ee5  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa4ee5) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
    #10 0x7fa19acb6e9b  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb6e9b) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
    #11 0x7fa19acb6f06 in std::terminate() (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb6f06) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
    #12 0x7fa19acb7167 in __cxa_throw (/lib/x86_64-linux-gnu/libstdc++.so.6+0xb7167) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
    #13 0x7fa19aca82ba  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xa82ba) (BuildId: 1fcdadafe5a79e1031ab0da645aa3798954cf53d)
    #14 0x563efa269705 in std::vector<unsigned char, std::allocator<unsigned char>>::_M_range_check(unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:1153:4
    #15 0x563efa269705 in std::vector<unsigned char, std::allocator<unsigned char>>::at(unsigned long) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_vector.h:1194:2
    #16 0x563efa269705 in i2p::sam::Session::MyDestination() const src/i2p.cpp:354:38
    #17 0x563efa269705 in i2p::sam::Session::CreateIfNotCreatedAlready() src/i2p.cpp:405:40
    #18 0x563efa268664 in i2p::sam::Session::Listen(i2p::Connection&) src/i2p.cpp:143:9
    #19 0x563efa07c088 in i2p_fuzz_target(Span<unsigned char const>) src/test/fuzz/i2p.cpp:38:14
    #20 0x563ef9f8269e in void std::__invoke_impl<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>(std::__invoke_other, void (*&)(Span<unsigned char const>), Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:61:14
    #21 0x563ef9f8269e in std::enable_if<is_invocable_r_v<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>, void>::type std::__invoke_r<void, void (*&)(Span<unsigned char const>), Span<unsigned char const>>(void (*&)(Span<unsigned char const>), Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:111:2
    #22 0x563ef9f8269e in std::_Function_handler<void (Span<unsigned char const>), void (*)(Span<unsigned char const>)>::_M_invoke(std::_Any_data const&, Span<unsigned char const>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:290:9
    #23 0x563efa1dd075 in std::function<void (Span<unsigned char const>)>::operator()(Span<unsigned char const>) const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/std_function.h:591:9
    #24 0x563efa1dd075 in LLVMFuzzerTestOneInput src/test/fuzz/fuzz.cpp:178:5
    #25 0x563ef9f3d742 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x97e742) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
    #26 0x563ef9f47445 in fuzzer::Fuzzer::CrashResistantMergeInternalStep(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, bool) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x988445) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
    #27 0x563ef9f2d5cd in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x96e5cd) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
    #28 0x563ef9f56a82 in main (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x997a82) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)
    #29 0x7fa19a823a8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #30 0x7fa19a823b48 in __libc_start_main csu/../csu/libc-start.c:360:3
    #31 0x563ef9f22074 in _start (/home/murch/Workspace/qa-merge/src/test/fuzz/fuzz+0x963074) (BuildId: 1f28ec48de7ec8f793559abd4fba645f972afe4f)

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 0 ; base unit: 0000000000000000000000000000000000000000
artifact_prefix='./'; Test unit written to ./crash-946784c8f03d9aeeef70e22b346a069e6940e186

➜  qa-merge git:(merge-fuzz) ✗ FUZZ=i2p src/test/fuzz/fuzz   crash-946784c8f03d9aeeef70e22b346a069e6940e186
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3335468885
INFO: Loaded 1 modules   (380311 inline 8-bit counters): 380311 [0x5612fee10ca0, 0x5612fee6da37), 
INFO: Loaded 1 PC tables (380311 PCs): 380311 [0x5612fee6da38,0x5612ff43b3a8), 
src/test/fuzz/fuzz: Running 1 inputs 1 time(s) each.
Running: crash-946784c8f03d9aeeef70e22b346a069e6940e186
Executed crash-946784c8f03d9aeeef70e22b346a069e6940e186 in 0 ms
***
*** NOTE: fuzzing was not performed, you have only
***       executed the target code on a fixed set of inputs.
***

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

738ef44

Operating system and version

Ubuntu 23.04

Machine specifications

No response

[maflcko]

Is m_private_key accessed by more than one thread? Which threads are running in the target? What happens if the target is run in tsan?

[murchandamus]

As far as I can tell, the merging process was the only thing running that touched this data. I’m not sure how I would be running the target in tsan.

[darosior]

I’m not sure how I would be running the target in tsan.

See the --with-sanitizers configure option. Available sanitizers are listed here.

So if you are using ./configure --with-sanitizers=fuzzer right now, you want to compile using ./configure --with-sanitizers=fuzzer,thread for tsan.

[dergoegge]

I'm also not able to reproduce. Running with tsan also doesn't yield anything so far (I'm not even sure there are any threads involved?).

[maflcko]

Jup, could be a single thread, and given that tsan doesn't complain, seems good to close this for now.

[maflcko]

Steps to reproduce:

cd qa-assets
git fetch origin 00c956e4a9c555da5640946ec28cdd5e501c0554
git checkout 00c956e4a9c555da5640946ec28cdd5e501c0554

FUZZ=i2p ../bitcoin-core/src/test/fuzz/fuzz -runs=1 ./fuzz_seed_corpus/i2p/

[maflcko]

cc @vasild

[maflcko]

549c82ad3a34a885ecca37a5f04c36dfbaa95d17 is the first bad commit
commit 549c82ad3a34a885ecca37a5f04c36dfbaa95d17
Author: Vasil Dimov <vd@FreeBSD.org>
Date:   Wed Apr 7 11:40:59 2021 +0200

    fuzz: use ConsumeBool() instead of !ConsumeBool()
    
    The former is shorter and ends up with a "random" bool anyway.

 src/test/fuzz/util.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[maflcko]

Tried a bisect, but I guess that didn't work, because the fuzz input format was changed at some point in 2021, so likely the fuzz inputs trigger something else when going back further than 549c82a

[mzumsande]

This cannot be reproduced by running a single seed, only by a combination of two seeds from the qa-assets mentioned in #28665 (comment):
FUZZ=i2p ./src/test/fuzz/fuzz -runs=1 ~/qa-assets/fuzz_seed_corpus/i2p/41717eadbc1239897d9a03b261a2590aff0474cf ~/qa-assets/fuzz_seed_corpus/i2p/56b13c4cfdbc0d48b8b03d0c5c235a499d1eca59

I think the problem is that one seed (41717eadbc1239897d9a03b261a2590aff0474cf) attempts to create a i2p privkey, which will be empty because i2p is not running within the fuzz environment (within normal usage of bitcoind, this should be impossible because init would abort).
This empty file is saved into the datadir, and then picked up by another seed (56b13c4cfdbc0d48b8b03d0c5c235a499d1eca59), leading to the error.
So I think that there are two things to fix:

1. Have the fuzz test clean up after itself (removing any private key it may write into the data directory) so that fuzz seeds don't depend on each other
2. Implement better checks in i2p.cpp to prevent parsing of empty private keys.

[maflcko]

On a general note I wonder how useful this target is at all, given that it can't even reach basic coverage in its own test code, despite years of fuzzing: https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/77f0ceb7175dbd00/7c13430491b21dca/fuzz.coverage/src/test/fuzz/i2p.cpp.gcov.html

Not sure whether the lack of coverage is due to this bug here, but that also makes me question why no one (including oss-fuzz) reported this bug earlier over the years?