Fuzz tests don't link correctly on macos #1056
Comments
brson
changed the title
|
I have reproduced the first error, I reproduced the error on MacOS Ventura, ARM, and I think the ld I have installed is revision 711 (https://github.com/apple-oss-distributions/ld64/releases/tag/ld64-711), the latest as of today. Unfortunately building their ld to debug is not trivial - there are some instructions at https://github.com/dmaclach/ld64, but I have not tried. The error is related to static initializers in the macho-o I also see a single other report of this error here: getsentry/sentry-cocoa#1990. They were explicitly placing data in I cannot find any rust source code that mentions I have tested with the sold linker, which also fails, but produces a potentially more useful error: And sold is easy to build, so could be useful for debugging. I have found a workaround, but do not understand the issue: Using libfuzzer seems to require activating some sanitizer, and cargo-fuzz defaults to asan. Telling it to use thread sanitizer instead makes the error go away in my limited testing: The libfuzzer docs mention a "fuzzer" sanitizer, but it does not seem to exist, or at least not be exposed by rustc. Using both the workaround, and linking with sold, produces a fuzzer that either does not work or is very slow - unclear. |
|
Also this issue does not present itself for trivial fuzz tests, which makes sense as seemingly nobody else has reported this error. I tried to reduce a test case but the results were kind of nonsense so far. Not linking to soroban-sdk at all makes the error go away, adding some seemingly-arbitrary soroban-sdk-using code makes it appear. |
|
I believe this is #1011 which is, coincidentally, the next thing on my list. We should coordinate :) |
|
Running cargo +nightly fuzz run fuzz_target_1 --sanitizer=threadProduces fuzz_target_1(99375,0x1f00d9e00) malloc: nano zone abandoned due to inability to reserve vm space.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3619271091
INFO: Loaded 1 modules (157408 inline 8-bit counters): 157408 [0x1030beaa8, 0x1030e5188),
INFO: Loaded 1 PC tables (157408 PCs): 157408 [0x1030e5188,0x10334bf88),
INFO: 3 files found in /Users/tylervanderhoeven/Downloads/soroban-examples-0.9.2/fuzzing/fuzz/corpus/fuzz_target_1
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 3 min: 32b max: 32b total: 96b rss: 66Mb
#4 INITED cov: 3452 ft: 3529 corp: 3/96b exec/s: 0 rss: 85Mb
#5 NEW cov: 3452 ft: 3542 corp: 4/128b lim: 32 exec/s: 0 rss: 85Mb L: 32/32 MS: 1 ChangeBinInt-
==99375== ERROR: libFuzzer: deadly signal
#0 __sanitizer_print_stack_trace <null>:103817808 (librustc-nightly_rt.tsan.dylib:arm64+0x4df6c)
#1 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009b1a8c)
#2 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009a7c20)
#3 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, int, __sanitizer::__sanitizer_siginfo_pad*, void*) <null>:103817808 (librustc-nightly_rt.tsan.dylib:arm64+0xb384)
#4 sighandler(int, __sanitizer::__sanitizer_siginfo_pad*, void*) <null>:103817808 (librustc-nightly_rt.tsan.dylib:arm64+0xb7f8)
#5 _sigtramp <null>:99638192 (libsystem_platform.dylib:arm64+0x3a20)
#6 <null> <null>
#7 <null> <null>
#8 abort <null>:99627872 (libsystem_c.dylib:arm64+0x76ae4)
#9 <null> <null> (0x4a0a8001037c2d74)
#10 <null> <null>:104865024 (fuzz_target_1:arm64+0x1001fcb84)
#11 <null> <null>:104865024 (fuzz_target_1:arm64+0x1001fcb78)
#12 <null> <null>:104865024 (fuzz_target_1:arm64+0x100573984)
#13 <null> <null>:104865024 (fuzz_target_1:arm64+0x100577e64)
#14 <null> <null>:104865024 (fuzz_target_1:arm64+0x100577c0c)
#15 <null> <null>:104865024 (fuzz_target_1:arm64+0x100577b80)
#16 <null> <null>:104865024 (fuzz_target_1:arm64+0x100577b74)
#17 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009d1378)
#18 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003ee7a8)
#19 <null> <null>:104865024 (fuzz_target_1:arm64+0x10045b610)
#20 <null> <null>:104865024 (fuzz_target_1:arm64+0x10045b5d8)
#21 <null> <null>:104865024 (fuzz_target_1:arm64+0x100460fe0)
#22 <null> <null>:104865024 (fuzz_target_1:arm64+0x100460f68)
#23 <null> <null>:104865024 (fuzz_target_1:arm64+0x100466908)
#24 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003f082c)
#25 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003f064c)
#26 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003f03e0)
#27 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003f02ec)
#28 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003f2140)
#29 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003f2084)
#30 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003f1f98)
#31 <null> <null>:104865024 (fuzz_target_1:arm64+0x100004e64)
#32 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002eec80)
#33 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002eebdc)
#34 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002eeb90)
#35 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002eeb28)
#36 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002ee958)
#37 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002ee3b0)
#38 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002e8120)
#39 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002e75ec)
#40 <null> <null>:104865024 (fuzz_target_1:arm64+0x1002e6720)
#41 <null> <null>:104865024 (fuzz_target_1:arm64+0x100349c38)
#42 <null> <null>:104865024 (fuzz_target_1:arm64+0x1004610c4)
#43 <null> <null>:104865024 (fuzz_target_1:arm64+0x100460fdc)
#44 <null> <null>:104865024 (fuzz_target_1:arm64+0x100460f68)
#45 <null> <null>:104865024 (fuzz_target_1:arm64+0x1003f1f18)
#46 <null> <null>:104865024 (fuzz_target_1:arm64+0x10000889c)
#47 <null> <null>:104865024 (fuzz_target_1:arm64+0x1000063f0)
#48 <null> <null>:104865024 (fuzz_target_1:arm64+0x100006358)
#49 <null> <null>:104865024 (fuzz_target_1:arm64+0x1000053a0)
#50 <null> <null>:104865024 (fuzz_target_1:arm64+0x1000086a4)
#51 <null> <null>:104865024 (fuzz_target_1:arm64+0x100008500)
#52 <null> <null>:104865024 (fuzz_target_1:arm64+0x10018fcb4)
#53 <null> <null>:104865024 (fuzz_target_1:arm64+0x10018ff68)
#54 <null> <null>:104865024 (fuzz_target_1:arm64+0x10018fda8)
#55 <null> <null>:104865024 (fuzz_target_1:arm64+0x10018fd14)
#56 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009a9088)
#57 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009a88c4)
#58 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009a9eb4)
#59 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009aa794)
#60 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009c1168)
#61 <null> <null>:104865024 (fuzz_target_1:arm64+0x1009cb81c)
#62 <null> <null> (0x000194da7f28)
#63 <null> <null> (0xe913000000000000)
NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 1 ChangeBinInt-; base unit: 11c60798a6766f4acd545178867e15eaad87e30d
0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0xdd,0xd3,0xd3,0xd3,0xd3,0xd3,0xd3,0xd3,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0x2c,0xa,
,,,,,,,,,,,,,,\335\323\323\323\323\323\323\323,,,,,,,,,\012
artifact_prefix='/Users/tylervanderhoeven/Downloads/soroban-examples-0.9.2/fuzzing/fuzz/artifacts/fuzz_target_1/'; Test unit written to /Users/tylervanderhoeven/Downloads/soroban-examples-0.9.2/fuzzing/fuzz/artifacts/fuzz_target_1/crash-112e0e539cf5f8915fe9af5fc30e882669928608
Base64: LCwsLCwsLCwsLCwsLCzd09PT09PT0ywsLCwsLCwsLAo=
────────────────────────────────────────────────────────────────────────────────
Failing input:
fuzz/artifacts/fuzz_target_1/crash-112e0e539cf5f8915fe9af5fc30e882669928608
Reproduce with:
cargo fuzz run --sanitizer=thread fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-112e0e539cf5f8915fe9af5fc30e882669928608
Minimize test case with:
cargo fuzz tmin --sanitizer=thread fuzz_target_1 fuzz/artifacts/fuzz_target_1/crash-112e0e539cf5f8915fe9af5fc30e882669928608
────────────────────────────────────────────────────────────────────────────────
Error: Fuzz target exited with exit status: 77I get something similar but slightly longer for fuzz_target_2(99581,0x1f00d9e00) malloc: nano zone abandoned due to inability to reserve vm space.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3717215921
INFO: Loaded 1 modules (158141 inline 8-bit counters): 158141 [0x10386ab08, 0x1038914c5),
INFO: Loaded 1 PC tables (158141 PCs): 158141 [0x1038914c8,0x103afb098),
INFO: 7 files found in /Users/tylervanderhoeven/Downloads/soroban-examples-0.9.2/fuzzing/fuzz/corpus/fuzz_target_2
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 7 min: 80b max: 82b total: 563b rss: 66Mb
#8 INITED cov: 2844 ft: 2856 corp: 3/240b exec/s: 0 rss: 82Mb
#27 NEW cov: 2845 ft: 2857 corp: 4/320b lim: 80 exec/s: 0 rss: 82Mb L: 80/80 MS: 4 ShuffleBytes-ChangeByte-ShuffleBytes-CopyPart-
NEW_FUNC[1/3]: 0x102fde364 (fuzz_target_2:arm64+0x10021e364)
NEW_FUNC[2/3]: 0x102fde2a8 (fuzz_target_2:arm64+0x10021e2a8)
#941 NEW cov: 2854 ft: 2866 corp: 5/402b lim: 86 exec/s: 470 rss: 83Mb L: 82/82 MS: 3 CopyPart-ChangeBit-ChangeBit-
#2048 pulse cov: 2854 ft: 2866 corp: 5/402b lim: 92 exec/s: 682 rss: 83Mb
#4096 pulse cov: 2854 ft: 2866 corp: 5/402b lim: 116 exec/s: 819 rss: 83Mb
==99581== ERROR: libFuzzer: deadly signal
#0 __sanitizer_print_stack_trace <null>:112206416 (librustc-nightly_rt.tsan.dylib:arm64+0x4df6c)
#1 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009bd598)
#2 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009b372c)
#3 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, int, __sanitizer::__sanitizer_siginfo_pad*, void*) <null>:112206416 (librustc-nightly_rt.tsan.dylib:arm64+0xb384)
#4 sighandler(int, __sanitizer::__sanitizer_siginfo_pad*, void*) <null>:112206416 (librustc-nightly_rt.tsan.dylib:arm64+0xb7f8)
#5 _sigtramp <null>:108026800 (libsystem_platform.dylib:arm64+0x3a20)
#6 <null> <null> (0x24758001950ffc28)
#7 <null> <null>
#8 abort <null>:108016384 (libsystem_c.dylib:arm64+0x76ae4)
#9 <null> <null> (0x229000103f72d74)
#10 <null> <null>:113253632 (fuzz_target_2:arm64+0x100207ad8)
#11 <null> <null>:113253632 (fuzz_target_2:arm64+0x100207acc)
#12 <null> <null>:113253632 (fuzz_target_2:arm64+0x10057ec54)
#13 <null> <null>:113253632 (fuzz_target_2:arm64+0x100583134)
#14 <null> <null>:113253632 (fuzz_target_2:arm64+0x100582ebc)
#15 <null> <null>:113253632 (fuzz_target_2:arm64+0x100582e50)
#16 <null> <null>:113253632 (fuzz_target_2:arm64+0x100582e44)
#17 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009dcf1c)
#18 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009dd05c)
#19 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003fa2a8)
#20 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003f9e60)
#21 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003f9d54)
#22 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003f9c40)
#23 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003f9ba4)
#24 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003fd314)
#25 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003fd258)
#26 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003fd16c)
#27 <null> <null>:113253632 (fuzz_target_2:arm64+0x100005e24)
#28 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f9c5c)
#29 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f9bb8)
#30 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f9b6c)
#31 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f9b04)
#32 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f9934)
#33 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f938c)
#34 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f30f8)
#35 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f25c4)
#36 <null> <null>:113253632 (fuzz_target_2:arm64+0x1002f16f8)
#37 <null> <null>:113253632 (fuzz_target_2:arm64+0x100354d3c)
#38 <null> <null>:113253632 (fuzz_target_2:arm64+0x10046c2c4)
#39 <null> <null>:113253632 (fuzz_target_2:arm64+0x10046c1dc)
#40 <null> <null>:113253632 (fuzz_target_2:arm64+0x10046c168)
#41 <null> <null>:113253632 (fuzz_target_2:arm64+0x1003fcbbc)
#42 <null> <null>:113253632 (fuzz_target_2:arm64+0x10000fddc)
#43 <null> <null>:113253632 (fuzz_target_2:arm64+0x1000089a4)
#44 <null> <null>:113253632 (fuzz_target_2:arm64+0x100008840)
#45 <null> <null>:113253632 (fuzz_target_2:arm64+0x100006980)
#46 <null> <null>:113253632 (fuzz_target_2:arm64+0x10000fd18)
#47 <null> <null>:113253632 (fuzz_target_2:arm64+0x10000f81c)
#48 <null> <null>:113253632 (fuzz_target_2:arm64+0x10001371c)
#49 <null> <null>:113253632 (fuzz_target_2:arm64+0x100013508)
#50 <null> <null>:113253632 (fuzz_target_2:arm64+0x10019ac60)
#51 <null> <null>:113253632 (fuzz_target_2:arm64+0x10019af14)
#52 <null> <null>:113253632 (fuzz_target_2:arm64+0x10019ad54)
#53 <null> <null>:113253632 (fuzz_target_2:arm64+0x10019acc0)
#54 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009b4b94)
#55 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009b43d0)
#56 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009b59c0)
#57 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009b62a0)
#58 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009ccc74)
#59 <null> <null>:113253632 (fuzz_target_2:arm64+0x1009d7328)
#60 <null> <null> (0x000194da7f28)
#61 <null> <null> (0x8672000000000000)
NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 3 ChangeBit-CrossOver-InsertRepeatedBytes-; base unit: 0fc16150022a500b675745ecd9f5b7b89e18df10
0x78,0x38,0x38,0x10,0x0,0x0,0x0,0x20,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x38,0x0,0x0,0x3d,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x38,0x38,0x38,0x38,0x0,
x88\020\000\000\000 \000\000\000\000\000\000\0008888888888888888888888888\000\000=\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\000\000\000\000\000\000\000\0008888\000
artifact_prefix='/Users/tylervanderhoeven/Downloads/soroban-examples-0.9.2/fuzzing/fuzz/artifacts/fuzz_target_2/'; Test unit written to /Users/tylervanderhoeven/Downloads/soroban-examples-0.9.2/fuzzing/fuzz/artifacts/fuzz_target_2/crash-83ac759c039c37f95eaa479c5a9c2fa30831d0ff
Base64: eDg4EAAAACAAAAAAAAAAODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4OAAAPQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA////////////////////////////////////////////////////////////////////////////////////////////////AAAAAAAAAAA4ODg4AA==
────────────────────────────────────────────────────────────────────────────────
Failing input:
fuzz/artifacts/fuzz_target_2/crash-83ac759c039c37f95eaa479c5a9c2fa30831d0ff
Reproduce with:
cargo fuzz run --sanitizer=thread fuzz_target_2 fuzz/artifacts/fuzz_target_2/crash-83ac759c039c37f95eaa479c5a9c2fa30831d0ff
Minimize test case with:
cargo fuzz tmin --sanitizer=thread fuzz_target_2 fuzz/artifacts/fuzz_target_2/crash-83ac759c039c37f95eaa479c5a9c2fa30831d0ff
────────────────────────────────────────────────────────────────────────────────
Error: Fuzz target exited with exit status: 77As I'm very new to fuzzing I don't know if this is the expected output but I'm doubtful. |
|
Confirming the error I shared with @brson earlier is now working: Using the following workaround: I'm running on an intel based mac, using the fuzz test here -> https://github.com/blend-capital/blend-contracts/blob/b-fuzz/test-suites/fuzz/fuzz_targets/fuzz_target_2.rs I am also noticing a slowdown on my mac vs my linux machine without the sanitizer. This could be due to higher specs on the linux machine, but thought it was worth noting. |
This does look correct to me, though ugly because the stack frame symbols haven't been resolved. The fuzzer requires the |
|
Also, both of the fuzzer examples are expected to find a bug - the example contains an intentional bug. |
They are tantalizingly similar. I'll test the fuzzer on linux-aarch64 and see if I also get linker errors. It looks like all my development so far has been on linux-x86. |
|
yeah this is .. hmm .. not certainly the same bug. I have an interesting datapoint from today re #1011 which is that it only happens when crossing from x64-linux host to aarch64-linux target. On a native aarch64-linux host (I rented a graviton machine for a bit) it does not occur. So that at least is something hinky with cross configs of the toolchain. (Of course, I still do not know if this bug is actually #1011 or not. Feel free to investigate in parallel while I look into it! I will try to keep you appraised of anything I learn) |
and others
There have been several reports that attempting to compile a
cargo-fuzztest on macos fails to link. Possibly arm-specific.Discord thread: https://discord.com/channels/897514728459468821/1141102329085567048/1141102329085567048
Some examples:
I don't see anything obviously related on the cargo-fuzz issue tracker.
The text was updated successfully, but these errors were encountered: