Oss-Fuzz builds failing for Tor

[nmathewson]

Hi! The fuzzers are crashing with a weird error on startup for Tor:

Step #5: ============================================================ T= 1567657161
Step #5: Tor 0.4.2.0-alpha-dev (git-2ff6ebc6fc3b3beb) died: Caught signal 11
Step #5: /workspace/out/address/oss-fuzz-consensus(backtrace+0x5b)[0x4e2adb]
Step #5: [0x0]
Step #5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f449f9a2390]
Step #5: AddressSanitizer:DEADLYSIGNAL
Step #5: =================================================================
Step #5: ==503==ERROR: AddressSanitizer: ABRT on unknown address 0x0000000001f7 (pc 0x7f449efda428 bp 0x7ffc10a4d030 sp 0x7ffc10a4ce88 T0)
Step #5: SCARINESS: 10 (signal)
Step #5:     #0 0x7f449efda427 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35427)
Step #5:     #1 0x7f449efdc029 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x37029)
Step #5:     #2 0x9408eb in crash_handler /src/tor/src/lib/err/backtrace.c:175:3
Step #5:     #3 0x7f449f9a238f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1138f)
Step #5: 
Step #5: AddressSanitizer can not provide additional info.
Step #5: SUMMARY: AddressSanitizer: ABRT (/lib/x86_64-linux-gnu/libc.so.6+0x35427) in gsignal
Step #5: ==503==ABORTING
Step #5: MS: 0 ; base unit: 0000000000000000000000000000000000000000
Step #5: 
Step #5: 
Step #5: artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Step #5: Base64: 
Step #5: oss-fuzz-descriptor:
Step #5: BAD BUILD: /workspace/out/address/oss-fuzz-descriptor seems to have either startup crash or exit:
Step #5: INFO: Seed: 2687920607
Step #5: INFO: Loaded 1 modules   (112728 inline 8-bit counters): 112728 [0x169b750, 0x16b6fa8), 
Step #5: INFO: Loaded 1 PC tables (112728 PCs): 112728 [0x16b6fa8,0x186f528), 
Step #5: INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes

(From https://oss-fuzz-build-logs.storage.googleapis.com/log-02b946c3-e4ad-42e6-bd96-6d78861d0a97.txt )

This does not happen when I try to build on my own with libfuzzer. Any insight here?

[jonathanmetzman]

I can repro this if I build tor's fuzzers using the check_build command.
It's possible you cannot repro this because your clang is older than the one used by OSS-Fuzz.

The stack given by ASAN looks pretty useless here. gdb on the other hand says this is the stack:

#0  0x0000000000000000 in ?? ()
#1  0x00000000005042d3 in __interceptor_getrandom () at /src/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:9557
#2  0x0000000000e37518 in arc4_seed_getrandom () at ././arc4random.c:188
#3  0x0000000000e372a4 in arc4_seed () at ././arc4random.c:344
#4  0x0000000000e36ed8 in arc4_stir () at ././arc4random.c:373
#5  0x0000000000e36e10 in evutil_secure_rng_init () at evutil_rand.c:174
#6  0x0000000000871d77 in tor_init_libevent_rng () at src/lib/evloop/compat_libevent.c:471
#7  0x00000000008721b4 in subsys_evloop_initialize () at src/lib/evloop/evloop_sys.c:21
#8  0x000000000058ca1a in subsystems_init_upto () at src/app/main/subsysmgr.c:100
#9  0x0000000000556521 in global_init () at src/test/fuzz/fuzzing_common.c:100
#10 0x000000000055642f in LLVMFuzzerTestOneInput () at src/test/fuzz/fuzzing_common.c:132
#11 0x000000000045cfc2 in ExecuteCallback () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:554
#12 0x000000000045eed6 in ReadAndExecuteSeedCorpora () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:741
#13 0x000000000045f4aa in Loop () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:794
#14 0x000000000044d609 in FuzzerDriver () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:825
#15 0x0000000000476e93 in main () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19

Is there anything actionable here? Or does it look like an ASAN bug?

[nmathewson]

Hm. It looks like the crash is happening inside libevent, when libevent tries to call getrandom(), pointing at an address on the stack. The libevent code looks a little weird here, but doesn't look like it should crash. (Fortunately, libevent's prng is only used for generating DNS transaction IDs.)

Conceivably it could be an ASAN bug -- I don't have much experience with finding those, however.

[jonathanmetzman]

Hmm...there's a similar crash in libsodium.
Gdb says that stack is:

#0  0x0000000000000000 in ?? ()
#1  0x0000000000501413 in __interceptor_getrandom () at /src/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:9557                                                                                                    
#2  0x0000000000557ac1 in _randombytes_linux_getrandom () at randombytes/sysrandom/randombytes_sysrandom.c:233
#3  0x0000000000557622 in randombytes_linux_getrandom () at randombytes/sysrandom/randombytes_sysrandom.c:250
#4  0x0000000000557467 in randombytes_sysrandom_init () at randombytes/sysrandom/randombytes_sysrandom.c:270
#5  0x00000000005571ca in randombytes_sysrandom_stir () at randombytes/sysrandom/randombytes_sysrandom.c:298
#6  0x00000000005546a3 in randombytes_stir () at randombytes/randombytes.c:82
#7  0x0000000000554a90 in sodium_init () at sodium/core.c:40
#8  0x00000000005533ce in LLVMFuzzerTestOneInput () at /src/secret_key_auth_fuzzer.cc:7
#9  0x000000000045a102 in ExecuteCallback () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:554
#10 0x000000000045c016 in ReadAndExecuteSeedCorpora () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:741
#11 0x000000000045c5ea in Loop () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:794
#12 0x000000000044a749 in FuzzerDriver () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:825
#13 0x0000000000473fd3 in main () at /src/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19

@guidovranken did you say you thought this was an ASAN bug? Did you solve the issue work around it in cryptofuzz?

cc @derwolfe

[guidovranken]

@jonathanmetzman this is not the ASAN bug that we discussed (this caused ASAN on 32 bit to find spurious/non reproducible issues).

It is a libsodium problem: jedisct1/libsodium#859
edit: or an LLVM/santizer problem.
I've just disabled libsodium in Cryptofuzz for now.

[guidovranken]

I'm probably stating the obvious.. But since two distinct libraries crash at getrandom, the sanitizer probably forgets to overwrite the getrandom hook (both traces result in EIP = NULL).

[jonathanmetzman]

@jonathanmetzman this is not the ASAN bug that we discussed (this caused ASAN on 32 bit to find spurious/non reproducible issues).

Right. But didn't you mention this in a deleted comment on my PR to revert the clang roll?

@vitalybuka
Does it seem possible that llvm/llvm-project@ac9ee01#diff-483266ff26e7d1b17130f1b371c4e62d introduced this?

I haven't bisected this for sure and I don't have a minimum reproducer, but it looks like this could be at fault since the libsodium bug only started occurring when clang was rolled to 8288453f6aac05080b751b680455349e09d49825 but did not occur when clang was at f7e52fbdb5a7af8ea0808e98458b497125a5eca1

[jonathanmetzman]

CC @morehouse

[jonathanmetzman]

@vitalybuka
I was able to confirm llvm/llvm-project@ac9ee01#diff-483266ff26e7d1b17130f1b371c4e62d
indeed breaks these projects.

[jonathanmetzman]

@guidovranken It looks like jedisct1/libsodium@2032201 is wrong (libsodium build still failing). I think it is because they are not checking for ASAN usage properly. This is right way. But it might be worth seeing if clang folks can fix upstream before bugging libsodium

[jonathanmetzman]

Ping @vitalybuka

[jonathanmetzman]

I have a tiny reproducer here. It must be compiled on glibc 2.23 or earlier (and possibly some later, probably any that doesn't have getrandom defined but I make no guarantees):

// getrandom.c
#include <stdlib.h>

ssize_t getrandom(void *buf, size_t buflen, unsigned int flags);

int main() {
char buf[10];
getrandom((void*) buf, 10, 0);
return 0;
}

$ clang getrandom.c -fsanitize=address -o getrandom && ./getrandom
AddressSanitizer:DEADLYSIGNAL
=================================================================
==306==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0x7ffc95e83830 sp 0x7ffc95e82ff8 T0)                                                                                 
==306==Hint: pc points to the zero page.
==306==The signal is caused by a READ memory access.
==306==Hint: address points to the zero page.

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (<unknown module>)
==306==ABORTING

ASAN's interceptor seems to let this compile when it shouldn't:

clang getrandom.c -o getrandom && ./getrandom                                                                                                                                   
/tmp/getrandom-7f43fc.o: In function `main':
getrandom.c:(.text+0x1b): undefined reference to `getrandom'
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)

[vitalybuka]

I have a tiny reproducer here. It must be compiled on glibc 2.23 or earlier (and possibly some later, probably any that doesn't have getrandom defined but I make no guarantees):

That's probably true for other interceptors. Are you asking to use __GLIBC_PREREQ(2, 25) compile time to decide to have this interceptor?

[jonathanmetzman]

Are you asking to use __GLIBC_PREREQ(2, 25) compile time to decide to have this interceptor?

I guess so. Is this going to be done when clang is compiled (ie: if clang is built with glibc <2.25) or when the target project is compiled? Either should work for us.

[vitalybuka]

I guess so. Is this going to be done when clang is compiled (ie: if clang is built with glibc <2.25) or when the target project is compiled? Either should work for us.

compiler-rt

[jonathanmetzman]

Thanks @vitalybuka.

Looks like the patch just missed the chrome clang roll

@guidovranken @nmathewson this should get fixed on its own within a few weeks when chrome rolls clang again (we use Chrome's clang because they use very close to trunk but ensure it is stable). I may try to roll clang manually before that since the roll should be small but I'm not sure since now is a particularly bad time to break things for us because of other things going on right now.