Fuzzer fails immediately due to abseil: AddressSanitizer: SEGV

[ibc]

Your environment

• Operating system: macOS Intel with Docker
• mediasoup version: 3.13.19

Issue description

root@143c5f473744:/mediasoup/worker# make fuzzer-run-all

"/usr/bin/python3" -m invoke fuzzer-run-all
cd "/mediasoup/worker" && LSAN_OPTIONS=verbosity=1:log_threads=1 "/mediasoup/worker/out/Release/build/mediasoup-worker-fuzzer" -artifact_prefix=fuzzer/reports/ -max_len=1400 fuzzer/new-corpus deps/webrtc-fuzzer-corpora/corpora/stun-corpus deps/webrtc-fuzzer-corpora/corpora/rtp-corpus deps/webrtc-fuzzer-corpora/corpora/rtcp-corpus
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__isoc99_printf'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__isoc99_sprintf'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__isoc99_snprintf'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__isoc99_fprintf'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__isoc99_vprintf'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__isoc99_vsprintf'
==3079==Registered root region at 0x7f5179f01ab0 of size 112
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f5179f01ab0 of size 112
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__isoc99_vsnprintf'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__isoc99_vfprintf'
==3079==Registered root region at 0x7f5179f01ab0 of size 112
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f5179f01ab0 of size 112
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept 'pthread_mutexattr_getrobust_np'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept 'xdr_quad_t'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept 'xdr_u_quad_t'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept 'xdr_destroy'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept 'crypt'
==3079==Registered root region at 0x7f517a5016e0 of size 96
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f517a5016e0 of size 96
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept 'crypt_r'
==3079==Registered root region at 0x7f5179f01ab0 of size 112
==3079==Registered root region at 0x7f517a1007a0 of size 32
==3079==Unregistered root region at 0x7f5179f01ab0 of size 112
==3079==Unregistered root region at 0x7f517a1007a0 of size 32
==3079==AddressSanitizer: failed to intercept '__cxa_rethrow_primary_exception'
==3079==AddressSanitizer: libc interceptors initialized
|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap  ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem     ||
MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
thread_local_quarantine_size_kb=1024K
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 0x7fff8000
==3079==Installed the sigaction for signal 11
==3079==Installed the sigaction for signal 7
==3079==Installed the sigaction for signal 8
==3079==T0: stack [0x7ffe8a861000,0x7ffe8b061000) size 0x800000; local=0x7ffe8b05ff94
==3079==AddressSanitizer Init done
AddressSanitizer:DEADLYSIGNAL
=================================================================
==3079==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x55dab3f0cc60 bp 0x7ffe8b05f3e0 sp 0x7ffe8b05f3c0 T0)
==3079==The signal is caused by a READ memory access.
==3079==Hint: address points to the zero page.
    #0 0x55dab3f0cc60 in absl::lts_20230802::container_internal::CommonFieldsGenerationInfoEnabled::generation() const /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/internal/raw_hash_set.h:821:39
    #1 0x55dab3f0c6df in void absl::lts_20230802::container_internal::InitializeSlots<std::allocator<char>, 40ul, 8ul>(absl::lts_20230802::container_internal::CommonFields&, std::allocator<char>) /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/internal/raw_hash_set.h:1408:43
    #2 0x55dab3f3fc45 in absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashMapPolicy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel>, absl::lts_20230802::container_internal::StringHash, absl::lts_20230802::container_internal::StringEq, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > >::initialize_slots() /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/internal/raw_hash_set.h:2505:5
    #3 0x55dab3f3f17b in absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashMapPolicy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel>, absl::lts_20230802::container_internal::StringHash, absl::lts_20230802::container_internal::StringEq, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > >::raw_hash_set(unsigned long, absl::lts_20230802::container_internal::StringHash const&, absl::lts_20230802::container_internal::StringEq const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > const&) /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/internal/raw_hash_set.h:1721:7
    #4 0x55dab3f3ed77 in absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashMapPolicy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel>, absl::lts_20230802::container_internal::StringHash, absl::lts_20230802::container_internal::StringEq, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > >::raw_hash_set<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel> const*>(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel> const*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel> const*, unsigned long, absl::lts_20230802::container_internal::StringHash const&, absl::lts_20230802::container_internal::StringEq const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > const&) /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/internal/raw_hash_set.h:1739:9
    #5 0x55dab3f3eb74 in absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashMapPolicy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel>, absl::lts_20230802::container_internal::StringHash, absl::lts_20230802::container_internal::StringEq, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > >::raw_hash_set(std::initializer_list<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel> >, unsigned long, absl::lts_20230802::container_internal::StringHash const&, absl::lts_20230802::container_internal::StringEq const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > const&) /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/internal/raw_hash_set.h:1788:9
    #6 0x55dab3f3e907 in absl::lts_20230802::container_internal::raw_hash_map<absl::lts_20230802::container_internal::FlatHashMapPolicy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel>, absl::lts_20230802::container_internal::StringHash, absl::lts_20230802::container_internal::StringEq, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > >::raw_hash_map(std::initializer_list<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel> >, unsigned long, absl::lts_20230802::container_internal::StringHash const&, absl::lts_20230802::container_internal::StringEq const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > const&) /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/internal/raw_hash_map.h:63:37
    #7 0x55dab3f32447 in absl::lts_20230802::flat_hash_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel, absl::lts_20230802::container_internal::StringHash, absl::lts_20230802::container_internal::StringEq, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > >::flat_hash_map(std::initializer_list<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, LogLevel> >, unsigned long, absl::lts_20230802::container_internal::StringHash const&, absl::lts_20230802::container_internal::StringEq const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, LogLevel> > const&) /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/flat_hash_map.h:160:15
    #8 0x55dab3df52db in __cxx_global_var_init.2 /mediasoup/worker/out/Release/build/../../../src/Settings.cpp:28:1
    #9 0x55dab3df60f5 in _GLOBAL__sub_I_Settings.cpp /mediasoup/worker/out/Release/build/../../../src/Settings.cpp
    #10 0x7f517aaa5eba in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29eba) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #11 0x55dab3e0ebc4 in _start (/mediasoup/worker/out/Release/build/mediasoup-worker-fuzzer+0x258bc4) (BuildId: 984892a760b005a230b8681ccdbd85f26b4c7f76)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /mediasoup/worker/out/Release/build/../../../subprojects/abseil-cpp-20230802.0/absl/container/internal/raw_hash_set.h:821:39 in absl::lts_20230802::container_internal::CommonFieldsGenerationInfoEnabled::generation() const
==3079==ABORTING
make: *** [Makefile:109: fuzzer-run-all] Error 1

[ibc]

It's basically failing here, in Settings.cpp:

absl::flat_hash_map<std::string, LogLevel> Settings::String2LogLevel =
{
	{ "debug", LogLevel::LOG_DEBUG },
	{ "warn",  LogLevel::LOG_WARN  },
	{ "error", LogLevel::LOG_ERROR },
	{ "none",  LogLevel::LOG_NONE  }
};

which BTW is perfectly valid according to docs: https://abseil.io/docs/cpp/guides/container#construction

[jmillan]

The error does not happen if this line is removed from meson.build.

Here few places with some info.

• Three NULL Pointer Dereference bugs found in re2-2023-09-01 google/re2#451
• [Bug]: SIGSEGV from absl::flat_hash_map::insert() abseil/abseil-cpp#1524 (comment)

[jmillan]

I'm going to try with a newest clang version.

EDIT: same error with clang version: Ubuntu clang version 16.0.6 (15), in the latest stable ubuntu (23.10)

[jmillan]

Why are we using abseil to just hold a map or string->integer?

I don't know, but remove it and the error will raise somewhere else.

[jmillan]

Apparently those flags (-fsanitize,fuzzer) need to be propagated everywhere including the abseil dependency.

[jmillan]

I'm working on the fix. I'll open a separate PR.

[ibc]

I'm pretty sure that if we remove '-fsanitize=address,fuzzer' then we are not fuzzing anything XD

Apparently those flags (-fsanitize,fuzzer) need to be propagated everywhere including the abseil dependency.

And how can we do that? Can we set some env or variable in meson.build that also makes other subprojects receive those C flags?

[jmillan]

'-fsanitize=address,fuzzer' then we are not fuzzing anything XD

Of course :-), but it gave me the clue of what was happening.

[ibc]

This issue blocks PR #1338 for obvious reasons :)

[jmillan]

Yes, I'll do a PR to wrapdb today

[jmillan]

PR mesonbuild/wrapdb#1412

[ibc]

Note: fixing this in PR #1338 as a bonus.