Zeek ASAN build with builtin-spicy/stack-use-after-scope: __fiber_switch_trampoline

[awelzel]

Hey - running an ASAN build of latest Zeek with builtin spicy, I'm running into stack-user-after-scope splashes..

I've double checked the environment and it should be clean, so I'm a bit at loss what's going on here.

$ which zeek; which spicyz; zeek --version; spicyz --version
/opt/zeek-dev/bin/zeek                                                                                                                                                  
/opt/zeek-dev/bin/spicyz                                                                                                                                                
zeek version 5.2.0-dev.398-debug                                                    
1.4.0  
# Within a spicy-http checkout
$ rm -rf build && mkdir build && (cd build && cmake ../ && make -j4)
$ ZEEK_SPICY_MODULE_PATH=$(pwd)/build/spicy-modules zeek -r tests/traces/http-post.pcap 
=================================================================
==3905188==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f4eb315d3e0 at pc 0x5567a793450b bp 0x7f4eb315d020 sp 0x7f4eb315d018
WRITE of size 8 at 0x7f4eb315d3e0 thread T0
    #0 0x5567a793450a in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider::_Alloc_hider(char*, std::allocator<char> const&) /usr/include/c++/10/bits/basic_string.h:157
    #1 0x5567a793450a in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) /usr/include/c++/10/bits/basic_string.h:526
    #2 0x5567a793450a in __fiber_switch_trampoline ../auxil/spicy/spicy/hilti/runtime/src/fiber.cc:122
    #3 0x5567aa9292f1 in fiber_asm_invoke (/opt/zeek-dev/bin/zeek+0x81482f1)

0x7f4eb315d3e0 is located 1047520 bytes inside of 1048576-byte region [0x7f4eb305d800,0x7f4eb315d800)
allocated by thread T0 here:
    #0 0x7f4eb9739e8f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5567aa928f49 in fiber_alloc ../auxil/spicy/spicy/3rdparty/fiber/src/fiber.c:207
    #2 0x5567a792699c in hilti::rt::detail::Fiber::Fiber(hilti::rt::detail::Fiber::Type) ../auxil/spicy/spicy/hilti/runtime/src/fiber.cc:171
    #3 0x5567a7928dfb in std::_MakeUniq<hilti::rt::detail::Fiber>::__single_object std::make_unique<hilti::rt::detail::Fiber, hilti::rt::detail::Fiber::Type>(hilti::rt::detail::Fiber::Type&&) /usr/include/c++/10/bits/unique_ptr.h:962
    #4 0x5567a7928dfb in hilti::rt::detail::FiberContext::FiberContext() ../auxil/spicy/spicy/hilti/runtime/src/fiber.cc:140
    #5 0x5567a7913d74 in hilti::rt::Context::Context(long) ../auxil/spicy/spicy/hilti/runtime/src/context.cc:28
    #6 0x5567a795f875 in std::_MakeUniq<hilti::rt::Context>::__single_object std::make_unique<hilti::rt::Context, long const&>(long const&) /usr/include/c++/10/bits/unique_ptr.h:962
    #7 0x5567a795f875 in hilti::rt::init() ../auxil/spicy/spicy/hilti/runtime/src/init.cc:39
    #8 0x5567a77bba35 in plugin::Zeek_Spicy::Plugin::InitPostScript() ../auxil/spicy-plugin/src/plugin.cc:656
    #9 0x5567a6640bce in zeek::plugin::Manager::InitPostScript() ../src/plugin/Manager.cc:534
    #10 0x5567a5f5c948 in zeek::detail::setup(int, char**, zeek::Options*) ../src/zeek-setup.cc:837
    #11 0x5567a7ad54a0 in main ../src/main.cc:23
    #12 0x7f4eb8d91d09 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: stack-use-after-scope /usr/include/c++/10/bits/basic_string.h:157 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Alloc_hider::_Alloc_hider(char*, std::allocator<char> const&)
Shadow bytes around the buggy address:
  0x0fea56623a20: 01 f2 01 f2 01 f2 01 f2 00 f2 f2 f2 00 f2 f2 f2
  0x0fea56623a30: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
  0x0fea56623a40: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
  0x0fea56623a50: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
  0x0fea56623a60: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
=>0x0fea56623a70: 00 f2 f2 f2 00 00 f2 f2 00 00 f2 f2[f8]f8 f8 f8
  0x0fea56623a80: f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00 00 00
  0x0fea56623a90: f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00 00 00 00
  0x0fea56623aa0: f2 f2 f2 f2 00 00 00 00 00 00 00 00 f2 f2 f2 f2
  0x0fea56623ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fea56623ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==3905188==ABORTING

Distro:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Zeek compile:

export CFLAGS="-ggdb -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
export CXXFLAGS="-ggdb -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
export LDFLAGS="-fuse-ld=lld"

./configure \
    --ccache \
    --generator=Ninja \
    --build-dir=./build \
    --prefix=$PREFIX \
    --enable-debug \
    --build-type=debug \
    --disable-zkg \
    --disable-btest \
    --disable-python \
    --disable-zeekctl \
    --disable-zeek-client \
    --disable-broker-tests \
    --enable-static-binpac  \
    --enable-static-broker \
    --binary-package \
    --sanitizers=address \
    --enable-fuzzers \
    $*

Linker and GCC:
$ ld.lld --version
Debian LLD 15.0.0 (compatible with GNU linkers)
$ gcc --version
gcc (Debian 10.2.1-6) 10.2.1 20210110
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[bbannier]

The fiber code in Spicy seemed to trip ASAN up in the past so we use a different implementation if running under a ASAN:

spicy/hilti/runtime/src/fiber.cc

Lines 31 to 40 in 9dfc953

	// Because the stack copying triggers false positives with ASAN, we use
	// individual stacks when that's active. We still force use of the stack
	// switcher trampoline in that case, so that we get that piece at least.
	//
	// TODO: If we could whitelist the memcpys, that would solve the problem, but I
	// haven't been able to do that using any of the sanitizer attributes; they
	// just seem to be ignored.
	static const auto DefaultFiberType = detail::Fiber::Type::IndividualStack;
	static const auto AlwaysUseStackSwitchTrampoline = true;
	static const auto FiberGuardFlags = 0; // leak sanitizer may abort with "Tracer caught signal 11" if pages get protected

.

All this code should only be active if we run with ASAN, see its detection and us setting HILTI_HAVE_ASAN:

spicy/hilti/runtime/include/config.h.in

Lines 14 to 23 in 9dfc953

	// GCC uses __SANITIZE_ADDRESS__, Clang uses __has_feature.
	#if defined(__SANITIZE_ADDRESS__)
	#define HILTI_HAVE_ASAN
	#endif
	#if defined(__has_feature)
	#if __has_feature(address_sanitizer)
	#define HILTI_HAVE_ASAN
	#endif
	#endif

Something seems to go wrong here.

[awelzel]

Ran into this again while trying to actually run the test cases added in zeek/zeek#2651.

@rsmmr - any chance you were using ASAN for prepping that branch and this is something else on my end?

Ah, seems the ASAN builds of that branch aren't happy, so probably not just me.

[rsmmr]

Doing some tests on macOS and Linux, I think this is a GCC issue: I'm getting no errors with clang on either macOS nor Linux; but I do indeed see this with GCC on Linux. Not sure yet what's going on there, I thought the ASAN libraries were compatible between the two. Will explore some more.