TSAN error reported in WithBoundScheduler.EventWaitUntilTimeTaken/0

[ben-clayton]

Reproduction steps:

git clone https://github.com/google/marl.git
cd marl
git submodule update --init
mkdir build
cd build
cmake .. -GNinja -DMARL_TSAN=1 -DMARL_BUILD_TESTS=1
ninja
./marl-unittests --gtest_filter=SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 --gtest_repeat=-1

---

EventWaitUntilTimeTaken for a single threaded worker appears to fail after repeated runs of the same test:

[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
ThreadSanitizer:DEADLYSIGNAL
==79999==ERROR: ThreadSanitizer: SEGV on unknown address 0x0000000007f0 (pc 0x7f7a2bfb4f0e bp 0x600001000000 sp 0x7f75ff043610 T79999)
==79999==The signal is caused by a READ memory access.
==79999==Hint: address points to the zero page.
    #0 <null> <null> (libtsan.so.0+0x8cf0e)
    #1 <null> <null> (libtsan.so.0+0xa3b74)
    #2 __tsan_read8 <null> (libtsan.so.0+0x918f7)
    #3 marl::Scheduler::WaitingFibers::Timeout::operator<(marl::Scheduler::WaitingFibers::Timeout const&) const ../src/scheduler.cpp:354 (marl-unittests+0x13eadd)
    #4 std::less<marl::Scheduler::WaitingFibers::Timeout>::operator()(marl::Scheduler::WaitingFibers::Timeout const&, marl::Scheduler::WaitingFibers::Timeout const&) const /usr/include/c++/8/bits/stl_function.h:386 (marl-unittests+0x13eadd)
    #5 std::_Rb_tree<marl::Scheduler::WaitingFibers::Timeout, marl::Scheduler::WaitingFibers::Timeout, std::_Identity<marl::Scheduler::WaitingFibers::Timeout>, std::less<marl::Scheduler::WaitingFibers::Timeout>, std::allocator<marl::Scheduler::WaitingFibers::Timeout> >::_M_get_insert_unique_pos(marl::Scheduler::WaitingFibers::Timeout const&) /usr/include/c++/8/bits/stl_tree.h:2061 (marl-unittests+0x13eadd)
    #6 std::pair<std::_Rb_tree_iterator<marl::Scheduler::WaitingFibers::Timeout>, bool> std::_Rb_tree<marl::Scheduler::WaitingFibers::Timeout, marl::Scheduler::WaitingFibers::Timeout, std::_Identity<marl::Scheduler::WaitingFibers::Timeout>, std::less<marl::Scheduler::WaitingFibers::Timeout>, std::allocator<marl::Scheduler::WaitingFibers::Timeout> >::_M_emplace_unique<marl::Scheduler::WaitingFibers::Timeout>(marl::Scheduler::WaitingFibers::Timeout&&) /usr/include/c++/8/bits/stl_tree.h:2379 (marl-unittests+0x13eadd)
    #7 std::pair<std::_Rb_tree_const_iterator<marl::Scheduler::WaitingFibers::Timeout>, bool> std::set<marl::Scheduler::WaitingFibers::Timeout, std::less<marl::Scheduler::WaitingFibers::Timeout>, std::allocator<marl::Scheduler::WaitingFibers::Timeout> >::emplace<marl::Scheduler::WaitingFibers::Timeout>(marl::Scheduler::WaitingFibers::Timeout&&) /usr/include/c++/8/bits/stl_set.h:463 (marl-unittests+0x13cc4d)
    #8 marl::Scheduler::WaitingFibers::add(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, marl::Scheduler::Fiber*) ../src/scheduler.cpp:332 (marl-unittests+0x13cc4d)
    #9 marl::Scheduler::Worker::suspend(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*) ../src/scheduler.cpp:470 (marl-unittests+0x13cc4d)
    #10 marl::Scheduler::Worker::wait(marl::lock&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::function<bool ()> const&) ../src/scheduler.cpp:446 (marl-unittests+0x13d1e9)
    #11 bool marl::Scheduler::Fiber::wait<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(marl::lock&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, std::function<bool ()> const&) ../include/marl/scheduler.h:486 (marl-unittests+0xb0600)
    #12 bool marl::ConditionVariable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, marl::Event::Shared::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)::{lambda()#1}>(marl::lock&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, marl::Event::Shared::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)::{lambda()#1}&&) ../include/marl/conditionvariable.h:171 (marl-unittests+0xb0600)
    #13 bool marl::Event::Shared::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ../include/marl/event.h:170 (marl-unittests+0xa85d6)
    #14 bool marl::Event::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) const ../include/marl/event.h:205 (marl-unittests+0xa85d6)
    #15 operator() ../src/event_test.cpp:186 (marl-unittests+0xa85d6)
    #16 _M_invoke /usr/include/c++/8/bits/std_function.h:297 (marl-unittests+0xa85d6)
    #17 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (marl-unittests+0x13d36c)
    #18 marl::Task::operator()() const ../include/marl/task.h:94 (marl-unittests+0x132715)
    #19 marl::Scheduler::Worker::runUntilIdle() ../src/scheduler.cpp:702 (marl-unittests+0x132715)
    #20 marl::Scheduler::Worker::runUntilShutdown() ../src/scheduler.cpp:587 (marl-unittests+0x13886f)
    #21 marl::Scheduler::Worker::run() ../src/scheduler.cpp:580 (marl-unittests+0x13bc5a)
    #22 operator() ../src/scheduler.cpp:717 (marl-unittests+0x13c06d)
    #23 _M_invoke /usr/include/c++/8/bits/std_function.h:297 (marl-unittests+0x13c06d)
    #24 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (marl-unittests+0xc2e25)
    #25 marl::OSFiber::run(marl::OSFiber*) ../src/osfiber_asm.h:129 (marl-unittests+0xc2e25)
    #26 marl_fiber_trampoline ../src/osfiber_x64.c:20 (marl-unittests+0x141c1a)

ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV (/usr/lib/x86_64-linux-gnu/libtsan.so.0+0x8cf0e) 
==79999==ABORTING

[ben-clayton]

Using clang-10's TSAN, we get a more sensible error:

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
ThreadSanitizer:DEADLYSIGNAL
==8770==ERROR: ThreadSanitizer: SEGV on unknown address 0x0000000007f0 (pc 0x0000004cb766 bp 0x000000000020 sp 0x7fc945192610 T8770)
==8770==The signal is caused by a READ memory access.
==8770==Hint: address points to the zero page.
    #0 ExtractTagFromStack<__sanitizer::BufferedStackTrace> /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.h:638 (marl-unittests+0x4cb766)
    #1 ObtainCurrentStack<__sanitizer::BufferedStackTrace> /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.h:656 (marl-unittests+0x4cb766)
    #2 TraceSwitch /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:572 (marl-unittests+0x4cb766)
    #3 __tsan_trace_switch_thunk /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl_amd64.S:53 (marl-unittests+0x4dfeb9)
    #4 TraceAddEvent /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.h:862 (marl-unittests+0x4d51b0)
    #5 MemoryAccess /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:874 (marl-unittests+0x4d51b0)
    #6 MemoryRead /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.h:742 (marl-unittests+0x4d51b0)
    #7 __tsan_vptr_read /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interface_inl.h:98 (marl-unittests+0x4d51b0)
    #8 marl::TrackedAllocator::allocate(marl::Allocation::Request const&) /home/ben/src/marl/build/../include/marl/memory.h:266 (marl-unittests+0x50c045)
    #9 marl::Scheduler::Fiber::create(marl::Allocator*, unsigned int, unsigned long, std::function<void ()> const&) /home/ben/src/marl/build/../include/marl/memory.h:169 (marl-unittests+0x5b5216)
    #10 marl::Scheduler::Fiber::create(marl::Allocator*, unsigned int, unsigned long, std::function<void ()> const&) /home/ben/src/marl/build/../src/scheduler.cpp:275 (marl-unittests+0x5b5216)
    #11 marl::Scheduler::Worker::createWorkerFiber() /home/ben/src/marl/build/../src/scheduler.cpp:717 (marl-unittests+0x5b6605)
    #12 marl::Scheduler::Worker::suspend(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*) /home/ben/src/marl/build/../src/scheduler.cpp:495 (marl-unittests+0x5b5db5)
    #13 marl::Scheduler::Worker::wait(marl::lock&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const*, std::function<bool ()> const&) /home/ben/src/marl/build/../src/scheduler.cpp:446 (marl-unittests+0x5b4f82)
    #14 bool marl::ConditionVariable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, marl::Event::Shared::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)::{lambda()#1}>(marl::lock&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, marl::Event::Shared::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)::{lambda()#1}&&) /home/ben/src/marl/build/../include/marl/scheduler.h:486 (marl-unittests+0x5366d7)
    #15 bool marl::ConditionVariable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, marl::Event::Shared::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)::{lambda()#1}>(marl::lock&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, marl::Event::Shared::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&)::{lambda()#1}&&) /home/ben/src/marl/build/../include/marl/conditionvariable.h:171 (marl-unittests+0x5366d7)
    #16 std::_Function_handler<void (), WithBoundScheduler_EventWaitUntilTimeTaken_Test::TestBody()::$_8>::_M_invoke(std::_Any_data const&) /home/ben/src/marl/build/../include/marl/event.h:170 (marl-unittests+0x53045d)
    #17 <null> /home/ben/src/marl/build/../include/marl/event.h:205 (marl-unittests+0x53045d)
    #18 WithBoundScheduler_EventWaitUntilTimeTaken_Test::TestBody()::$_8::operator()() const /home/ben/src/marl/build/../src/event_test.cpp:186 (marl-unittests+0x53045d)
    #19 std::_Function_handler<void (), WithBoundScheduler_EventWaitUntilTimeTaken_Test::TestBody()::$_8>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:300 (marl-unittests+0x53045d)
    #20 marl::Scheduler::Worker::runUntilIdle() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:688 (marl-unittests+0x5b72e7)
    #21 <null> /home/ben/src/marl/build/../include/marl/task.h:94 (marl-unittests+0x5b72e7)
    #22 marl::Scheduler::Worker::runUntilIdle() /home/ben/src/marl/build/../src/scheduler.cpp:703 (marl-unittests+0x5b72e7)
    #23 marl::Scheduler::Worker::run() /home/ben/src/marl/build/../src/scheduler.cpp:587 (marl-unittests+0x5b69e0)
    #24 marl::Scheduler::Worker::run() /home/ben/src/marl/build/../src/scheduler.cpp:580 (marl-unittests+0x5b69e0)
    #25 marl::Scheduler::Worker::createWorkerFiber()::$_5::operator()() const /home/ben/src/marl/build/../src/scheduler.cpp:718 (marl-unittests+0x5b7d3d)
    #26 std::_Function_handler<void (), marl::Scheduler::Worker::createWorkerFiber()::$_5>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:300 (marl-unittests+0x5b7d3d)
    #27 marl::OSFiber::run(marl::OSFiber*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:688 (marl-unittests+0x53d223)
    #28 marl::OSFiber::run(marl::OSFiber*) /home/ben/src/marl/build/../src/osfiber_asm.h:129 (marl-unittests+0x53d223)
    #29 marl_fiber_trampoline /home/ben/src/marl/build/../src/osfiber_x64.c:20 (marl-unittests+0x5bc129)

ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV /home/brian/src/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.h:638 in ExtractTagFromStack<__sanitizer::BufferedStackTrace>
==8770==ABORTING

This seems to occur reliably when repeatedly running the same test without killing the process:

./marl-unittests --gtest_repeat=-1 --gtest_filter=SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0

The time taken with each test iteration climbs with each run:

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (234 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (234 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (234 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 2) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (332 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (332 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (332 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 3) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (427 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (427 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (427 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 4) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (530 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (530 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (531 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 5) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (619 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (619 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (619 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 6) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (723 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (724 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (724 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 7) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (813 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (813 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (815 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 8) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (922 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (922 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (922 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 9) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (1101 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (1101 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1101 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 10) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (1100 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (1100 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1100 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 11) . . .

Before exploding at the same test iteration.

It would seem that we're (or TSAN) is leaking memory until TSAN can no longer allocate its own records. Interestingly, ASAN reports no memory leaks.

Further investigation required...

[ben-clayton]

Looks like an issue with TSAN: https://groups.google.com/g/thread-sanitizer/c/k3gEHSXcvMY/m/7TLI7iONAwAJ

[ben-clayton]

In the thread-sanitizer group conversation Dmitry Vyukov recommended using the TSAN annotations.

I've tried this in ben-clayton@fdc62f3.

With this:

• Performance remains stable between test iterations
• We only got to around 6 iterations before, we now get to around 18...
• ... before running out of memory and crashing and burning again.

./marl-unittests --gtest_filter=SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 --gtest_repeat=-1

Repeating all tests (iteration 1) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (447 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (447 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (447 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 2) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (351 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (351 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (351 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 3) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (352 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (352 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (352 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 4) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (354 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (354 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (355 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 5) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (352 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (352 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (352 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 6) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (353 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (354 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (355 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 7) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (353 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (353 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (353 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 8) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (357 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (357 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (357 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 9) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (356 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (357 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (357 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 10) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (356 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (356 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (356 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 11) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (356 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (356 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (356 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 12) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (356 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (356 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (356 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 13) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (359 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (359 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (359 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 14) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (362 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (362 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (362 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 15) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (356 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (356 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (356 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 16) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (358 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (358 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (358 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 17) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[       OK ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0 (356 ms)
[----------] 1 test from SchedulerParams/WithBoundScheduler (356 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (356 ms total)
[  PASSED  ] 1 test.

Repeating all tests (iteration 18) . . .

Note: Google Test filter = SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SchedulerParams/WithBoundScheduler
[ RUN      ] SchedulerParams/WithBoundScheduler.EventWaitUntilTimeTaken/0
ASSERT: Failed to protect page at 0x7f2e481fa000: Cannot allocate memory
Aborted (core dumped)

[ben-clayton]

Conversation continued in g/thread-sanitizer
Whatever the cause of this bug, it certainly seems to be out of the control of marl.
Closing.