ThreadPool asserting while running tests in parallel

[pepyakin]

After I've upgraded binaryen from 9f4e32 to the latest master in https://github.com/pepyakin/binaryen-rs I started to get assertion errors while running tests, like:

binaryen-bdb9684fb6e3b36a: /home/travis/build/pepyakin/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp:50: wasm::Thread::Thread(): Assertion `!ThreadPool::get()→isRunning()' failed.

When I run these tests in single-thread mode the error don't show up.

I'm using binaryen-c API.

[kripken]

Hmm, I wonder if there is some difference in startup between C++ and Rust that could cause that? Not sure what it could be, though.

1. What test is it running where that fails?
2. Is it possible to get a full stack trace for that assert?
3. Building with BINARYEN_THREAD_DEBUG (just src/support/threads.cpp needs it) will produce runtime logging that could also help here.
4. If it's feasible to bisect to find the bad commit, that could help too (looks like the range is from last september til today).

[pepyakin]

It seems there are actually different failures that show up non-deterministically!

Assertion failed: (doWorkers.size() == num), function work, file /Users/pepyakin/dev/my/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp, line 161.

Assertion failed: (old == threads.size()), function resetThreadsAreReady, file /Users/pepyakin/dev/my/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp, line 195.

Assertion failed: (!running), function work, file /Users/pepyakin/dev/my/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp, line 162.

Assertion failed: (ec == 0), function unlock, file /BuildRoot/Library/Caches/com.apple.xbs/Sources/libcxx/libcxx-307.5/src/mutex.cpp, line 48.

And it also can just succeed or never terminate!

[pepyakin]

So, when it doesn't terminate, there are couple of threads sitting in wasm::Thread::mainLoop:98

[pepyakin]

https://github.com/pepyakin/binaryen-rs/blob/d307d5ae20918effb6f273f80671a0376a0165f4/src/lib.rs#L1236

Also, the same failure was spotted while executing this test
https://github.com/pepyakin/binaryen-rs/blob/d307d5ae20918effb6f273f80671a0376a0165f4/src/lib.rs#L1252

Assertion failed: (ec == 0), function unlock, file /BuildRoot/Library/Caches/com.apple.xbs/Sources/libcxx/libcxx-307.5/src/mutex.cpp, line 48.
Process 98665 stopped
* thread #2, name = 'tests::test_hello_world', stop reason = signal SIGABRT
    frame #0: 0x00007fff90249d42 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff90249d42 <+10>: jae    0x7fff90249d4c            ; <+20>
    0x7fff90249d44 <+12>: movq   %rax, %rdi
    0x7fff90249d47 <+15>: jmp    0x7fff90242caf            ; cerror_nocancel
    0x7fff90249d4c <+20>: retq
Target 0: (binaryen-34bc79fc7b73fca8) stopped.
(lldb) bt
* thread #2, name = 'tests::test_hello_world', stop reason = signal SIGABRT
  * frame #0: 0x00007fff90249d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff90337457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff901af420 libsystem_c.dylib`abort + 129
    frame #3: 0x00007fff90176893 libsystem_c.dylib`__assert_rtn + 320
    frame #4: 0x00007fff8ece7142 libc++.1.dylib`std::__1::mutex::unlock() + 54
    frame #5: 0x0000000100389292 binaryen-34bc79fc7b73fca8`wasm::ThreadPool::initialize(unsigned long) [inlined] std::__1::unique_lock<std::__1::mutex>::~unique_lock(this=0x000070000679bd20) at __mutex_base:157
    frame #6: 0x0000000100389272 binaryen-34bc79fc7b73fca8`wasm::ThreadPool::initialize(unsigned long) [inlined] std::__1::unique_lock<std::__1::mutex>::~unique_lock(this=0x000070000679bd20) at __mutex_base:155
    frame #7: 0x0000000100389272 binaryen-34bc79fc7b73fca8`wasm::ThreadPool::initialize(this=0x0000000100c021e0, num=8) at threads.cpp:126
    frame #8: 0x0000000100387a14 binaryen-34bc79fc7b73fca8`wasm::ThreadPool::get() at threads.cpp:143
    frame #9: 0x00000001000e8477 binaryen-34bc79fc7b73fca8`wasm::PassRunner::run(this=0x000070000679c970)::$_0::operator()() const at pass.cpp:276
    frame #10: 0x00000001000e6dd3 binaryen-34bc79fc7b73fca8`wasm::PassRunner::run(this=0x000070000679d550) at pass.cpp:311
    frame #11: 0x000000010040a969 binaryen-34bc79fc7b73fca8`wasm::WasmValidator::validate(this=0x000070000679d758, module=0x0000000100c02730, features=1, flags=2) at wasm-validator.cpp:1029
    frame #12: 0x00000001000469e4 binaryen-34bc79fc7b73fca8`::BinaryenModuleValidate(module=0x0000000100c02730) at binaryen-c.cpp:2004
    frame #13: 0x0000000100006ee9 binaryen-34bc79fc7b73fca8`binaryen::Module::is_valid::h6f7fe82e8fe3f983 + 57
    frame #14: 0x000000010000b512 binaryen-34bc79fc7b73fca8`binaryen::tests::test_hello_world::hcd6bc71bba10d3ff + 450
    frame #15: 0x000000010002c542 binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> [inlined] test::run_test::{{closure}} at lib.rs:1510 [opt]
    frame #16: 0x000000010002c53d binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> [inlined] core::ops::function::FnOnce::call_once<closure,(())> at function.rs:223 [opt]
    frame #17: 0x000000010002c53d binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> at lib.rs:142 [opt]
    frame #18: 0x0000000100479aef binaryen-34bc79fc7b73fca8`panic_unwind::__rust_maybe_catch_panic at lib.rs:101 [opt]
    frame #19: 0x000000010001ca41 binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> at panicking.rs:459 [opt]
    frame #20: 0x000000010001c9fc binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:365 [opt]
    frame #21: 0x000000010001c9fc binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] test::run_test::run_test_inner::{{closure}} at lib.rs:1447 [opt]
    frame #22: 0x000000010001c8a9 binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> at backtrace.rs:133 [opt]
    frame #23: 0x00000001000222d8 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> [inlined] std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> at mod.rs:406 [opt]
    frame #24: 0x00000001000222c2 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> [inlined] std::panic::{{impl}}::call_once<(),closure> at panic.rs:300 [opt]
    frame #25: 0x00000001000222c2 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> at panicking.rs:480 [opt]
    frame #26: 0x0000000100479aef binaryen-34bc79fc7b73fca8`panic_unwind::__rust_maybe_catch_panic at lib.rs:101 [opt]
    frame #27: 0x0000000100038a02 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> at panicking.rs:459 [opt]
    frame #28: 0x00000001000389c9 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:365 [opt]
    frame #29: 0x00000001000389c9 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::thread::{{impl}}::spawn::{{closure}}<closure,()> at mod.rs:405 [opt]
    frame #30: 0x000000010003897c binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> at boxed.rs:817 [opt]
    frame #31: 0x0000000100461908 binaryen-34bc79fc7b73fca8`std::sys_common::thread::start_thread [inlined] alloc::boxed::{{impl}}::call_once<(),()> at boxed.rs:827 [opt]
    frame #32: 0x0000000100461905 binaryen-34bc79fc7b73fca8`std::sys_common::thread::start_thread at thread.rs:24 [opt]
    frame #33: 0x000000010046b9c9 binaryen-34bc79fc7b73fca8`std::sys::unix::thread::{{impl}}::new::thread_start at thread.rs:90 [opt]
    frame #34: 0x00007fff9033493b libsystem_pthread.dylib`_pthread_body + 180
    frame #35: 0x00007fff90334887 libsystem_pthread.dylib`_pthread_start + 286
    frame #36: 0x00007fff9033408d libsystem_pthread.dylib`thread_start + 13

[pepyakin]

https://github.com/pepyakin/binaryen-rs/blob/d307d5ae20918effb6f273f80671a0376a0165f4/src/lib.rs#L1236

Assertion failed: (!running), function work, file /Users/pepyakin/dev/my/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp, line 162.
Process 98697 stopped
* thread #2, name = 'tests::test_hello_world', stop reason = signal SIGABRT
    frame #0: 0x00007fff90249d42 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff90249d42 <+10>: jae    0x7fff90249d4c            ; <+20>
    0x7fff90249d44 <+12>: movq   %rax, %rdi
    0x7fff90249d47 <+15>: jmp    0x7fff90242caf            ; cerror_nocancel
    0x7fff90249d4c <+20>: retq
Target 0: (binaryen-34bc79fc7b73fca8) stopped.
(lldb) bt
* thread #2, name = 'tests::test_hello_world', stop reason = signal SIGABRT
  * frame #0: 0x00007fff90249d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff90337457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff901af420 libsystem_c.dylib`abort + 129
    frame #3: 0x00007fff90176893 libsystem_c.dylib`__assert_rtn + 320
    frame #4: 0x00000001003899be binaryen-34bc79fc7b73fca8`wasm::ThreadPool::work(this=0x00000001025003c0, doWorkers=size=1)>, std::__1::allocator<std::__1::function<wasm::ThreadWorkState ()> > >&) at threads.cpp:162
    frame #5: 0x00000001000e8931 binaryen-34bc79fc7b73fca8`wasm::PassRunner::run(this=0x000070000ef42970)::$_0::operator()() const at pass.cpp:299
    frame #6: 0x00000001000e6dd3 binaryen-34bc79fc7b73fca8`wasm::PassRunner::run(this=0x000070000ef43550) at pass.cpp:311
    frame #7: 0x000000010040a969 binaryen-34bc79fc7b73fca8`wasm::WasmValidator::validate(this=0x000070000ef43758, module=0x0000000100c09c50, features=1, flags=2) at wasm-validator.cpp:1029
    frame #8: 0x00000001000469e4 binaryen-34bc79fc7b73fca8`::BinaryenModuleValidate(module=0x0000000100c09c50) at binaryen-c.cpp:2004
    frame #9: 0x0000000100006ee9 binaryen-34bc79fc7b73fca8`binaryen::Module::is_valid::h6f7fe82e8fe3f983 + 57
    frame #10: 0x000000010000b512 binaryen-34bc79fc7b73fca8`binaryen::tests::test_hello_world::hcd6bc71bba10d3ff + 450
    frame #11: 0x000000010002c542 binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> [inlined] test::run_test::{{closure}} at lib.rs:1510 [opt]
    frame #12: 0x000000010002c53d binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> [inlined] core::ops::function::FnOnce::call_once<closure,(())> at function.rs:223 [opt]
    frame #13: 0x000000010002c53d binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> at lib.rs:142 [opt]
    frame #14: 0x0000000100479aef binaryen-34bc79fc7b73fca8`panic_unwind::__rust_maybe_catch_panic at lib.rs:101 [opt]
    frame #15: 0x000000010001ca41 binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> at panicking.rs:459 [opt]
    frame #16: 0x000000010001c9fc binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:365 [opt]
    frame #17: 0x000000010001c9fc binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] test::run_test::run_test_inner::{{closure}} at lib.rs:1447 [opt]
    frame #18: 0x000000010001c8a9 binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> at backtrace.rs:133 [opt]
    frame #19: 0x00000001000222d8 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> [inlined] std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> at mod.rs:406 [opt]
    frame #20: 0x00000001000222c2 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> [inlined] std::panic::{{impl}}::call_once<(),closure> at panic.rs:300 [opt]
    frame #21: 0x00000001000222c2 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> at panicking.rs:480 [opt]
    frame #22: 0x0000000100479aef binaryen-34bc79fc7b73fca8`panic_unwind::__rust_maybe_catch_panic at lib.rs:101 [opt]
    frame #23: 0x0000000100038a02 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> at panicking.rs:459 [opt]
    frame #24: 0x00000001000389c9 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:365 [opt]
    frame #25: 0x00000001000389c9 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::thread::{{impl}}::spawn::{{closure}}<closure,()> at mod.rs:405 [opt]
    frame #26: 0x000000010003897c binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> at boxed.rs:817 [opt]
    frame #27: 0x0000000100461908 binaryen-34bc79fc7b73fca8`std::sys_common::thread::start_thread [inlined] alloc::boxed::{{impl}}::call_once<(),()> at boxed.rs:827 [opt]
    frame #28: 0x0000000100461905 binaryen-34bc79fc7b73fca8`std::sys_common::thread::start_thread at thread.rs:24 [opt]
    frame #29: 0x000000010046b9c9 binaryen-34bc79fc7b73fca8`std::sys::unix::thread::{{impl}}::new::thread_start at thread.rs:90 [opt]
    frame #30: 0x00007fff9033493b libsystem_pthread.dylib`_pthread_body + 180
    frame #31: 0x00007fff90334887 libsystem_pthread.dylib`_pthread_start + 286
    frame #32: 0x00007fff9033408d libsystem_pthread.dylib`thread_start + 13

[pepyakin]

https://github.com/pepyakin/binaryen-rs/blob/d307d5ae20918effb6f273f80671a0376a0165f4/src/lib.rs#L1252

Assertion failed: (!ThreadPool::get()->isRunning()), function Thread, file /Users/pepyakin/dev/my/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp, line 50.
test tests::test_relooper_with_different_module ... ok
Process 98716 stopped
* thread #2, name = 'tests::test_simple', stop reason = signal SIGABRT
    frame #0: 0x00007fff90249d42 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff90249d42 <+10>: jae    0x7fff90249d4c            ; <+20>
    0x7fff90249d44 <+12>: movq   %rax, %rdi
    0x7fff90249d47 <+15>: jmp    0x7fff90242caf            ; cerror_nocancel
    0x7fff90249d4c <+20>: retq
Target 0: (binaryen-34bc79fc7b73fca8) stopped.
(lldb) bt
* thread #2, name = 'tests::test_simple', stop reason = signal SIGABRT
  * frame #0: 0x00007fff90249d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff90337457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff901af420 libsystem_c.dylib`abort + 129
    frame #3: 0x00007fff90176893 libsystem_c.dylib`__assert_rtn + 320
    frame #4: 0x00000001003872f3 binaryen-34bc79fc7b73fca8`wasm::Thread::Thread(this=0x0000000100c025e0) at threads.cpp:50
    frame #5: 0x0000000100387ef5 binaryen-34bc79fc7b73fca8`wasm::Thread::Thread(this=0x0000000100c025e0) at threads.cpp:49
    frame #6: 0x0000000100389444 binaryen-34bc79fc7b73fca8`std::__1::unique_ptr<wasm::Thread, std::__1::default_delete<wasm::Thread> > wasm::make_unique<wasm::Thread>() at utilities.h:60
    frame #7: 0x0000000100388882 binaryen-34bc79fc7b73fca8`wasm::ThreadPool::initialize(this=0x0000000100c02390, num=8) at threads.cpp:115
    frame #8: 0x0000000100387a14 binaryen-34bc79fc7b73fca8`wasm::ThreadPool::get() at threads.cpp:143
    frame #9: 0x00000001000e8477 binaryen-34bc79fc7b73fca8`wasm::PassRunner::run(this=0x000070000a1c78f0)::$_0::operator()() const at pass.cpp:276
    frame #10: 0x00000001000e6dd3 binaryen-34bc79fc7b73fca8`wasm::PassRunner::run(this=0x000070000a1c84d0) at pass.cpp:311
    frame #11: 0x000000010040a969 binaryen-34bc79fc7b73fca8`wasm::WasmValidator::validate(this=0x000070000a1c86d8, module=0x0000000102500180, features=1, flags=2) at wasm-validator.cpp:1029
    frame #12: 0x00000001000469e4 binaryen-34bc79fc7b73fca8`::BinaryenModuleValidate(module=0x0000000102500180) at binaryen-c.cpp:2004
    frame #13: 0x0000000100006ee9 binaryen-34bc79fc7b73fca8`binaryen::Module::is_valid::h6f7fe82e8fe3f983 + 57
    frame #14: 0x000000010000b809 binaryen-34bc79fc7b73fca8`binaryen::tests::test_simple::h103cd8ffdd599a23 + 617
    frame #15: 0x000000010002c542 binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> [inlined] test::run_test::{{closure}} at lib.rs:1510 [opt]
    frame #16: 0x000000010002c53d binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> [inlined] core::ops::function::FnOnce::call_once<closure,(())> at function.rs:223 [opt]
    frame #17: 0x000000010002c53d binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> at lib.rs:142 [opt]
    frame #18: 0x0000000100479aef binaryen-34bc79fc7b73fca8`panic_unwind::__rust_maybe_catch_panic at lib.rs:101 [opt]
    frame #19: 0x000000010001ca41 binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> at panicking.rs:459 [opt]
    frame #20: 0x000000010001c9fc binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:365 [opt]
    frame #21: 0x000000010001c9fc binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] test::run_test::run_test_inner::{{closure}} at lib.rs:1447 [opt]
    frame #22: 0x000000010001c8a9 binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> at backtrace.rs:133 [opt]
    frame #23: 0x00000001000222d8 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> [inlined] std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> at mod.rs:406 [opt]
    frame #24: 0x00000001000222c2 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> [inlined] std::panic::{{impl}}::call_once<(),closure> at panic.rs:300 [opt]
    frame #25: 0x00000001000222c2 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> at panicking.rs:480 [opt]
    frame #26: 0x0000000100479aef binaryen-34bc79fc7b73fca8`panic_unwind::__rust_maybe_catch_panic at lib.rs:101 [opt]
    frame #27: 0x0000000100038a02 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> at panicking.rs:459 [opt]
    frame #28: 0x00000001000389c9 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:365 [opt]
    frame #29: 0x00000001000389c9 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::thread::{{impl}}::spawn::{{closure}}<closure,()> at mod.rs:405 [opt]
    frame #30: 0x000000010003897c binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> at boxed.rs:817 [opt]
    frame #31: 0x0000000100461908 binaryen-34bc79fc7b73fca8`std::sys_common::thread::start_thread [inlined] alloc::boxed::{{impl}}::call_once<(),()> at boxed.rs:827 [opt]
    frame #32: 0x0000000100461905 binaryen-34bc79fc7b73fca8`std::sys_common::thread::start_thread at thread.rs:24 [opt]
    frame #33: 0x000000010046b9c9 binaryen-34bc79fc7b73fca8`std::sys::unix::thread::{{impl}}::new::thread_start at thread.rs:90 [opt]
    frame #34: 0x00007fff9033493b libsystem_pthread.dylib`_pthread_body + 180
    frame #35: 0x00007fff90334887 libsystem_pthread.dylib`_pthread_start + 286
    frame #36: 0x00007fff9033408d libsystem_pthread.dylib`thread_start + 13
(lldb)

Third one from BinaryenModuleValidate

[pepyakin]

https://github.com/pepyakin/binaryen-rs/blob/d307d5ae20918effb6f273f80671a0376a0165f4/src/lib.rs#L1236

Another one from BinaryenModuleValidate

Assertion failed: (!ThreadPool::test to_cstr::tests::test_use_cases ... get()->isRunning()), function Thread, file /Users/pepyakin/dev/my/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp, line 50.
ok
Process 98751 stopped
* thread #2, name = 'tests::test_hello_world', stop reason = signal SIGABRT
    frame #0: 0x00007fff90249d42 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff90249d42 <+10>: jae    0x7fff90249d4c            ; <+20>
    0x7fff90249d44 <+12>: movq   %rax, %rdi
    0x7fff90249d47 <+15>: jmp    0x7fff90242caf            ; cerror_nocancel
    0x7fff90249d4c <+20>: retq
Target 0: (binaryen-34bc79fc7b73fca8) stopped.
(lldb) bt
* thread #2, name = 'tests::test_hello_world', stop reason = signal SIGABRT
  * frame #0: 0x00007fff90249d42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff90337457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff901af420 libsystem_c.dylib`abort + 129
    frame #3: 0x00007fff90176893 libsystem_c.dylib`__assert_rtn + 320
    frame #4: 0x00000001003872f3 binaryen-34bc79fc7b73fca8`wasm::Thread::Thread(this=0x0000000102b00c00) at threads.cpp:50
    frame #5: 0x0000000100387ef5 binaryen-34bc79fc7b73fca8`wasm::Thread::Thread(this=0x0000000102b00c00) at threads.cpp:49
    frame #6: 0x0000000100389444 binaryen-34bc79fc7b73fca8`std::__1::unique_ptr<wasm::Thread, std::__1::default_delete<wasm::Thread> > wasm::make_unique<wasm::Thread>() at utilities.h:60
    frame #7: 0x0000000100388882 binaryen-34bc79fc7b73fca8`wasm::ThreadPool::initialize(this=0x0000000100e003a0, num=8) at threads.cpp:115
    frame #8: 0x0000000100387a14 binaryen-34bc79fc7b73fca8`wasm::ThreadPool::get() at threads.cpp:143
    frame #9: 0x00000001000e8477 binaryen-34bc79fc7b73fca8`wasm::PassRunner::run(this=0x000070000b31e970)::$_0::operator()() const at pass.cpp:276
    frame #10: 0x00000001000e6dd3 binaryen-34bc79fc7b73fca8`wasm::PassRunner::run(this=0x000070000b31f550) at pass.cpp:311
    frame #11: 0x000000010040a969 binaryen-34bc79fc7b73fca8`wasm::WasmValidator::validate(this=0x000070000b31f758, module=0x0000000100d004a0, features=1, flags=2) at wasm-validator.cpp:1029
    frame #12: 0x00000001000469e4 binaryen-34bc79fc7b73fca8`::BinaryenModuleValidate(module=0x0000000100d004a0) at binaryen-c.cpp:2004
    frame #13: 0x0000000100006ee9 binaryen-34bc79fc7b73fca8`binaryen::Module::is_valid::h6f7fe82e8fe3f983 + 57
    frame #14: 0x000000010000b512 binaryen-34bc79fc7b73fca8`binaryen::tests::test_hello_world::hcd6bc71bba10d3ff + 450
    frame #15: 0x000000010002c542 binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> [inlined] test::run_test::{{closure}} at lib.rs:1510 [opt]
    frame #16: 0x000000010002c53d binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> [inlined] core::ops::function::FnOnce::call_once<closure,(())> at function.rs:223 [opt]
    frame #17: 0x000000010002c53d binaryen-34bc79fc7b73fca8`test::{{impl}}::call_box<(),closure> at lib.rs:142 [opt]
    frame #18: 0x0000000100479aef binaryen-34bc79fc7b73fca8`panic_unwind::__rust_maybe_catch_panic at lib.rs:101 [opt]
    frame #19: 0x000000010001ca41 binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> at panicking.rs:459 [opt]
    frame #20: 0x000000010001c9fc binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:365 [opt]
    frame #21: 0x000000010001c9fc binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> [inlined] test::run_test::run_test_inner::{{closure}} at lib.rs:1447 [opt]
    frame #22: 0x000000010001c8a9 binaryen-34bc79fc7b73fca8`std::sys_common::backtrace::__rust_begin_short_backtrace<closure,()> at backtrace.rs:133 [opt]
    frame #23: 0x00000001000222d8 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> [inlined] std::thread::{{impl}}::spawn::{{closure}}::{{closure}}<closure,()> at mod.rs:406 [opt]
    frame #24: 0x00000001000222c2 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> [inlined] std::panic::{{impl}}::call_once<(),closure> at panic.rs:300 [opt]
    frame #25: 0x00000001000222c2 binaryen-34bc79fc7b73fca8`std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> at panicking.rs:480 [opt]
    frame #26: 0x0000000100479aef binaryen-34bc79fc7b73fca8`panic_unwind::__rust_maybe_catch_panic at lib.rs:101 [opt]
    frame #27: 0x0000000100038a02 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> at panicking.rs:459 [opt]
    frame #28: 0x00000001000389c9 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> at panic.rs:365 [opt]
    frame #29: 0x00000001000389c9 binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> [inlined] std::thread::{{impl}}::spawn::{{closure}}<closure,()> at mod.rs:405 [opt]
    frame #30: 0x000000010003897c binaryen-34bc79fc7b73fca8`alloc::boxed::{{impl}}::call_box<(),closure> at boxed.rs:817 [opt]
    frame #31: 0x0000000100461908 binaryen-34bc79fc7b73fca8`std::sys_common::thread::start_thread [inlined] alloc::boxed::{{impl}}::call_once<(),()> at boxed.rs:827 [opt]
    frame #32: 0x0000000100461905 binaryen-34bc79fc7b73fca8`std::sys_common::thread::start_thread at thread.rs:24 [opt]
    frame #33: 0x000000010046b9c9 binaryen-34bc79fc7b73fca8`std::sys::unix::thread::{{impl}}::new::thread_start at thread.rs:90 [opt]
    frame #34: 0x00007fff9033493b libsystem_pthread.dylib`_pthread_body + 180
    frame #35: 0x00007fff90334887 libsystem_pthread.dylib`_pthread_start + 286
    frame #36: 0x00007fff9033408d libsystem_pthread.dylib`thread_start + 13

[kripken]

I have a poor understanding of Rust stack traces, but it looks like

• Rust starts to panic
• During panicking it starts to run a binaryen test
• The binaryen test tries to do stuff with threads but hits an assert

Is it possible Rust panicking leaves things in a state where mallocs or other system operations will fail?

[pepyakin]

I don't know about C++ memory model and concurrency semantics but doesn't this should be wrapped in somekind of mutex?

binaryen/src/support/threads.cpp

Lines 140 to 146 in 63ebdb6

	ThreadPool* ThreadPool::get() {
	if (!pool) {
	pool = make_unique<ThreadPool>();
	pool->initialize(getNumCores());
	}
	return pool.get();
	}

Because, at the time we get into initialze some other thread might get there and initialize pool twice?

Rust starts to panic

No there are no panics. What you see is machinery to catch a panic and a test setup.

[pepyakin]

Example of BINARYEN_THREAD_DEBUG

[POOL] initialize()
[POOL] initialize()
[POOL] work() sequentially
[POOL] reset threads are ready
test to_cstr::tests::test_use_cases ... ok
test tests::test_relooper_with_different_module ... ok
(block
[THREAD  (block 0x70000c3a1000$] block$2$breakchecking for work

  [THREAD 0x70000bd98000(] nopchecking for work
)
  ([THREAD block0x70000be1b000]
checking for work
   (br $block$2$break)
  )
 )
 (block
  (nop[THREAD )0x70000be9e000
]  checking for work
)
)
[THREAD 0x70000c424000] checking for work
test relooper::tests::test ... [POOL] notify thread is ready
ok
[THREAD 0x70000c4a7000] checking for work
[THREAD 0x70000c52a000] checking for work
[POOL] initialize() waiting
[THREAD 0x70000c5ad000] checking for work
[POOL] notify thread is ready
[POOL] notify thread is ready
[POOL] work() on threads
[POOL] notify thread is ready
[POOL] notify thread is ready
[POOL] notify thread is ready
[POOL] notify thread is ready
[POOL] are threads ready?
[POOL] notify thread is ready
[POOL] reset threads are ready
Assertion failed: (!ThreadPool::get()->isRunning()), function Thread, file /Users/pepyakin/dev/my/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp, line 52.
error: An unknown error occurred

running 8 tests
test to_cstr::tests::test_to_cstr_stash_option ... ok
test tests::test_use_same_expr_twice ... ok
test to_cstr::tests::test_use_cases ... [POOL] ok
initialize()
[POOL] work() sequentially
[POOL] work() sequentially
[POOL] reset threads are ready
test tests::test_relooper_with_different_module ... [THREAD ok
0x70001060e000] checking for work
([THREAD test tests::test_unreachable ... ok
block
0x700010005000] checking for work
 (block [THREAD $block$2$break
0x700010088000  (nop)
 ]  checking for work
(block
[POOL] notify thread is ready
   (br $block$2$break)
  )
 )
[THREAD  0x70001010b000(block
]  checking for work
 (nop)
[THREAD  )
0x700010691000)]
checking for work
[THREAD 0x700010714000] test relooper::tests::test ... checking for work
ok
[THREAD 0x700010797000] checking for work
[POOL] initialize() waiting
[POOL] notify thread is ready
[THREAD 0x70001040b000] checking for work
[POOL] notify thread is ready
[POOL] notify thread is ready
[POOL] notify thread is ready
[POOL] notify thread is ready
[POOL] work() on threads
[POOL] notify thread is ready
[POOL] are threads ready?
[POOL] notify thread is ready
[THREAD 0x70001060e000] thread waiting
[THREAD 0x700010005000] thread waiting
[THREAD 0x700010088000] thread waiting
[THREAD 0x700010691000] thread waiting
[THREAD 0x70001010b000] thread waiting
[THREAD 0x700010714000] thread waiting
[POOL] reset threads are ready
Assertion failed: (old == threads.size()), function resetThreadsAreReady, file /Users/pepyakin/dev/my/binaryen-rs/binaryen-sys/binaryen/src/support/threads.cpp, line 197.

[kripken]

No there are no panics. What you see is machinery to catch a panic and a test setup.

I see, thanks. As I said, I know very little about Rust :)

I don't know about C++ memory model and concurrency semantics but doesn't this should be wrapped in somekind of mutex?

Hmm, good point. The intention was that that should be safe, as the threadpool is only requested from the main thread. However, I see we added asserts in the thread constructors, which get the pool to check if it's running, which run in their own threads ;) lol

Another issue is that I think you are running tests in multiple threads? We didn't test that, so we need some more carefulness to handle that properly.

Fixes for both should be in #1377

[pepyakin]

I see, thanks. As I said, I know very little about Rust :)

Yeah, feel free to ask in future!

Another issue is that I think you are running tests in multiple threads? We didn't test that, so we need some more carefulness to handle that properly.

Exactly! I assumed C API is supposed to run fine in multithread environment, is this assumption correct?

[kripken]

It should, that's very important even, but it was never tested ;) Our test harness is just not multithreaded. We should probably add a specific test for this...