Deadlock when dropping Runtime #2046

appaquet · 2020-01-04T16:24:55Z

Version

0.2.6

Platform

Linux

Description

~~See this gist for minimum code to reproduce and backtrace. It seems to only happen if the spawn_blocking block uses a cloned mpsc sender.~~

Inconsistently, a deadlock occurs when dropping a Runtime in a configuration of channels and spawn_blocking as seen here. It looks like it tries to go through the tokio::runtime::thread_pool::queue::global::Queue<T> pointers mutex twice in the same stack when going through the closed channel code path. This fix seems to be fixing the issue.

Stack trace

Thread 328 (Thread 0x7f2a0b9fc700 (LWP 13634)):
#0  __lll_lock_wait (futex=futex@entry=0x7f2a0c0020b0, private=0) at lowlevellock.c:52
#1  0x00007f2a1088f163 in __GI___pthread_mutex_lock (mutex=0x7f2a0c0020b0) at ../nptl/pthread_mutex_lock.c:80
#2  0x000055beadd4912a in std::sys::unix::mutex::Mutex::lock (self=0x7f2a0c0020b0 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/sys/unix/mutex.rs:55
#3  0x000055beadd8d921 in std::sys_common::mutex::Mutex::raw_lock (self=0x7f2a0c0020b0 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/sys_common/mutex.rs:36
#4  0x000055beadd9b5e5 in std::sync::mutex::Mutex<T>::lock (self=0x7f2a0c00a360 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/sync/mutex.rs:220
#5  0x000055beadd5a1c0 in tokio::runtime::thread_pool::queue::global::Queue<T>::push (self=0x7f2a0c00a360 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/queue/global.rs:90
#6  0x000055beadda8737 in tokio::runtime::thread_pool::queue::inject::Inject<T>::push (self=0x7f2a0c002bb0 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/queue/inject.rs:19
#7  0x000055beadd95421 in tokio::runtime::thread_pool::slice::Set::inject_task (self=0x7f2a0c002b90 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:69
#8  0x000055beadd9573d in tokio::runtime::thread_pool::slice::Set::schedule::{{closure}} (current_worker=0x7f2a0b9f9d48 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:106
#9  0x000055beadd5996c in tokio::runtime::thread_pool::current::get::{{closure}} (cell=0x7f2a0b9fc390 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/current.rs:61
#10 0x000055beadda32c6 in std::thread::local::LocalKey<T>::try_with (self=0x55beae5ef058 = {...}, f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/thread/local.rs:262
#11 0x000055beadda1fc8 in std::thread::local::LocalKey<T>::with (self=0x55beae5ef058 = {...}, f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/thread/local.rs:239
#12 0x000055beadd598f9 in tokio::runtime::thread_pool::current::get (f=size=2 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/current.rs:59
#13 0x000055beadd956c1 in tokio::runtime::thread_pool::slice::Set::schedule (self=0x7f2a0c002b90 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:99
#14 0x000055beadd3739d in tokio::runtime::thread_pool::shared::Shared::schedule (self=0x7f2a0c001fe0 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/shared.rs:43
#15 0x000055beadd3762d in <tokio::runtime::thread_pool::shared::Shared as tokio::task::Schedule>::schedule (self=0x7f2a0c001fe0 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/shared.rs:90
#16 0x000055beadc327d5 in tokio::task::harness::Harness<T,S>::wake_by_ref (self=0x7f2a0b9f9f80 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:315
#17 0x000055beadc3393d in tokio::task::harness::Harness<T,S>::wake_by_val (self=tokio::task::harness::Harness<std::future::GenFuture<generator-0>, tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:299
#18 0x000055beadc6a6d2 in tokio::task::waker::wake_by_val (ptr=0x7f29ec000d90) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/waker.rs:84
#19 0x000055beadbe3ec2 in core::task::wake::Waker::wake (self=core::task::wake::Waker = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/task/wake.rs:252
#20 0x000055beadc01bbd in futures_channel::oneshot::Inner<T>::drop_tx (self=0x7f2a0c0023c0 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-channel-0.3.1/src/oneshot.rs:220
#21 0x000055beadc02362 in <futures_channel::oneshot::Sender<T> as core::ops::drop::Drop>::drop (self=0x7f2a0c002440 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-channel-0.3.1/src/oneshot.rs:374
#22 0x000055beadbc0021 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#23 0x000055beadbc5600 in core::ptr::real_drop_in_place () at common/src/utils/mod.rs:37
#24 0x000055beadbbfd11 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#25 0x000055beadbc10e1 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#26 0x000055beadbc4ce2 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#27 0x000055beadc6083a in tokio::task::core::Core<T>::transition_to_consumed (self=0x7f2a0c002430 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/core.rs:106
#28 0x000055beadc52a34 in tokio::task::harness::Harness<T,S>::do_cancel::{{closure}}::{{closure}} () at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:363
#29 0x000055beadbbfb73 in core::ops::function::FnOnce::call_once () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ops/function.rs:227
#30 0x000055beadbb5bf1 in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=size=1 = {...}, _args=()) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:317
#31 0x000055beadb8cff7 in std::panicking::try::do_call (data=0x7f2a0b9fa398 "ؤ\237\v*\177") at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:287
#32 0x000055beae2ccaca in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:78
#33 0x000055beadb8a851 in std::panicking::try (f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:265
#34 0x000055beadbb6303 in std::panic::catch_unwind (f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:396
#35 0x000055beadc5257c in tokio::task::harness::Harness<T,S>::do_cancel::{{closure}} () at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:361
#36 0x000055beadbf73b8 in tokio::loom::std::causal_cell::CausalCell<T>::with_mut (self=0x7f2a0c002430 = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/loom/std/causal_cell.rs:41
#37 0x000055beadc4fb26 in tokio::task::harness::Harness<T,S>::do_cancel (self=tokio::task::harness::Harness<std::future::GenFuture<generator-0>, tokio::runtime::thread_pool::shared::Shared> = {...}, res=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:359
#38 0x000055beadc47272 in tokio::task::harness::Harness<T,S>::cancel (self=tokio::task::harness::Harness<std::future::GenFuture<generator-0>, tokio::runtime::thread_pool::shared::Shared> = {...}, from_queue=true) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:342
#39 0x000055beadc75e82 in tokio::task::raw::cancel (ptr=0x7f2a0c002400, from_queue=true) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/raw.rs:196
#40 0x000055beadd98a17 in tokio::task::raw::RawTask::cancel_from_queue (self=tokio::task::raw::RawTask = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/raw.rs:145
#41 0x000055beadd2bb11 in tokio::task::Task<S>::shutdown (self=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/mod.rs:385
#42 0x000055beadd9549b in tokio::runtime::thread_pool::slice::Set::inject_task::{{closure}} (res=...) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:71
#43 0x000055beadd5a2c5 in tokio::runtime::thread_pool::queue::global::Queue<T>::push (self=0x7f2a0c00a360 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/queue/global.rs:95
#44 0x000055beadda8737 in tokio::runtime::thread_pool::queue::inject::Inject<T>::push (self=0x7f2a0c002bb0 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/queue/inject.rs:19
#45 0x000055beadd95421 in tokio::runtime::thread_pool::slice::Set::inject_task (self=0x7f2a0c002b90 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:69
#46 0x000055beadd9573d in tokio::runtime::thread_pool::slice::Set::schedule::{{closure}} (current_worker=0x7f2a0b9fa8b8 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:106
#47 0x000055beadd5996c in tokio::runtime::thread_pool::current::get::{{closure}} (cell=0x7f2a0b9fc390 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/current.rs:61
#48 0x000055beadda32c6 in std::thread::local::LocalKey<T>::try_with (self=0x55beae5ef058 = {...}, f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/thread/local.rs:262
#49 0x000055beadda1fc8 in std::thread::local::LocalKey<T>::with (self=0x55beae5ef058 = {...}, f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/thread/local.rs:239
#50 0x000055beadd598f9 in tokio::runtime::thread_pool::current::get (f=size=2 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/current.rs:59
#51 0x000055beadd956c1 in tokio::runtime::thread_pool::slice::Set::schedule (self=0x7f2a0c002b90 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/slice.rs:99
#52 0x000055beadd3739d in tokio::runtime::thread_pool::shared::Shared::schedule (self=0x7f2a0c001fe0 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/shared.rs:43
#53 0x000055beadd3762d in <tokio::runtime::thread_pool::shared::Shared as tokio::task::Schedule>::schedule (self=0x7f2a0c001fe0 = {...}, task=tokio::task::Task<tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/shared.rs:90
#54 0x000055beadc32f55 in tokio::task::harness::Harness<T,S>::wake_by_ref (self=0x7f2a0b9faaf0 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:315
#55 0x000055beadc3399d in tokio::task::harness::Harness<T,S>::wake_by_val (self=tokio::task::harness::Harness<std::future::GenFuture<generator-0>, tokio::runtime::thread_pool::shared::Shared> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:299
#56 0x000055beadc6a612 in tokio::task::waker::wake_by_val (ptr=0x7f2a0c002400) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/waker.rs:84
#57 0x000055beaddec2b2 in core::task::wake::Waker::wake (self=core::task::wake::Waker = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/task/wake.rs:252
#58 0x000055beaddeb5de in futures_core::task::__internal::atomic_waker::AtomicWaker::wake (self=0x7f29ec000cb0 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.1/src/task/__internal/atomic_waker.rs:355
#59 0x000055beadba3409 in futures_channel::mpsc::SenderInner<T>::close_channel (self=0x7f29ec000d48 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-channel-0.3.1/src/mpsc/mod.rs:567
#60 0x000055beadba4684 in <futures_channel::mpsc::SenderInner<T> as core::ops::drop::Drop>::drop (self=0x7f29ec000d48 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-channel-0.3.1/src/mpsc/mod.rs:795
#61 0x000055beadbc5471 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#62 0x000055beadbc3cb6 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#63 0x000055beadbc1511 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#64 0x000055beadbc6441 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#65 0x000055beadbc49e6 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#66 0x000055beadbc7471 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#67 0x000055beadbc1f31 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#68 0x000055beadbc5eb2 in core::ptr::real_drop_in_place () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ptr/mod.rs:175
#69 0x000055beadc6092a in tokio::task::core::Core<T>::transition_to_consumed (self=0x7f29ec000d40 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/core.rs:106
#70 0x000055beadc52a54 in tokio::task::harness::Harness<T,S>::do_cancel::{{closure}}::{{closure}} () at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:363
#71 0x000055beadbbed53 in core::ops::function::FnOnce::call_once () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ops/function.rs:227
#72 0x000055beadbb5c41 in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=size=1 = {...}, _args=()) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:317
#73 0x000055beadb8d277 in std::panicking::try::do_call (data=0x7f2a0b9fae18 "X\257\237\v*\177") at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:287
#74 0x000055beae2ccaca in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:78
#75 0x000055beadb8c291 in std::panicking::try (f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:265
#76 0x000055beadbb6143 in std::panic::catch_unwind (f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:396
#77 0x000055beadc5267c in tokio::task::harness::Harness<T,S>::do_cancel::{{closure}} () at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:361
#78 0x000055beadbf6068 in tokio::loom::std::causal_cell::CausalCell<T>::with_mut (self=0x7f29ec000d40 = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/loom/std/causal_cell.rs:41
#79 0x000055beadc51f06 in tokio::task::harness::Harness<T,S>::do_cancel (self=tokio::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule> = {...}, res=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:359
#80 0x000055beadc47112 in tokio::task::harness::Harness<T,S>::cancel (self=tokio::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule> = {...}, from_queue=true) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/harness.rs:342
#81 0x000055beadc75d82 in tokio::task::raw::cancel (ptr=0x7f29ec000d10, from_queue=true) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/raw.rs:196
#82 0x000055beadd98a17 in tokio::task::raw::RawTask::cancel_from_queue (self=tokio::task::raw::RawTask = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/raw.rs:145
#83 0x000055beadd2bbf1 in tokio::task::Task<S>::shutdown (self=tokio::task::Task<tokio::runtime::blocking::schedule::NoopSchedule> = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/task/mod.rs:385
#84 0x000055beadd7041e in tokio::runtime::blocking::pool::Inner::run2 (self=0x7f2a0c009300 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/blocking/pool.rs:322
#85 0x000055beadd6fd71 in tokio::runtime::blocking::pool::Inner::run::{{closure}}::{{closure}} () at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/blocking/pool.rs:269
#86 0x000055beadd592d6 in tokio::runtime::global::with_state::{{closure}} (cell=0x7f2a0b9fc248 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/global.rs:100
#87 0x000055beadda304b in std::thread::local::LocalKey<T>::try_with (self=0x55beae5ef050 = {...}, f=size=2 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/thread/local.rs:262
#88 0x000055beadda18d8 in std::thread::local::LocalKey<T>::with (self=0x55beae5ef050 = {...}, f=size=2 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/thread/local.rs:239
#89 0x000055beadd5922b in tokio::runtime::global::with_state (state=tokio::runtime::global::State::ThreadPool = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/global.rs:83
#90 0x000055beadd59546 in tokio::runtime::global::with_thread_pool (thread_pool=0x7f2a0c0093b0 = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/global.rs:75
#91 0x000055beadd94ace in tokio::runtime::thread_pool::spawner::Spawner::enter (self=0x7f2a0c0093b0 = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/thread_pool/spawner.rs:44
#92 0x000055beadd58eef in tokio::runtime::spawner::Spawner::enter (self=0x7f2a0c0093a8 = {...}, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/spawner.rs:32
#93 0x000055beadd6fdaf in tokio::runtime::blocking::pool::Inner::run::{{closure}} () at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/blocking/pool.rs:269
#94 0x000055beadd9a857 in tokio::time::clock::Clock::enter (self=0x7f2a0c009300, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/time/clock.rs:30
#95 0x000055beadd6e2d2 in tokio::runtime::time::variant::with_default (handle=0x7f2a0c0093c0, clock=0x7f2a0c009300, f=size=1 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/time.rs:43
#96 0x000055beadd6fd28 in tokio::runtime::blocking::pool::Inner::run (self=0x7f2a0c009300 = {...}) at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/blocking/pool.rs:268
#97 0x000055beadd6fc31 in tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}} () at /home/appaquet/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.6/src/runtime/blocking/pool.rs:252
#98 0x000055beadd67fcb in std::sys_common::backtrace::__rust_begin_short_backtrace (f=size=2 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/sys_common/backtrace.rs:129
#99 0x000055beadd8e32e in std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/thread/mod.rs:469
#100 0x000055beadd54a2e in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=size=1 = {...}, _args=()) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:317
#101 0x000055beadd38335 in std::panicking::try::do_call (data=0x7f2a0b9fb7e0 <incomplete sequence \360\222>) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:287
#102 0x000055beae2ccaca in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:78
#103 0x000055beadd37baa in std::panicking::try (f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:265
#104 0x000055beadd54aae in std::panic::catch_unwind (f=size=1 = {...}) at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:396
#105 0x000055beadd8e12e in std::thread::Builder::spawn_unchecked::{{closure}} () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/thread/mod.rs:468
#106 0x000055beadd8e824 in core::ops::function::FnOnce::call_once{{vtable-shim}} () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ops/function.rs:227
#107 0x000055beae2b6c9f in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/liballoc/boxed.rs:942
#108 0x000055beae2cbb90 in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/liballoc/boxed.rs:942
#109 std::sys_common::thread::start_thread () at src/libstd/sys_common/thread.rs:13
#110 std::sys::unix::thread::Thread::new::thread_start () at src/libstd/sys/unix/thread.rs:79
#111 0x00007f2a1088c669 in start_thread (arg=<optimized out>) at pthread_create.c:479
#113 0x00007f2a1079a323 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The text was updated successfully, but these errors were encountered:

udoprog · 2020-01-04T16:33:14Z

The channel will only end when all senders are dropped (see docs for recv), and you still have one in scope (tx).

This works:
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=f75d8f3ccadd1f83b6605be50c3fc1c4

appaquet · 2020-01-04T16:47:47Z

This totally makes sense. I hopped to conclusion a bit too fast hunting down a deadlock I have by dropping a tokio_compat Runtime. See this gist. Unfortunately, I still don't have a minimal example for it. Feel free to close this bug if you want, I'll open another one if I find a minimal example.

appaquet · 2020-01-04T17:24:36Z

Well, I haven't found a minimal example yet, but I have found the source. It goes through the tokio::runtime::thread_pool::queue::global::Queue<T> pointers Mutex twice on the same stack. I may be wrong, but I think dropping the lock on the closed queue path should be safe. See my diff. I can open the PR if this seems appropriate.

appaquet · 2020-01-04T18:08:19Z

I think I got it.

This produces the same backtrace: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=d91e66ff621acdbfe836decf028c9a04

Backtrace: https://gist.github.com/appaquet/87de2b93871c47ae5e81f849fe8c8a93

This doesn't happen with my fix here: master...appaquet:queue-reintrant-deadlock

appaquet · 2020-01-08T01:18:42Z

I though 855d39f in 0.2.8 may have fixed the issue, but I still get the same deadlock. Dropping the lock before calling the closure still fixes the issue in my project: master...appaquet:queue-reintrant-deadlock

carllerche · 2020-01-08T02:22:00Z

Thanks, I will take a look

carllerche · 2020-01-09T17:56:31Z

Ah, thanks so much for the minimal repro. I missed it originally. I should have a PR up soon.

Previously, when the threaded scheduler was in the shutdown process, it would hold a lock while dropping in-flight tasks. If those tasks included a drop handler that attempted to wake a second task, the wake operation would attempt to acquire a lock held by the scheduler. This results in a deadlock. Dropping the lock before dropping tasks resolves the problem. Fixes #2046

carllerche · 2020-01-09T18:58:40Z

PR is up

Previously, when the threaded scheduler was in the shutdown process, it would hold a lock while dropping in-flight tasks. If those tasks included a drop handler that attempted to wake a second task, the wake operation would attempt to acquire a lock held by the scheduler. This results in a deadlock. Dropping the lock before dropping tasks resolves the problem. Fixes #2046

appaquet changed the title ~~Deadlock when using channels + spawn_blocking + block_on~~ Deadlock when dropping Runtime Jan 4, 2020

appaquet mentioned this issue Jan 6, 2020

Deadlock in destructor, somewhere in channels #2058

Closed

dfoxfranke mentioned this issue Jan 6, 2020

UdpSocket-related deadlock when dropping runtime #2061

Closed

Keats mentioned this issue Jan 8, 2020

Update tokio and re-enable check_site test getzola/zola#915

Merged

carllerche added the C-bug Category: This is a bug. label Jan 8, 2020

carllerche mentioned this issue Jan 9, 2020

rt: fix shutdown deadlock in threaded scheduler #2082

Merged

carllerche closed this as completed in #2082 Jan 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deadlock when dropping Runtime #2046

Deadlock when dropping Runtime #2046

appaquet commented Jan 4, 2020 •

edited

udoprog commented Jan 4, 2020

appaquet commented Jan 4, 2020

appaquet commented Jan 4, 2020

appaquet commented Jan 4, 2020 •

edited

appaquet commented Jan 8, 2020

carllerche commented Jan 8, 2020

carllerche commented Jan 9, 2020

carllerche commented Jan 9, 2020

Deadlock when dropping Runtime #2046

Deadlock when dropping Runtime #2046

Comments

appaquet commented Jan 4, 2020 • edited

Version

Platform

Description

udoprog commented Jan 4, 2020

appaquet commented Jan 4, 2020

appaquet commented Jan 4, 2020

appaquet commented Jan 4, 2020 • edited

appaquet commented Jan 8, 2020

carllerche commented Jan 8, 2020

carllerche commented Jan 9, 2020

carllerche commented Jan 9, 2020

appaquet commented Jan 4, 2020 •

edited

appaquet commented Jan 4, 2020 •

edited