Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Potential panic in FuturesUnordered #2529

Open
gperinazzo opened this issue Dec 8, 2021 · 21 comments · Fixed by #2535
Open

Potential panic in FuturesUnordered #2529

gperinazzo opened this issue Dec 8, 2021 · 21 comments · Fixed by #2535
Labels
A-stream bug

Comments

@gperinazzo
Copy link

@gperinazzo gperinazzo commented Dec 8, 2021

EDIT(taiki-e): The regression that occurred in 0.3.18 has been fixed in 0.3.19, but the underlying problem that 0.3.18 tried to fix has not been fixed yet. This issue has been reused to track that problem as there is a detailed discussion on that problem.


Hey folks,

After upgrading to futures 0.3.18 from 0.3.17, we've noticed our application is panicking under high load. The panic seems to originate in src/stream/futures_unordered/ready_to_run_queue.rs:104 due to the error message inconsistent in drop:

thread 'tokio-runtime-worker' panicked at 'inconsistent in drop', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/abort.rs:11:5

 stack backtrace:
    0: rust_begin_unwind
              at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
    1: std::panicking::begin_panic_fmt
              at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:460:5
    2: futures_util::stream::futures_unordered::abort::abort
    3: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::ops::drop::Drop>::drop
    4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
    5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
    6: <mysql_async::BoxFuture<T> as core::future::future::Future>::poll
    7: <mysql_async::conn::pool::futures::get_conn::GetConn as core::future::future::Future>::poll
    8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
    9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll

This panic causes a double panic that aborts the application. This seems to only happen when under high load, so it may be a race condition that only happens when there's multiple threads running futures from the FuturesUnordered. I haven't been able to create a minimal project to reproduce.

@gperinazzo gperinazzo changed the title FuturesOrdered panic in 0.3.18 FuturesUnordered panic in 0.3.18 Dec 9, 2021
@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 9, 2021

Was able to make a minimal example. It's pretty unreliable so you may need to run it for a while. Running in debug also seems to trigger the error faster than running in release.

 use futures::stream::{FuturesUnordered, StreamExt};
 
 async fn run_loop() {
     async fn test() {
         tokio::spawn(async {
             tokio::time::sleep(std::time::Duration::from_millis(1)).await;
         })
         .await
         .unwrap();
     }
     loop {
         let mut futures = FuturesUnordered::new();
         for _ in 0..4 {
             futures.push(test());
         }
         let _ = futures.next().await;
         drop(futures);
     }
 }
 
 #[tokio::main]
 async fn main() {
     run_loop().await
 }

Here is the backtrace generated when it panics:

thread 'main' panicked at 'inconsistent in drop', /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/abort.rs:11:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
   1: std::panicking::begin_panic_fmt
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:460:5
   2: futures_util::stream::futures_unordered::abort::abort
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/abort.rs:11:5
   3: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::clear
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/ready_to_run_queue.rs:104:42
   4: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::ops::drop::Drop>::drop
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/mod.rs:580:18
   5: core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<core::future::from_generator::GenFuture<playground::run_loop::{{closure}}::test::{{closure}}>>>
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ptr/mod.rs:188:1
   6: core::mem::drop
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/mem/mod.rs:898:24
   7: playground::run_loop::{{closure}}
             at ./src/main.rs:17:9
   8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
   9: playground::main::{{closure}}
             at ./src/main.rs:23:5
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
  11: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/park/thread.rs:263:54
  12: tokio::coop::with_budget::{{closure}}
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:106:9
  13: std::thread::local::LocalKey<T>::try_with
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:399:16
  14: std::thread::local::LocalKey<T>::with
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:375:9
  15: tokio::coop::with_budget
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:99:5
  16: tokio::coop::budget
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:76:5
  17: tokio::park::thread::CachedParkThread::block_on
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/park/thread.rs:263:31
  18: tokio::runtime::enter::Enter::block_on
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/enter.rs:151:13
  19: tokio::runtime::thread_pool::ThreadPool::block_on
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/thread_pool/mod.rs:77:9
  20: tokio::runtime::Runtime::block_on
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/mod.rs:463:43
  21: playground::main
             at ./src/main.rs:23:5
  22: core::ops::function::FnOnce::call_once
             at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'panicking twice to abort the program', /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/abort.rs:6:13
stack backtrace:
   0:     0x55f74103addc - std::backtrace_rs::backtrace::libunwind::trace::h3fea1eb2e0ba2ac9
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x55f74103addc - std::backtrace_rs::backtrace::trace_unsynchronized::h849d83492cbc0d59
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55f74103addc - std::sys_common::backtrace::_print_fmt::he3179d37290f23d3
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x55f74103addc - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h140f6925cad14324
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x55f74105698c - core::fmt::write::h31b9cd1bedd7ea38
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/fmt/mod.rs:1150:17
   5:     0x55f741038a65 - std::io::Write::write_fmt::h1fdf66f83f70913e
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/io/mod.rs:1667:15
   6:     0x55f74103c900 - std::sys_common::backtrace::_print::he7ac492cd19c3189
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x55f74103c900 - std::sys_common::backtrace::print::hba20f8920229d8e8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x55f74103c900 - std::panicking::default_hook::{{closure}}::h714d63979ae18678
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:210:50
   9:     0x55f74103c4b7 - std::panicking::default_hook::hf1afb64e69563ca8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:227:9
  10:     0x55f74103cfb4 - std::panicking::rust_panic_with_hook::h02231a501e274a13
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:624:17
  11:     0x55f7410242c2 - std::panicking::begin_panic::{{closure}}::h3e31aaa543823395
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:544:9
  12:     0x55f741023b79 - std::sys_common::backtrace::__rust_end_short_backtrace::h183aedad7027b78d
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:141:18
  13:     0x55f741024203 - std::panicking::begin_panic::hab564a8571e43804
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:543:12
  14:     0x55f741023671 - <futures_util::stream::futures_unordered::abort::abort::DoublePanic as core::ops::drop::Drop>::drop::hd10fb214b1fe1391
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/abort.rs:6:13
  15:     0x55f7410236cb - core::ptr::drop_in_place<futures_util::stream::futures_unordered::abort::abort::DoublePanic>::h9d3548bc7115d38f
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ptr/mod.rs:188:1
  16:     0x55f741023609 - futures_util::stream::futures_unordered::abort::abort::hcb1516dd393c59b8
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/abort.rs:12:1
  17:     0x55f740f417d3 - futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::clear::hd3e1526563b13317
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/ready_to_run_queue.rs:104:42
  18:     0x55f740f3848a - <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::ops::drop::Drop>::drop::h1f3294c5552dfe48
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/stream/futures_unordered/mod.rs:580:18
  19:     0x55f740f38be3 - core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<core::future::from_generator::GenFuture<playground::run_loop::{{closure}}::test::{{closure}}>>>::h5449b52effbf085e
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ptr/mod.rs:188:1
  20:     0x55f740f36166 - core::mem::drop::h36e4c622fcb52977
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/mem/mod.rs:898:24
  21:     0x55f740f35448 - playground::run_loop::{{closure}}::hb1fa7de71e337272
                               at /home/guilherme/Projects/playground/src/main.rs:17:9
  22:     0x55f740f2c08b - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h5cfe28750a98defa
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
  23:     0x55f740f35a64 - playground::main::{{closure}}::h62dde6d44e9d9107
                               at /home/guilherme/Projects/playground/src/main.rs:23:5
  24:     0x55f740f2c11b - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h7a8b81dafb2b7391
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/future/mod.rs:80:19
  25:     0x55f740f3cf10 - tokio::park::thread::CachedParkThread::block_on::{{closure}}::h590c4888c9fa719a
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/park/thread.rs:263:54
  26:     0x55f740f32b90 - tokio::coop::with_budget::{{closure}}::hf20cf8f2eff9aa4a
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:106:9
  27:     0x55f740f2b8f8 - std::thread::local::LocalKey<T>::try_with::h040973a9d010b9c9
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:399:16
  28:     0x55f740f2b77d - std::thread::local::LocalKey<T>::with::hcfe29299a13b9b13
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/thread/local.rs:375:9
  29:     0x55f740f3c966 - tokio::coop::with_budget::h5ee5df02a49d6b6a
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:99:5
  30:     0x55f740f3c966 - tokio::coop::budget::hab2dbd7c022946ce
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/coop.rs:76:5
  31:     0x55f740f3c966 - tokio::park::thread::CachedParkThread::block_on::h49116734c7a78377
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/park/thread.rs:263:31
  32:     0x55f740f3d066 - tokio::runtime::enter::Enter::block_on::hb139c2e42e0fbed2
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/enter.rs:151:13
  33:     0x55f740f331ab - tokio::runtime::thread_pool::ThreadPool::block_on::hb113965d8c9fe0fa
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/thread_pool/mod.rs:77:9
  34:     0x55f740f2ddab - tokio::runtime::Runtime::block_on::h8c32b678e65eab3a
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.14.0/src/runtime/mod.rs:463:43
  35:     0x55f740f39b78 - playground::main::h1cd681073ddf11ba
                               at /home/guilherme/Projects/playground/src/main.rs:23:5
  36:     0x55f740f3866b - core::ops::function::FnOnce::call_once::h8b9e810d04134aad
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:227:5
  37:     0x55f740f3ff4e - std::sys_common::backtrace::__rust_begin_short_backtrace::h12a66cefae139e80
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:125:18
  38:     0x55f740f3dfd1 - std::rt::lang_start::{{closure}}::ha6f842ca8ef64dd8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/rt.rs:63:18
  39:     0x55f74103d5ba - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h2790017aba790142
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/ops/function.rs:259:13
  40:     0x55f74103d5ba - std::panicking::try::do_call::hd5d0fbb7d2d2d85d
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:403:40
  41:     0x55f74103d5ba - std::panicking::try::h675520ee37b0fdf7
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:367:19
  42:     0x55f74103d5ba - std::panic::catch_unwind::h803430ea0284ce79
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panic.rs:129:14
  43:     0x55f74103d5ba - std::rt::lang_start_internal::{{closure}}::h3a398a8154de3106
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/rt.rs:45:48
  44:     0x55f74103d5ba - std::panicking::try::do_call::hf60f106700df94b2
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:403:40
  45:     0x55f74103d5ba - std::panicking::try::hb2022d2bc87a9867
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:367:19
  46:     0x55f74103d5ba - std::panic::catch_unwind::hbf801c9d61f0c2fb
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panic.rs:129:14
  47:     0x55f74103d5ba - std::rt::lang_start_internal::hdd488b91dc742b96
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/rt.rs:45:20
  48:     0x55f740f3dfa0 - std::rt::lang_start::h98bc61135b206bb8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/rt.rs:62:5
  49:     0x55f740f39c0c - main
  50:     0x7f62efa100b3 - __libc_start_main
  51:     0x55f740f2972e - _start
  52:                0x0 - <unknown>
thread panicked while panicking. aborting.
Illegal instruction (core dumped)

@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 10, 2021

I've run the code above for 6 hours on 0.3.17 with no crashes so far. With the 0.3.18 version of the crate it usually crashes within the first 20 minutes.

I think it's safe to assume that the issue was introduced by the changes in 0.3.18.

@taiki-e
Copy link
Member

@taiki-e taiki-e commented Dec 10, 2021

Thanks for the report! Considering that 9080b83 is the only one that changed the code related to FuturesUnordered between 0.3.17 and 0.3.18 (0.3.17...0.3.18), it seems likely to be related to #2493 (cc @ibraheemdev).

@taiki-e taiki-e added A-stream bug labels Dec 10, 2021
@ibraheemdev
Copy link
Contributor

@ibraheemdev ibraheemdev commented Dec 10, 2021

That must mean ReadyToRunQueue is dropped outside of FuturesUnordered::drop. Any idea where?

@ibraheemdev
Copy link
Contributor

@ibraheemdev ibraheemdev commented Dec 10, 2021

Could it be

let inner = match arc_self.ready_to_run_queue.upgrade() {

@ibraheemdev
Copy link
Contributor

@ibraheemdev ibraheemdev commented Dec 10, 2021

Hm the stack trace shows:

<futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::ops::drop::Drop>::drop

@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 10, 2021

Yes, the debug traceback makes it clear that it's panicking in the clear called inside the drop implementation.

The initial backtrace is from a release binary and does not show all calls. It's probably being inlined.

@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 10, 2021

That does make sense, another thread may be holding a reference through that Weak reference. In that case it could run clear while another thread is trying to enqueue.

@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 10, 2021

Okay, I understand the issue now: ReadyToRunQueue::clearrequires unique access to the ReadyToRunQueue. During the call to FuturesUnordered::clear or FuturesUnordered::drop you have unique access to FuturesUnordered, but not to ReadyToRunQueue.

This comment is thus wrong and you should not call ReadyToRunQueue::clear there:

// SAFETY: we just cleared all the tasks and we have &mut self

This can be prevented from being mistakenly changed again by making ReadyToRunQueue::clear require a mutable reference.

@magnet
Copy link

@magnet magnet commented Dec 14, 2021

I've also hit this issue. Since this is a heisenbug that anyone can get through cargo update, I suggest 0.3.18 be yanked and a 0.3.19 cut with a fix. I'd be happy to provide a patch but I'm not sure if a revert #2493 would be accepted or if @ibraheemdev wants try to fix it differently.

@taiki-e
Copy link
Member

@taiki-e taiki-e commented Dec 14, 2021

(I've yanked futures/futures-util 0.3.18)

thomasmatecki added a commit to thomasmatecki/rg3d that referenced this issue Dec 15, 2021
bors bot added a commit to nervosnetwork/ckb that referenced this issue Dec 16, 2021
3243: chore: downgrade futures version r=yangby-cryptape,chanhsu001 a=driftluo

### What problem does this PR solve?

ref rust-lang/futures-rs#2529, 0.3.18 has been yank

### Check List

Tests

- No code ci-runs-only: [ quick_checks,linters ]

### Release note

```release-note
None: Exclude this PR from the release note.
```



Co-authored-by: driftluo <driftluo@foxmail.com>
paulip1792 added a commit to paulip1792/snarkOS that referenced this issue Dec 16, 2021
str4d added a commit to str4d/zcash that referenced this issue Dec 16, 2021
In the three days since the last update, `futures 0.3.18` was yanked due
to panics: rust-lang/futures-rs#2529
bors bot added a commit to probe-rs/probe-rs that referenced this issue Dec 17, 2021
929: Fall back to older version of futures, version 0.3.18 was yanked r=Yatekii a=Tiwalun

Revert futures to version 0.3.17 in target-gen, version 0.3.18 was yanked.

See rust-lang/futures-rs#2529.


Co-authored-by: Dominik Boehi <dominik.boehi@gmail.com>
@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 18, 2021

Thank you @taiki-e!
Just for future reference, the FuturesUnordered::clear method is probably still unsound and may cause this panic.

@taiki-e
Copy link
Member

@taiki-e taiki-e commented Dec 18, 2021

@gperinazzo You are right. I'll re-use this issue to track that problem.

EDIT: To clarify, FuturesUnordered will panic to prevent unsoundness, so "unsound" is not strictly correct.

@taiki-e taiki-e reopened this Dec 18, 2021
@taiki-e taiki-e changed the title FuturesUnordered panic in 0.3.18 Potential panic in FuturesUnordered Dec 18, 2021
@taiki-e
Copy link
Member

@taiki-e taiki-e commented Dec 18, 2021

Published 0.3.19 & updated title and description of this issue.

EDIT(taiki-e): The regression that occurred in 0.3.18 has been fixed in 0.3.19, but the underlying problem that 0.3.18 tried to fix has not been fixed yet. This issue has been reused to track that problem as there is a detailed discussion on that problem.

@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 18, 2021

Just to confirm: I was able to trigger this panic with the following code using futures-util 0.3.19. Instead of dropping the FuturesUnordered, it clears it instead.

use futures::stream::{FuturesUnordered, StreamExt};
 
async fn run_loop() {
    async fn test() {
        tokio::spawn(async {
            tokio::time::sleep(std::time::Duration::from_micros(100)).await;
        })
        .await
        .unwrap();
    }
    let mut futures = FuturesUnordered::new();
    loop {
        for _ in 0..24 {
            futures.push(test());
        }
        let _ = futures.next().await;
        futures.clear();
    }
}
 
#[tokio::main]
async fn main() {
    run_loop().await
}

Here's the backtrace:

thread 'main' panicked at 'inconsistent in drop', /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/abort.rs:11:5
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
   2: core::panicking::panic_display
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:64:5
   3: futures_util::stream::futures_unordered::abort::abort
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/abort.rs:11:5
   4: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::clear
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/ready_to_run_queue.rs:104:42
   5: futures_util::stream::futures_unordered::FuturesUnordered<Fut>::clear
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/mod.rs:577:18
   6: playground::run_loop::{{closure}}
             at ./src/main.rs:19:10
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/future/mod.rs:80:19
   8: playground::main::{{closure}}
             at ./src/main.rs:25:6
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/future/mod.rs:80:19
  10: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/park/thread.rs:263:54
  11: tokio::coop::with_budget::{{closure}}
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:102:9
  12: std::thread::local::LocalKey<T>::try_with
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/local.rs:399:16
  13: std::thread::local::LocalKey<T>::with
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/local.rs:375:9
  14: tokio::coop::with_budget
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:95:5
  15: tokio::coop::budget
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:72:5
  16: tokio::park::thread::CachedParkThread::block_on
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/park/thread.rs:263:31
  17: tokio::runtime::enter::Enter::block_on
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/enter.rs:151:13
  18: tokio::runtime::thread_pool::ThreadPool::block_on
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/thread_pool/mod.rs:77:9
  19: tokio::runtime::Runtime::block_on
             at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/mod.rs:463:43
  20: playground::main
             at ./src/main.rs:25:6
  21: core::ops::function::FnOnce::call_once
             at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'panicking twice to abort the program', /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/abort.rs:6:13
stack backtrace:
   0:     0x559e209d7aac - std::backtrace_rs::backtrace::libunwind::trace::hf6a6dfd7da937cb0
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x559e209d7aac - std::backtrace_rs::backtrace::trace_unsynchronized::hc596a19e4891f7f3
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x559e209d7aac - std::sys_common::backtrace::_print_fmt::hb16700db31584325
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x559e209d7aac - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h231c4190cfa75162
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x559e209f326c - core::fmt::write::h2a1462b5f8eea807
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/fmt/mod.rs:1163:17
   5:     0x559e209d4db5 - std::io::Write::write_fmt::h71ddfebc68685972
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/io/mod.rs:1696:15
   6:     0x559e209d9630 - std::sys_common::backtrace::_print::hcc197d4bebf2b369
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x559e209d9630 - std::sys_common::backtrace::print::h335a66af06738c7c
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x559e209d9630 - std::panicking::default_hook::{{closure}}::h6fac9ac9c8b79e52
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:210:50
   9:     0x559e209d91e5 - std::panicking::default_hook::h341c1030c6a1161b
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:227:9
  10:     0x559e209d9ce4 - std::panicking::rust_panic_with_hook::h50680ff4b44510c6
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:624:17
  11:     0x559e209bf512 - std::panicking::begin_panic::{{closure}}::h1b89cc6a9f0c65b3
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:544:9
  12:     0x559e209bec59 - std::sys_common::backtrace::__rust_end_short_backtrace::hf940603df819fbdb
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:139:18
  13:     0x559e209bf453 - std::panicking::begin_panic::h11022e2314b2fb38
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:543:12
  14:     0x559e209beac1 - <futures_util::stream::futures_unordered::abort::abort::DoublePanic as core::ops::drop::Drop>::drop::he8507aa55a3a84b7
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/abort.rs:6:13
  15:     0x559e209beb1b - core::ptr::drop_in_place<futures_util::stream::futures_unordered::abort::abort::DoublePanic>::hff39063cc2b382f6
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ptr/mod.rs:188:1
  16:     0x559e209bea7f - futures_util::stream::futures_unordered::abort::abort::h5967e6ccbe1f47a5
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/abort.rs:12:1
  17:     0x559e208c8b43 - futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<Fut>::clear::h81a7f4488e0e44d2
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/ready_to_run_queue.rs:104:42
  18:     0x559e208d8ddc - futures_util::stream::futures_unordered::FuturesUnordered<Fut>::clear::he172cdbc1363475b
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.19/src/stream/futures_unordered/mod.rs:577:18
  19:     0x559e208c96fa - playground::run_loop::{{closure}}::h25f81113a9122934
                               at /home/guilherme/Projects/playground/src/main.rs:19:10
  20:     0x559e208df09b - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h9b7c151830a6f9ba
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/future/mod.rs:80:19
  21:     0x559e208c9d64 - playground::main::{{closure}}::h82c4b817c85c916a
                               at /home/guilherme/Projects/playground/src/main.rs:25:6
  22:     0x559e208df12b - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hbc207e880d0c5150
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/future/mod.rs:80:19
  23:     0x559e208daab0 - tokio::park::thread::CachedParkThread::block_on::{{closure}}::he246966113faec30
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/park/thread.rs:263:54
  24:     0x559e208cd060 - tokio::coop::with_budget::{{closure}}::hc6272a8800003682
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:102:9
  25:     0x559e208d1fb8 - std::thread::local::LocalKey<T>::try_with::h3afe2a6ce7ca958d
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/local.rs:399:16
  26:     0x559e208d1c9d - std::thread::local::LocalKey<T>::with::haf2f1f3e313f4027
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/local.rs:375:9
  27:     0x559e208da4b6 - tokio::coop::with_budget::h4ccdde3408d40b67
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:95:5
  28:     0x559e208da4b6 - tokio::coop::budget::h07e230f3c95151b4
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:72:5
  29:     0x559e208da4b6 - tokio::park::thread::CachedParkThread::block_on::h9690827b744d6f8e
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/park/thread.rs:263:31
  30:     0x559e208ca466 - tokio::runtime::enter::Enter::block_on::he88fd54a6b975f24
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/enter.rs:151:13
  31:     0x559e208dc99b - tokio::runtime::thread_pool::ThreadPool::block_on::h047d95c27394e031
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/thread_pool/mod.rs:77:9
  32:     0x559e208cb50b - tokio::runtime::Runtime::block_on::hdd1e602b66718309
                               at /home/guilherme/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/mod.rs:463:43
  33:     0x559e208cd2a8 - playground::main::he8e0915a5ce2489b
                               at /home/guilherme/Projects/playground/src/main.rs:25:6
  34:     0x559e208cb87b - core::ops::function::FnOnce::call_once::h5831a4d9276cf31d
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5
  35:     0x559e208dbbfe - std::sys_common::backtrace::__rust_begin_short_backtrace::h85e2aa80bcec238b
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:123:18
  36:     0x559e208d3051 - std::rt::lang_start::{{closure}}::h890e36e0b0c884e3
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/rt.rs:146:18
  37:     0x559e209d778b - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hc56adab7a77ec6e3
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:259:13
  38:     0x559e209d778b - std::panicking::try::do_call::h29f013120c5abc65
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
  39:     0x559e209d778b - std::panicking::try::h86d5b2b66caec4cf
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
  40:     0x559e209d778b - std::panic::catch_unwind::h7dd136d787f51397
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
  41:     0x559e209d778b - std::rt::lang_start_internal::{{closure}}::h4a199351e630a8a5
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/rt.rs:128:48
  42:     0x559e209d778b - std::panicking::try::do_call::h20ceb9e5dff838c6
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
  43:     0x559e209d778b - std::panicking::try::hc2abb46a5e41bd43
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
  44:     0x559e209d778b - std::panic::catch_unwind::h52c3eb4408ad6dfb
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
  45:     0x559e209d778b - std::rt::lang_start_internal::hd15a47be08101c28
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/rt.rs:128:20
  46:     0x559e208d3020 - std::rt::lang_start::he8455f954ebd2376
                               at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/rt.rs:145:17
  47:     0x559e208cd33c - main
  48:     0x7fb2a4d91b25 - __libc_start_main
  49:     0x559e208c670e - _start
  50:                0x0 - <unknown>
thread panicked while panicking. aborting.
Illegal instruction (core dumped)

@ibraheemdev
Copy link
Contributor

@ibraheemdev ibraheemdev commented Dec 18, 2021

If clear is unsound, then isn't next too?

        loop {
            // Safety: &mut self guarantees the mutual exclusion `dequeue`
            // expects
            let task = match unsafe { self.ready_to_run_queue.dequeue() } {

@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 18, 2021

dequeue expects mutual exclusion from the consumer side of the MPSC queue, this is safe as long as no other thread is calling dequeue. Because only FuturesUnordered calls dequeue (and the clear method, but it requires mutual exclusion) in the code, it's safe to do so if you have a mut reference to FuturesUnordered.

In some cases, while another thread is pushing a into the queue, the dequeue method returns Inconsistent, which means it's in the middle of a push operation. pool_next seems to handle this case by yielding and asking to be woken up right away again (which should give the pushing thread enough time to finish the operation).

The clear method, however, expects no one to be pushing to the queue while it's executing. It'll explicitly panic when that happens, and that is the panic being observed:

            // SAFETY: We have the guarantee of mutual exclusion required by `dequeue`.
            match self.dequeue() {
                Dequeue::Empty => break,
                Dequeue::Inconsistent => abort("inconsistent in drop"), // <- this is where it's generating the stack traces
                Dequeue::Data(ptr) => drop(Arc::from_raw(ptr)),
            }

Maybe we can remove this requirement from clear, and handle the Inconsistent return in some other way? You could just spin lock in there until the queue is empty, ignoring the Inconsistent case, or make this method async and do the same as poll_next.

@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Dec 18, 2021

Another alternative is to just create a new ReadyToRunQueue and replace it after dropping all futures. That will either drop the existing queue on the spot (and call clear) or when the last task that currently has a valid pointer to it finishes.

@taiki-e
Copy link
Member

@taiki-e taiki-e commented Jan 9, 2022

Another alternative is to just create a new ReadyToRunQueue and replace it after dropping all futures.

SGTM

@SHaaD94
Copy link

@SHaaD94 SHaaD94 commented Jun 18, 2022

Hi everyone, just encountered this issue in "0.3.21".
Any ideas about when could it be fixed?

Cheers.

@gperinazzo
Copy link
Author

@gperinazzo gperinazzo commented Jun 18, 2022

Was it while calling clear on FuturesUnordered? If that's the case, you can fix your issue by dropping it and making a new one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-stream bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants