Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tokio-based tcp transports panic when dialing due to threadpools with unset runtimes. #2230

Open
mental32 opened this issue Sep 13, 2021 · 7 comments
Open

tokio-based tcp transports panic when dialing due to threadpools with unset runtimes. #2230

mental32 opened this issue Sep 13, 2021 · 7 comments

Comments

@mental32
Copy link

mental32 commented Sep 13, 2021

I've spent the last couple of hours trying to figure this out but the best I can boil it down into is:

  • When using a tokio-based transport such as libp2p::tokio_development_transport
  • A call to libp2p::swarm::Swarm::new ends up using the Default constructed fields and calls SwarmBuilder::build
  • And using a Default constructed NetworkConfig which sets the executor to None a call is made to or_with_executor(f) to construct a futures::executor, ThreadPool as the backing executor.
  • Any further calls to that use the executor that also interact with network machinery (like Swarm::dial_addr) will eventually lead to a panic because:
    • it'll try to crate a tokio TcpStream which at some point calls tokio::io::PollEvented::new_with_interest
    • Which then calls tokio::Handle::current and that uses tokio::runtime::context::io_handle()
  • At which point we fail with the missing runtime message: "there is no reactor running, must be called from the context of a Tokio 1.x runtime"

Here are my deps:

[dependencies]
futures = "0.3.17"
libp2p = { version = "0.39.1", default-features = false, features = [
    "deflate",
    "gossipsub",
    "mdns",
    "tcp-tokio",
    "dns-tokio",
    "websocket",
    "noise",
    "mplex",
    "yamux",
] }

tokio = { version = "1.11.0", features = ["full"] }

Potential solutions I can think of are:

  • Ask the transport for an executor during SwarmBuilder::build so tokio transports can produce a well-configured executor and the default would just be the futures ThreadPool executor closure that is present now.

  • Document that tokio transports must set their own executor manually using SwarmBuilder

Expected behavior

The swarm can dial the supplied Multiaddr.

Actual behavior

Tokio panics because of an unset thread local deep in the io machinery, the context required to access a runtime.

Reproducible example

Directory layout:

/tmp/asdasfa
 ├──Cargo.lock
 ├──Cargo.toml
 └──src
    └──main.rs

src/main.rs

use std::time::Duration;

use libp2p::core::identity::Keypair;
use libp2p::gossipsub::{self, Gossipsub, GossipsubEvent, MessageAuthenticity, ValidationMode};
use libp2p::mdns::{Mdns, MdnsConfig, MdnsEvent};
use libp2p::swarm::NetworkBehaviourEventProcess;
use libp2p::PeerId;
use libp2p::{Multiaddr, NetworkBehaviour};

use tokio::sync::mpsc::{unbounded_channel, UnboundedSender};

#[derive(NetworkBehaviour)]
pub struct Node {
    /// GossipSub is our main PUB/SUB behaviour.
    pub gossipsub: Gossipsub,

    /// mDNS (DNS Multicast) is used for local network peer discovery.
    pub mdns: Mdns,

    #[behaviour(ignore)]
    pub tx: UnboundedSender<Box<dyn Iterator<Item = (PeerId, Multiaddr)> + Send>>,
}

impl NetworkBehaviourEventProcess<GossipsubEvent> for Node {
    fn inject_event(&mut self, event: GossipsubEvent) {
        match dbg!(event) {
            GossipsubEvent::Message { .. } => (),
            GossipsubEvent::Subscribed { .. } => (),
            GossipsubEvent::Unsubscribed { .. } => (),
        }
    }
}

impl NetworkBehaviourEventProcess<MdnsEvent> for Node {
    fn inject_event(&mut self, event: MdnsEvent) {
        let _ = match event {
            MdnsEvent::Discovered(it) => self.tx.send(Box::new(it)),
            MdnsEvent::Expired(_) => todo!(),
        };
    }
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let local_key = Keypair::generate_ed25519();
    let peer_id = PeerId::from_public_key(local_key.public());

    let transport = libp2p::tokio_development_transport(local_key.clone())?;

    // Set a custom gossipsub
    let gossipsub_config = gossipsub::GossipsubConfigBuilder::default()
        .heartbeat_interval(Duration::from_secs(10)) // This is set to aid debugging by not cluttering the log space
        .validation_mode(ValidationMode::Strict) // This sets the kind of message validation. The default is Strict (enforce message signing)
        .build()
        .expect("Valid config");

    // Create a Swarm to manage peers and events
    let mdns = Mdns::new(MdnsConfig::default()).await.unwrap();

    // build a gossipsub network behaviour
    let gossipsub = gossipsub::Gossipsub::new(
        MessageAuthenticity::Signed(local_key.clone()),
        gossipsub_config,
    )
    .expect("Correct configuration");

    let (tx, mut rx) = unbounded_channel();

    let behaviour = Node {
        gossipsub,
        mdns,
        tx,
    };

    let mut swarm = libp2p::swarm::Swarm::new(transport, behaviour, peer_id.clone());

    let listen_addr = "/ip4/0.0.0.0/tcp/0".parse().unwrap();

    swarm.listen_on(listen_addr).unwrap();

    use futures::StreamExt;

    loop {
        tokio::select! {
            event = swarm.next() => {
                dbg!(event);
            }

            it = rx.recv() => { if let Some(it) = it {
                for (_p, m) in it {
                    swarm.dial_addr(m).unwrap();  // boom!
                }
            } }

            else => continue,
        };
    }
}

/cargo.toml

[package]
name = "asdasfa"
version = "0.1.0"
edition = "2018"

[dependencies]
futures = "0.3.17"
libp2p = { version = "0.39.1", default-features = false, features = [
    "deflate",
    "gossipsub",
    "mdns",
    "tcp-tokio",
    "dns-tokio",
    "websocket",
    "noise",
    "mplex",
    "yamux",
] }

tokio = { version = "1.11.0", features = ["full"] }

Output & Backtrace

Client A with backtrace enabled:

[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/172.18.0.1/tcp/45485",
    },
)
[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/127.0.0.1/tcp/45485",
    },
)
[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/172.28.0.1/tcp/45485",
    },
)
[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/172.17.0.1/tcp/45485",
    },
)
[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/192.168.0.157/tcp/45485",
    },
)
[src/main.rs:91] p = PeerId(
    "12D3KooWHKzF4W3DVoAxjNFkUYABXighTckwqX13VZLDpbt169iW",
)
[src/main.rs:95] m = "/ip4/192.168.0.157/tcp/45709"
[src/main.rs:91thread 'libp2p-swarm-task-0' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:26
stack backtrace:
] p = PeerId(
    "12D3KooWHKzF4W3DVoAxjNFkUYABXighTckwqX13VZLDpbt169iW",
)
[src/main.rs:95] m = "/ip4/172.17.0.1/tcp/45709"
[src/main.rs:91] p = PeerId(
    "12D3KooWHKzF4W3DVoAxjNFkUYABXighTckwqX13VZLDpbt169iW",
)
[src/main.rs:95] m = "/ip4/172.18.0.1/tcp/45709"
[src/main.rs:91] p = PeerId(
    "12D3KooWHKzF4W3DVoAxjNFkUYABXighTckwqX13VZLDpbt169iW",
)
[src/main.rs:thread '95libp2p-swarm-task-1] ' panicked at 'mthere is no reactor running, must be called from the context of a Tokio 1.x runtime = ', /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:26
"/ip4/172.28.0.1/tcp/45709"
[src/main.rs:91] p = thread 'PeerIdlibp2p-swarm-task-2' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:26
(
    "12D3KooWHKzF4W3DVoAxjNFkUYABXighTckwqX13VZLDpbt169iW",
)
[src/main.rs:95] m = "/ip4/127.0.0.1/tcp/45709"
thread 'libp2p-swarm-task-3' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:26
[src/main.rs:86] event = Some(
    IncomingConnection {
        local_addr: "/ip4/192.168.0.157/tcp/45485",
        send_back_addr: "/ip4/192.168.0.157/tcp/54260",
    },
)
[src/main.rs:86] event = Some(
    IncomingConnection {
        local_addr: "/ip4/127.0.0.1/tcp/45485",
        send_back_addr: "/ip4/127.0.0.1/tcp/43562",
    },
)
[src/main.rs:86] event = Some(
    IncomingConnection {
        local_addr: "/ip4/172.28.0.1/tcp/45485",
        send_back_addr: "/ip4/192.168.0.157/tcp/59462",
    },
)
[src/main.rs:86] event = Some(
    IncomingConnection {
        local_addr: "/ip4/172.18.0.1/tcp/45485",
        send_back_addr: "/ip4/192.168.0.157/tcp/53954",
    },
)
   0: rust_begin_unwind
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/panicking.rs:93:14
   2: core::option::expect_failed
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/option.rs:1618:5
   3: core::option::Option<T>::expect
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/option.rs:698:21
   4: tokio::runtime::context::io_handle::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:13
   5: std::thread::local::LocalKey<T>::try_with
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/thread/local.rs:399:16
   6: std::thread::local::LocalKey<T>::with
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/thread/local.rs:375:9
   7: tokio::runtime::context::io_handle
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:16:9
   8: tokio::io::driver::Handle::current
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/driver/mod.rs:262:13
   9: tokio::io::poll_evented::PollEvented<E>::new_with_interest
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/poll_evented.rs:101:58
  10: tokio::io::poll_evented::PollEvented<E>::new
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/poll_evented.rs:81:9
  11: tokio::net::tcp::stream::TcpStream::from_std
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/net/tcp/stream.rs:191:18
  12: <tokio::net::tcp::stream::TcpStream as core::convert::TryFrom<std::net::tcp::TcpStream>>::try_from
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/net/tcp/stream.rs:1251:9
  13: <libp2p_tcp::provider::tokio::Tcp as libp2p_tcp::provider::Provider>::new_stream::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-tcp-0.29.0/src/provider/tokio.rs:68:26
  14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  15: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  16: libp2p_tcp::GenTcpConfig<T>::do_dial::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-tcp-0.29.0/src/lib.rs:363:22
  17: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  18: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  19: <libp2p_dns::GenDnsConfig<T,C,P> as libp2p_core::transport::Transport>::dial::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-dns-0.29.0/src/lib.rs:292:29
  20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  21: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  22: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/either.rs:90:33
  23: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  24: <libp2p_core::either::EitherFuture<AFuture,BFuture> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/either.rs:390:43
  25: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  26: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/and_then.rs:157:38
  27: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  28: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/and_then.rs:157:38
  29: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  30: <libp2p_core::transport::timeout::Timeout<InnerFut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/timeout.rs:175:15
  31: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  32: <libp2p_core::transport::map::MapFuture<T,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/map.rs:133:26
  33: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  34: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  35: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  36: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  37: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  38: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  39: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  40: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  41: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  42: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  43: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  44: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  45: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  46: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  47: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  48: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  49: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  50: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/try_flatten.rs:49:61
  51: <futures_util::future::try_future::TryFlatten<Fut1,Fut2> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  52: <futures_util::future::try_future::AndThen<Fut1,Fut2,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  53: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  54: futures_util::future::future::FutureExt::poll_unpin
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/mod.rs:562:9
  55: <libp2p_core::connection::manager::task::Task<F,M,H,I,O,E> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/connection/manager/task.rs:221:27
  56: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  57: <futures_task::future_obj::LocalFutureObj<T> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.17/src/future_obj.rs:84:18
  58: <futures_task::future_obj::FutureObj<T> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.17/src/future_obj.rs:127:9
  59: futures_util::future::future::FutureExt::poll_unpin
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/mod.rs:562:9
  60: futures_executor::thread_pool::Task::run
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:322:27
  61: futures_executor::thread_pool::PoolState::work
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:154:39
  62: futures_executor::thread_pool::ThreadPoolBuilder::create::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:284:42
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: rust_begin_unwind
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/panicking.rs:93:14
   2: core::option::expect_failed
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/option.rs:1618:5
   3: core::option::Option<T>::expect
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/option.rs:698:21
   4: tokio::runtime::context::io_handle::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:13
   5: std::thread::local::LocalKey<T>::try_with
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/thread/local.rs:399:16
   6: std::thread::local::LocalKey<T>::with
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/thread/local.rs:375:9
   7: tokio::runtime::context::io_handle
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:16:9
   8: tokio::io::driver::Handle::current
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/driver/mod.rs:262:13
   9: tokio::io::poll_evented::PollEvented<E>::new_with_interest
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/poll_evented.rs:101:58
  10: tokio::io::poll_evented::PollEvented<E>::new
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/poll_evented.rs:81:9
  11: tokio::net::tcp::stream::TcpStream::from_std
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/net/tcp/stream.rs:191:18
  12: <tokio::net::tcp::stream::TcpStream as core::convert::TryFrom<std::net::tcp::TcpStream>>::try_from
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/net/tcp/stream.rs:1251:9
  13: <libp2p_tcp::provider::tokio::Tcp as libp2p_tcp::provider::Provider>::new_stream::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-tcp-0.29.0/src/provider/tokio.rs:68:26
  14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  15: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  16: libp2p_tcp::GenTcpConfig<T>::do_dial::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-tcp-0.29.0/src/lib.rs:363:22
  17: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  18: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  19: <libp2p_dns::GenDnsConfig<T,C,P> as libp2p_core::transport::Transport>::dial::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-dns-0.29.0/src/lib.rs:292:29
  20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  21: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  22: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/either.rs:90:33
  23: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  24: <libp2p_core::either::EitherFuture<AFuture,BFuture> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/either.rs:390:43
  25: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  26: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/and_then.rs:157:38
  27: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  28: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/and_then.rs:157:38
  29: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  30: <libp2p_core::transport::timeout::Timeout<InnerFut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/timeout.rs:175:15
  31: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  32: <libp2p_core::transport::map::MapFuture<T,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/map.rs:133:26
  33: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  34: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  35: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  36: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  37: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  38: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  39: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  40: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  41: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  42: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  43: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  44: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  45: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  46: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  47: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  48: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  49: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  50: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/try_flatten.rs:49:61
  51: <futures_util::future::try_future::TryFlatten<Fut1,Fut2> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  52: <futures_util::future::try_future::AndThen<Fut1,Fut2,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  53: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  54: futures_util::future::future::FutureExt::poll_unpin
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/mod.rs:562:9
  55: <libp2p_core::connection::manager::task::Task<F,M,H,I,O,E> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/connection/manager/task.rs:221:27
  56: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  57: <futures_task::future_obj::LocalFutureObj<T> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.17/src/future_obj.rs:84:18
  58: <futures_task::future_obj::FutureObj<T> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.17/src/future_obj.rs:127:9
  59: futures_util::future::future::FutureExt::poll_unpin
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/mod.rs:562:9
  60: futures_executor::thread_pool::Task::run
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:322:27
  61: futures_executor::thread_pool::PoolState::work
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:154:39
  62: futures_executor::thread_pool::ThreadPoolBuilder::create::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:284:42
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: rust_begin_unwind
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/panicking.rs:93:14
   2: core::option::expect_failed
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/option.rs:1618:5
   3: core::option::Option<T>::expect
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/option.rs:698:21
   4: tokio::runtime::context::io_handle::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:13
   5: std::thread::local::LocalKey<T>::try_with
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/thread/local.rs:399:16
   6: std::thread::local::LocalKey<T>::with
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/thread/local.rs:375:9
   7: tokio::runtime::context::io_handle
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:16:9
   8: tokio::io::driver::Handle::current
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/driver/mod.rs:262:13
   9: tokio::io::poll_evented::PollEvented<E>::new_with_interest
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/poll_evented.rs:101:58
  10: tokio::io::poll_evented::PollEvented<E>::new
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/poll_evented.rs:81:9
  11: tokio::net::tcp::stream::TcpStream::from_std
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/net/tcp/stream.rs:191:18
  12: <tokio::net::tcp::stream::TcpStream as core::convert::TryFrom<std::net::tcp::TcpStream>>::try_from
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/net/tcp/stream.rs:1251:9
  13: <libp2p_tcp::provider::tokio::Tcp as libp2p_tcp::provider::Provider>::new_stream::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-tcp-0.29.0/src/provider/tokio.rs:68:26
  14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  15: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  16: libp2p_tcp::GenTcpConfig<T>::do_dial::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-tcp-0.29.0/src/lib.rs:363:22
  17: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  18: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  19: <libp2p_dns::GenDnsConfig<T,C,P> as libp2p_core::transport::Transport>::dial::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-dns-0.29.0/src/lib.rs:292:29
  20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  21: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  22: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/either.rs:90:33
  23: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  24: <libp2p_core::either::EitherFuture<AFuture,BFuture> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/either.rs:390:43
  25: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  26: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/and_then.rs:157:38
  27: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  28: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/and_then.rs:157:38
  29: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  30: <libp2p_core::transport::timeout::Timeout<InnerFut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/timeout.rs:175:15
  31: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  32: <libp2p_core::transport::map::MapFuture<T,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/map.rs:133:26
  33: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  34: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  35: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  36: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  37: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  38: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  39: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  40: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  41: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  42: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  43: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  44: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  45: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  46: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  47: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  48: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  49: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  50: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/try_flatten.rs:49:61
  51: <futures_util::future::try_future::TryFlatten<Fut1,Fut2> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  52: <futures_util::future::try_future::AndThen<Fut1,Fut2,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  53: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  54: futures_util::future::future::FutureExt::poll_unpin
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/mod.rs:562:9
  55: <libp2p_core::connection::manager::task::Task<F,M,H,I,O,E> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/connection/manager/task.rs:221:27
  56: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  57: <futures_task::future_obj::LocalFutureObj<T> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.17/src/future_obj.rs:84:18
  58: <futures_task::future_obj::FutureObj<T> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.17/src/future_obj.rs:127:9
  59: futures_util::future::future::FutureExt::poll_unpin
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/mod.rs:562:9
  60: futures_executor::thread_pool::Task::run
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:322:27
  61: futures_executor::thread_pool::PoolState::work
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:154:39
  62: futures_executor::thread_pool::ThreadPoolBuilder::create::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:284:42
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: rust_begin_unwind
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/panicking.rs:93:14
   2: core::option::expect_failed
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/option.rs:1618:5
   3: core::option::Option<T>::expect
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/option.rs:698:21
   4: tokio::runtime::context::io_handle::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:13
   5: std::thread::local::LocalKey<T>::try_with
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/thread/local.rs:399:16
   6: std::thread::local::LocalKey<T>::with
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/std/src/thread/local.rs:375:9
   7: tokio::runtime::context::io_handle
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:16:9
   8: tokio::io::driver::Handle::current
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/driver/mod.rs:262:13
   9: tokio::io::poll_evented::PollEvented<E>::new_with_interest
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/poll_evented.rs:101:58
  10: tokio::io::poll_evented::PollEvented<E>::new
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/io/poll_evented.rs:81:9
  11: tokio::net::tcp::stream::TcpStream::from_std
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/net/tcp/stream.rs:191:18
  12: <tokio::net::tcp::stream::TcpStream as core::convert::TryFrom<std::net::tcp::TcpStream>>::try_from
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/net/tcp/stream.rs:1251:9
  13: <libp2p_tcp::provider::tokio::Tcp as libp2p_tcp::provider::Provider>::new_stream::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-tcp-0.29.0/src/provider/tokio.rs:68:26
  14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  15: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  16: libp2p_tcp::GenTcpConfig<T>::do_dial::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-tcp-0.29.0/src/lib.rs:363:22
  17: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  18: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  19: <libp2p_dns::GenDnsConfig<T,C,P> as libp2p_core::transport::Transport>::dial::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-dns-0.29.0/src/lib.rs:292:29
  20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/mod.rs:80:19
  21: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  22: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/either.rs:90:33
  23: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  24: <libp2p_core::either::EitherFuture<AFuture,BFuture> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/either.rs:390:43
  25: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  26: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/and_then.rs:157:38
  27: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  28: <libp2p_core::transport::and_then::AndThenFuture<TFut,TMap,TMapOut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/and_then.rs:157:38
  29: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  30: <libp2p_core::transport::timeout::Timeout<InnerFut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/timeout.rs:175:15
  31: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  32: <libp2p_core::transport::map::MapFuture<T,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/transport/map.rs:133:26
  33: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  34: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  35: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  36: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  37: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  38: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  39: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  40: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  41: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  42: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  43: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  44: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  45: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/into_future.rs:34:9
  46: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/map.rs:55:37
  47: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  48: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  49: <F as futures_core::future::TryFuture>::try_poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/future.rs:82:9
  50: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/try_future/try_flatten.rs:49:61
  51: <futures_util::future::try_future::TryFlatten<Fut1,Fut2> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  52: <futures_util::future::try_future::AndThen<Fut1,Fut2,F> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/lib.rs:95:13
  53: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  54: futures_util::future::future::FutureExt::poll_unpin
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/mod.rs:562:9
  55: <libp2p_core::connection::manager::task::Task<F,M,H,I,O,E> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-core-0.29.0/src/connection/manager/task.rs:221:27
  56: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/6d64f7f695943541fe12bb960971403f440d7225/library/core/src/future/future.rs:119:9
  57: <futures_task::future_obj::LocalFutureObj<T> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.17/src/future_obj.rs:84:18
  58: <futures_task::future_obj::FutureObj<T> as core::future::future::Future>::poll
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.17/src/future_obj.rs:127:9
  59: futures_util::future::future::FutureExt::poll_unpin
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.17/src/future/future/mod.rs:562:9
  60: futures_executor::thread_pool::Task::run
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:322:27
  61: futures_executor::thread_pool::PoolState::work
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:154:39
  62: futures_executor::thread_pool::ThreadPoolBuilder::create::{{closure}}
             at /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.17/src/thread_pool.rs:284:42
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

What it looks like from Client B:

[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/172.17.0.1/tcp/45709",
    },
)
[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/172.18.0.1/tcp/45709",
    },
)
[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/172.28.0.1/tcp/45709",
    },
)
[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/127.0.0.1/tcp/45709",
    },
)
[src/main.rs:86] event = Some(
    NewListenAddr {
        listener_id: ListenerId(
            1,
        ),
        address: "/ip4/192.168.0.157/tcp/45709",
    },
)
[src/main.rs:86] event = Some(
    IncomingConnection {
        local_addr: "/ip4/192.168.0.157/tcp/45709",
        send_back_addr: "/ip4/192.168.0.157/tcp/46742",
    },
)
[src/main.rs:86] event = Some(
    IncomingConnection {
        local_addr: "/ip4/172.17.0.1/tcp/45709",
        send_back_addr: "/ip4/192.168.0.157/tcp/34418",
    },
)
[src/main.rs:86] event = Some(
    IncomingConnection {
        local_addr: "/ip4/172.18.0.1/tcp/45709",
        send_back_addr: "/ip4/192.168.0.157/tcp/41348",
    },
)
[src/main.rs:86] event = Some(
    IncomingConnection {
        local_addr: "/ip4/172.28.0.1/tcp/45709",
        send_back_addr: "/ip4/192.168.0.157/tcp/51612",
    },
)
[src/main.rs:91] p = PeerId(
    "12D3KooWSqubw22QQeUif2AWeQHxS5Vjo4v5KojjfiyGRLpVFNhE",
)
[src/main.rs:95] m = "/ip4/192.168.0.157/tcp/45485"
[src/main.rs:91] p = PeerId(
    "12D3KooWSqubw22QQeUif2AWeQHxS5Vjo4v5KojjfiyGRLpVFNhE",
)
[src/main.rs:95] m = "/ip4/172.18.0.1/tcp/45485"
thread 'libp2p-swarm-task-0' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[src/main.rs:91] p = PeerId(
    "12D3KooWSqubw22QQeUif2AWeQHxS5Vjo4v5KojjfiyGRLpVFNhE",
)
[src/main.rs:95] m = "/ip4/127.0.0.1/tcp/45485"
[src/main.rs:91] p = PeerId(
    "12D3KooWSqubw22QQeUif2AWeQHxS5Vjo4v5KojjfiyGRLpVFNhE",
)
[src/main.rs:95] m = "/ip4/172.28.0.1/tcp/45485"
thread 'libp2p-swarm-task-2' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:26
thread 'libp2p-swarm-task-1' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:26
[src/main.rs:91] p = PeerId(
    "12D3KooWSqubw22QQeUif2AWeQHxS5Vjo4v5KojjfiyGRLpVFNhE",
)
[src/main.rs:95] m = "/ip4/172.17.0.1/tcp/45485"
thread 'libp2p-swarm-task-3' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', /home/mental/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.11.0/src/runtime/context.rs:18:26
@mental32
Copy link
Author

mental32 commented Sep 13, 2021

Note: I think this only applies to people who use a tokio transport and don't use the SwarmBuilder api directly.

@tomaka
Copy link
Member

tomaka commented Sep 13, 2021

Support for tokio was initially not there but was added later because of popular demand.
Indeed, if you're using tokio there's no choice but to pass an executor with SwarmBuilder::executor.

@mxinden
Copy link
Member

mxinden commented Sep 14, 2021

Great bug report @mental32. Very much appreciate the level of detail.

Ask the transport for an executor during SwarmBuilder::build so tokio transports can produce a well-configured executor and the default would just be the futures ThreadPool executor closure that is present now.

Off the top of my head having the transport provide the executor is a violation of concerns. I.e. I would not expect a transport to decide on the executor to be used. In addition, given that a transport could potentially wrap multiple other transports, choosing the right executor would not be trivial.

Document that tokio transports must set their own executor manually using SwarmBuilder

While not ideal, this would be my preference.

With the above in mind, what do you think @mental32? Do you have some time to send in a documentation patch?

@mental32
Copy link
Author

mental32 commented Sep 14, 2021

@tomaka @mxinden thanks for your insights!

TBH I think it'd be best if there was a section in the docs for Swarm / SwarmBuilder (and maybe tokio_development_transport?) that specifies that a manual executor would have to be set using the SwarmBuilder API, it's the easiest change and if people are already spending time in the docs or examples looking around for a way to use tokio they probably should run into it.

@mxinden I'll fork, add some doc changes, and PR something in soon-ish 👍

@thomaseizinger
Copy link
Member

thomaseizinger commented Sep 15, 2021

There is also a proposal in #2173 to introduce features on the libp2p-swarm crate to fix this.

@mxinden
Copy link
Member

mxinden commented Mar 27, 2022

We might want to consider not using a ThreadPool in SwarmBuilder::build:

rust-libp2p/swarm/src/lib.rs

Lines 1335 to 1348 in 6cc3b4e

// If no executor has been explicitly configured, try to set up a thread pool.
let pool_config =
self.pool_config.or_else_with_executor(|| {
match ThreadPoolBuilder::new()
.name_prefix("libp2p-swarm-task-")
.create()
{
Ok(tp) => Some(Box::new(move |f| tp.spawn_ok(f))),
Err(err) => {
log::warn!("Failed to create executor thread pool: {:?}", err);
None
}
}
});

We could instead use a LocalPool instead. While, obviously, this would hurt default performance, if I am not mistaken, it would prevent this new-comer foot gun, namely no tokio runtime running on the other threads of the ThreadPool.

@thomaseizinger
Copy link
Member

thomaseizinger commented Mar 27, 2022

I think I would prefer a type-safe builder pattern that forces you to choose between calling with_futures_executor(), with_tokio_executor() etc. (Those functions could/should be feature-flagged ). The documentation of those functions can then explain, what this executor is used for and when people should consider using the tokio for example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tomaka @thomaseizinger @mxinden @mental32