Rocket hangs when spawning future on current runtime

[MikaelCall]

Description

I manually create a tokio runtime and launch my rocket. Then, when I spawn threads on that runtime, the future is not executed and rocket hangs (cannot even be killed via Ctrl+C, but SIGTERM works).

To Reproduce

Start a server using cargo run with non-working code example below and curl to trigger.

curl http://127.0.0.1:8000/spawn -X POST -H "Content-Type: application/json" -d "apa"

[package]
name = "spawn"
version = "0.1.0"
edition = "2018"

[dependencies]
crossbeam = "0.8"
# rocket = { version = "0.5.0-rc.1", features = ["json", "uuid"] }
rocket = { git = "https://github.com/SergioBenitez/Rocket", features = ["json", "uuid"] }
tokio = { version = "1.5", features = ["full"] }

use std::time::Duration;

use rocket::{Build, Rocket};

async fn tokio_main() {
    tokio::task::spawn_blocking(move || {
        let mut cnt = 0;
        loop {
            cnt += 1;
            println!("Spawn: {}", cnt);
            std::thread::sleep(Duration::from_secs(5));
            run_on_current_thread(cnt);
        }
    });

    let mut cnt: usize = 0;
    loop {
        cnt += 1;
        println!("Heartbeat: {}", cnt);
        tokio::time::sleep(Duration::from_secs(1)).await;
    }
}

// This panics if called outside the context of a Tokio runtime.
fn run_on_current_thread(id: usize) {
    let handle = tokio::runtime::Handle::try_current().unwrap();

    let (tx, rx) = crossbeam::channel::bounded(1);
    println!("-- run_on_current_thread -- ... spawn ...");
    handle.spawn(async move {
        let res = work(id).await;
        let _ = tx.send(res);
    });
    println!("-- run_on_current_thread -- ... spawned ...");
    let res = rx.recv().expect("recv gives message in blocking mode");
    println!("-- run_on_current_thread -- ... done!");
    res
}

async fn work(id: usize) {
    println!("Start working: {}", id);
    tokio::time::sleep(Duration::from_secs(3)).await;
    println!("Done working: {}", id);
}

// #[rocket::launch]
fn rocket() -> Rocket<Build> {
    rocket::build().mount("/", rocket::routes![spawn_endpoint])
}

#[rocket::post("/spawn", format = "application/json", data = "<text>")]
async fn spawn_endpoint(text: String) -> Result<String, ()> {
    run_on_current_thread(123);

    Ok(text.to_uppercase())
}

fn main() {
    tokio::runtime::Builder::new_current_thread() // new_multi_thread()
        .thread_name("rocket-worker-thread")
        .enable_all()
        .build()
        .expect("should be able to create tokio runtime")
        .block_on(rocket().launch())
        .expect("error launching rocket")
        // .block_on(tokio_main());
}

Expected Behavior

I expected to see "the same behavior as when running tokio_main", i.e. the async fn work() code should be executed and print to the screen.

Environment:

• OS Distribution and Kernel: Ubuntu 20.04
• Rocket Version: [e.g. 0.5.0-rc.1 and master@5a2535f8]

Additional Context

• Using new_multi_thread does not work either
• Using #[rocket::launch] instead of main does not work either
• tokio_main() works as expected
• The program does not panic (when calling 'tokio::runtime::Handle::try_current()`) so the runtime should have been setup correctly IMHO

[jebrosen]

run_on_current_thread is a blocking function (specifically, the let res = rx.recv()). In your first example that you say is working, you do call this function via a call to tokio::spawn_blocking. Similarly, in the async fn spawn_endpoint you should call run_on_current_thread inside a tokio::spawn_blocking call:

#[rocket::post("/spawn", format = "application/json", data = "<text>")]
async fn spawn_endpoint(text: String) -> Result<String, ()> {
    tokio::task::spawn_blocking(|| run_on_current_thread(123)).await;

    Ok(text.to_uppercase())
}

This behavior is not particular to Rocket, either: removing the spawn_blocking call in the tokio_main version prevents any work from being done, and the Heartbeat is never even reached.

[MikaelCall]

Thank you @jebrosen for solving my issue and clarifying that this is rather a async/tokio understanding issue. Using spawn_blocking in the endpoint does indeed work. However, removing the spawn_blocking call in the tokio_main version will reach the Heartbeat and display it each second (why did you expect that it wouldn't be executed?).

I have a follow-up (tokio) question based on your answer:

Where and why does my spawned future "disapear", i.e. the code in handle.spawn(async move { .. }); or AFAICT equivalently, tokio::task::spawn(async move { .. });. I was expecting that the async block would be schedueled and executed on the async tokio runtime but that does not seem to happen. I've tried with 1 and 2 threads in worker_threads when creating the runtime.

[SergioBenitez]

I'm closing this as the core issue has been solved. Feel free to continue commenting.

[jebrosen]

I think this has been more completely answered by Alice on the Tokio discord, but I'll leave this here for anyone else stumbling upon it:

However, removing the spawn_blocking call in the tokio_main version will reach the Heartbeat and display it each second (why did you expect that it wouldn't be executed?).

I removed the spawn_blocking wrapping from tokio_main and instead called run_on_current_thread directly, and the heartbeat was never reached. Instead I see the messages spawned, spawn, and then a deadlock at the recv() call. Did you try it with the single_thread runtime? (That makes a huge difference for this particular example. If you use a multi_thread runtime, you might not observe this particular deadlock right away or at all.)

Where and why does my spawned future "disapear", i.e. the code in handle.spawn(async move { .. }); or AFAICT equivalently, tokio::task::spawn(async move { .. });. I was expecting that the async block would be schedueled and executed on the async tokio runtime but that does not seem to happen.

After calling spawn, the task has indeed been scheduled on the tokio runtime. But, Rust futures are a form of cooperative multitasking, which means that they can only switch between other tasks at an await point. In your example you used a single thread runtime, and that single thread will be stuck at the recv() call forever. spawn_blocking is designed for running blocking code in these kinds of situations: it runs the function, including the recv(), on a separate thread that is not responsible for executing async code.

So, without spawn_blocking you end up with something like this:

single thread runtime: rocket or tokio_main -> ... -> run_on_current_thread() -> spawn work() -> recv(), wait for work() to finish -> (oops, deadlock here. work() has not even started yet) -> await something -> (tokio runs other spawned tasks, including work())

And with spawn_blocking:

single thread runtime: rocket or tokio_main -> ... -> spawn_blocking run_on_current_thread() -> await the spawn_blocking task -> (tokio runs other spawned tasks, including work()) -> (the spawn_blocking call completes at some point) -> resume from awaiting the spawn_blocking task

spawn_blocking background thread: (idle) -> start running run_on_current_thread() -> spawn work() -> recv(), wait for work() to finish -> return res -> (idle)

I've tried with 1 and 2 threads in worker_threads when creating the runtime.

Tuning the number of worker threads may happen to work in some examples, but it is not a reliable solution for this kind of problem. Suppose you had 8 worker threads, and 8 simultaneous users accessed the spawn_endpoint(): if each worker thread is running a blocking recv() call, those workers cannot switch over to the actual work they are waiting on and you get the same deadlock. So, changing the number of worker threads just changes the conditions needed to run into the problem.