Cannot run even basic Tokio programs

[shepmaster]

#[tokio::main]
async fn main() {}

fails with this error (Playground): (Updated on 2022-07-06)

error: unsupported operation: can't call foreign function: epoll_create1
  --> /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.3/src/sys/unix/selector/epoll.rs:34:9
   |
34 |         syscall!(epoll_create1(flag)).map(|ep| Selector {
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't call foreign function: epoll_create1
   |
   = help: this is likely not a bug in the program; it indicates that the program performed an operation that the interpreter does not support
   = note: backtrace:
   = note: inside `mio::sys::unix::selector::epoll::Selector::new` at /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.3/src/sys/unix/mod.rs:7:28
   = note: inside `mio::poll::Poll::new` at /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.8.3/src/poll.rs:272:13
   = note: inside `tokio::io::driver::Driver::new` at /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/io/driver/mod.rs:117:20
   = note: inside `tokio::runtime::driver::create_io_stack` at /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/runtime/driver.rs:22:29
   = note: inside `tokio::runtime::driver::Driver::new` at /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/runtime/driver.rs:170:52
   = note: inside `tokio::runtime::Builder::build_threaded_runtime` at /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/runtime/builder.rs:753:39
   = note: inside `tokio::runtime::Builder::build` at /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.19.2/src/runtime/builder.rs:532:34
note: inside `main` at src/main.rs:1:1
  --> src/main.rs:1:1
   |
1  | #[tokio::main]
   | ^^^^^^^^^^^^^^
   = note: this error originates in the macro `syscall` (in Nightly builds, run with -Z macro-backtrace for more info)

epoll_create1(2)

[shepmaster]

I kind of feel like there's going to be an infinite amount of these, considering that any FFI function won't be supported. I don't have a good solution other than some crazy pluggable module system...

[RalfJung]

Does this try to spawn a thread later? We can stub out sched_affinity easily, but concurrency is not supported by Miri and supporting it is probably hard.

[shepmaster]

Does this try to spawn a thread later

I... literally have no idea ;-) This was just the first non-trivial thing I thought to test the playground's support of Miri + crates with.

My magical ideal world is the ability to run Miri on some random bit of code and know if I did the bad thing.

[oli-obk]

So, we got a little further. I guess it would be doable to start magically creating function pointers to "dynamically loaded libraries" (like we care what any code thinks about the platform they are running on).

error[E0080]: constant evaluation error: miri does not support dynamically loading libraries (requested symbol: epoll_create1)
  --> /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/sys/unix/dlsym.rs:43:11
   |
43 |     match libc::dlsym(libc::RTLD_DEFAULT, name.as_ptr() as *const _) as usize {
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ miri does not support dynamically loading libraries (requested symbol: epoll_create1)
   |
   = note: inside call to `mio::sys::unix::dlsym::fetch` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/sys/unix/dlsym.rs:30:33
   = note: inside call to `<mio::sys::unix::dlsym::DlSym<F>><unsafe extern "C" fn(i32) -> i32>::get` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/sys/unix/epoll.rs:38:19
   = note: inside call to `mio::sys::unix::epoll::Selector::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/poll.rs:655:23
   = note: inside call to `mio::poll::Poll::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.8/src/lib.rs:237:18
   = note: inside call to `tokio_reactor::Reactor::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/builder.rs:321:27
   = note: inside call to `tokio::runtime::threadpool::builder::Builder::build` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/mod.rs:142:9
   = note: inside call to `tokio::runtime::threadpool::Runtime::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/mod.rs:103:23
note: inside call to `tokio::runtime::threadpool::run::<futures::future::result_::FutureResult<(), ()>>` at src/main.rs:2:5
  --> src/main.rs:2:5
   |
2  |     tokio::run(futures::future::ok::<_, ()>(()))
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: inside call to `main` at /home/oliver/.rustup/toolchains/nightly-2019-01-25-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:64:34
   = note: inside call to closure at /home/oliver/.rustup/toolchains/nightly-2019-01-25-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:52:53
   = note: inside call to closure at /home/oliver/.rustup/toolchains/nightly-2019-01-25-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:297:40
   = note: inside call to `std::panicking::try::do_call::<[closure@DefId(1/1:1899 ~ std[6b76]::rt[0]::lang_start_internal[0]::{{closure}}[0]) 0:&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at /home/oliver/.rustup/toolchains/nightly-2019-01-25-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:293:5
   = note: inside call to `std::panicking::try::<i32, [closure@DefId(1/1:1899 ~ std[6b76]::rt[0]::lang_start_internal[0]::{{closure}}[0]) 0:&dyn std::ops::Fn() ->i32 + std::marker::Sync + std::panic::RefUnwindSafe]>` at /home/oliver/.rustup/toolchains/nightly-2019-01-25-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:388:9
   = note: inside call to `std::panic::catch_unwind::<[closure@DefId(1/1:1899 ~ std[6b76]::rt[0]::lang_start_internal[0]::{{closure}}[0]) 0:&dyn std::ops::Fn() ->i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>` at /home/oliver/.rustup/toolchains/nightly-2019-01-25-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:52:25
   = note: inside call to `std::rt::lang_start_internal` at /home/oliver/.rustup/toolchains/nightly-2019-01-25-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:64:5
   = note: inside call to `std::rt::lang_start::<()>`

[oli-obk]

Dynamically loading functions works now... but huh?

error[E0080]: constant evaluation error: attempted to do invalid arithmetic on pointers that would leak base addresses, e.g., comparing pointers into different allocations
  --> /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/sys/unix/dlsym.rs:32:16
   |
32 |             if self.addr.load(Ordering::SeqCst) == 1 {
   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ attempted to do invalid arithmetic on pointers that would leak base addresses, e.g., comparing pointers into different allocations
   |
   = note: inside call to `<mio::sys::unix::dlsym::DlSym<F>><unsafe extern "C" fn(i32) -> i32>::get` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/sys/unix/epoll.rs:38:19
   = note: inside call to `mio::sys::unix::epoll::Selector::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/poll.rs:655:23
   = note: inside call to `mio::poll::Poll::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.8/src/lib.rs:237:18
   = note: inside call to `tokio_reactor::Reactor::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/builder.rs:321:27
   = note: inside call to `tokio::runtime::threadpool::builder::Builder::build` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/mod.rs:142:9
   = note: inside call to `tokio::runtime::threadpool::Runtime::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/mod.rs:103:23

[RalfJung]

What is the type of addr?

[oli-obk]

I found the issue. Our pointer vs int comparison code gets the alignment of the allocation in question, and will not allow you to compare a function pointers 0-size 1-align pointer against a literal 1 because that could leak the base address of the allocation...

What is the type of addr?

AtomicUsize, but there's a pointer value, or 0 or 1 in it, no other values are possible

[oli-obk]

I added a hack to also allow 1 for now, will think about this in more detail later. Now I'm hitting a stacked borrows error (see code at https://github.com/carllerche/mio/blob/3eb6a80329a466e00272bf72c8a08d939f439cda/src/sys/unix/dlsym.rs#L35)

I even think the stacked borrows are right. I mean we are transmuting away the "celliness" of that field. So now we have an immutable reference that can just change under us. It won't of course, due to what the unsafe code promises, but still. I don't know why that function doesn't just return a copy of F, I mean it is a function pointer after all, and that is guaranteed Copy

error[E0080]: constant evaluation error: Location is not frozen long enough
  --> /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/sys/unix/dlsym.rs:35:17
   |
35 |                 mem::transmute::<&AtomicUsize, Option<&F>>(&self.addr)
   |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Location is not frozen long enough
   |
   = note: inside call to `<mio::sys::unix::dlsym::DlSym<F>><unsafe extern "C" fn(i32) -> i32>::get` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/sys/unix/epoll.rs:38:19
   = note: inside call to `mio::sys::unix::epoll::Selector::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.16/src/poll.rs:655:23
   = note: inside call to `mio::poll::Poll::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.8/src/lib.rs:237:18
   = note: inside call to `tokio_reactor::Reactor::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/builder.rs:321:27
   = note: inside call to `tokio::runtime::threadpool::builder::Builder::build` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/mod.rs:142:9
   = note: inside call to `tokio::runtime::threadpool::Runtime::new` at /home/oliver/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.14/src/runtime/threadpool/mod.rs:103:23
note: inside call to `tokio::runtime::threadpool::run::<futures::future::result_::FutureResult<(), ()>>` at src/main.rs:2:5

[RalfJung]

So Tokio makes the assumption that a function pointer will never be 1? Uh... that's probably fair on most platforms, but doesn't strike me as something we should put into the spec. I am not sure what the best way is for Miri to reflect such assumptions. This is directly related to being able to model "well-known addresses" as they might exist on embedded platforms.

[oli-obk]

You mean like we shouldn't even treat zero specially? 🤣

[RalfJung]

Well, that is not just a Miri thing. ;)

[RalfJung]

Current status: num_cpus works now and is tested on CI. Support for dlsym-shims is being added by #808. Once that lands, it seems worth checking where tokio is getting stuck these days.

I would be very surprised if this is not blocked by #811, but it would be a happy surprise. ;)

[RalfJung]

@shepmaster could you try again where your test fails in Miri these days? The stacktrace in the OP indicates num_cpus which we actually have on CI now, so it should get a little further at least.

[RalfJung]

Ah, I didn't know tokio is available on the playground. I updated the OP. The failing foreign function these days is epoll_create1. I expect this will basically require a way to talk with the host OS, I doubt we want to re-implement all these file descriptor operations.

[Kestrer]

By the way there's a pull request to fix this in Tokio now, so running the Tokio runtime without I/O or time support should be possible in Miri soon, hopefully.

[RalfJung]

That's pretty cool. :) Though... what does tokio even do then if there's no I/O or timers?^^ (Please forgive the naive question, I am not a tokio user.)

[Kestrer]

@RalfJung Tokio just provides nice wrappers around raw futures in general. One example is its attribute macros, it's much nicer to write:

#[tokio::test]
async fn test_something() {
    a().await;
    b().await;
}

than the manual version:

fn test_something() {
    futures_executor::block_on(async {
        a().await;
        b().await;
    });
}

It also supports spawning futures on the executor with tokio::spawn.

[Darksonn]

It is currently possible to run Tokio under miri with -Zmiri-disable-isolation. However you will need to create a runtime without IO enabled. For example, like this:

fn main() {
    let rt = tokio::runtime::Builder::new_current_thread()
        .build()
        .unwrap();
    
    rt.block_on(real_main());
}

async fn real_main() {
    ...
}

MIRIFLAGS="-Zmiri-disable-isolation" cargo +nightly miri run

[DebugSteven]

I'd like to work on this issue.

[RalfJung]

Good next steps for Tokio support would be:

• Complete socketpair support #3442
• Complete and fix our eventfd implementation #3445
• Implement basic epoll support #3448

I think that should bring us pretty close to closing this issue -- though I would say that supporting timers should also be done before we consider "basic" support to work. Async file system or network access is definitely beyond the scope of this issue though.