Build a dedicated async runtime

[huachaohuang]

I think Rust async runtime is significant for the long-term success of Engula. However, to be honest, I am not satisfied with the async ecosystem so far (you can check some links below). I know it will be better, but there is a lot of uncertainty here that I don't want to bet on an existing runtime.

On the other hand, I think a purpose-built async runtime for Engula is beneficial:

• We can control the scheduling algorithm for performance. Although I don't have some statistics yet, I don't think Tokio does well in IO related tasks according to my experience when benchmarking demo 1. It seems that Tokio simply spawns a thread pool to run synchronous file operations, instead of leveraging the actual asynchronous io from the underlying platforms. And no random access reads are supported, that really makes me sad. I can't imagine building a storage engine without that.
• We can add simulation tests from the ground up when necessary, which may be too hard to do in an existing runtime, especially when we get something specific to Engula.

IMO, building an async runtime with io-uring is the way we should go. We don't need to care too much about supporting different operating systems so far. There is a tokio-uring project, but it's still too young to be useful. Monoio looks more interesting in that term.

However, building our own runtime also means that we have to abandon the existing ecosystem. For example, things like Hyper/Tonic will not work. But I still think this is the right way to go. It may slow us down at the beginning, but it will be a long-term win. And it's the only way for Engula to be the top one. We still have the chance to do it at this early stage. Once we start adding a lot of features and get pressure from users, we can never get back.

Ref: #57 (comment)

References:

• Why asynchronous Rust doesn't work
• Building a shared vision for Async Rust

[huachaohuang]

We don't need to start from scratch actually. We can start with building our own runtime abstraction (e.g. engula-runtime). For example, we can simply re-export things from Tokio and replace the implementation when we get enough context for design.

[zojw]

PS: it seems the current existing rust io-uring framework is advise users to use the thread-per-core model.

• tokio-uring
• glommio
• or Monoio

they return future with !Send will great influence the codebase, for example: async_trait should change #[async_trait] to #[async_trait(?Send)], and some framework like tonic only support Send future..

maybe we only have the chance to decide whether or not to use "thread-per-core model"(TPC) in the early phase and could be hard to change after that....

and for engula, it has some different character from other project that maybe we need take care:

• some part of engula code should be a library used by other developers, so choose !Send maybe let other user don't use TPC model hard to use(although TPC is good model.
• some parts of engula is standalone deployed process, so we maybe can choose storage using TPC but kernel service using SEDA?

[huachaohuang]

@ihciah Your monoio looks very interesting. So let me venture to ask, are you interested in helping us with the async runtime?

[ihciah]

@ihciah Your monoio looks very interesting. So let me venture to ask, are you interested in helping us with the async runtime?

Thanks for mention out project! However, our runtime is currently at the very beginning stage, and we can only support HTTP in compactiable mode, so tonic and tower are not supported well now.

We will do these jobs(figure out how to go with tower and tonic) in a few months, and I will contact you at that time.

[youjiali1995]

seastar is the best async runtime in the world, I think, and its rust implementation is glommio. If there are different jobs or multiple pipeline stages in one process, you have to control the resource usage of each job or stage, especially in thread-per-core architecture, otherwise your process is likely to be out of control under heavy load. seastar and glommio provides the ability of control.

[youjiali1995]

Of course, both seastar and glommio support it. Sequential read/write has no obvious performance benefits on NVMe SSD comparing to rand read/write and cloud disk(i3 local disk and ebs) has very unique property -- its sequential performance is totally equal to rand performance.

[huachaohuang]

its sequential performance is totally equal to rand performance.

Oh, this sounds very interesting, can you provide more details about this?

[youjiali1995]

I concluded it on hackathon.

Each AWS i3 NVMe SSD can provide up to 180K IOPS or 700MB/s throughput. If you reach the IOPS limit, you reach the throughput limit too. It's very interesting. It means it's not necessary to batch a lot of data to write for the max throughput. Below are fio results of single NVMe SSD on i3.4xlarge.

# iodepth diff(performance of `-rw=write` is equal to `-rw=randwrite`, so I just recorded randwrite's.)
iodepth=1 bs=4kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=82.7MiB/s][w=21.2k IOPS][eta 00m:00s]

iodepth=1 bs=4kb write
Jobs: 1 (f=1): [W(1)][100.0%][w=80.6MiB/s][w=20.6k IOPS][eta 00m:00s]

iodepth=4 bs=4kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=277MiB/s][w=70.9k IOPS][eta 00m:00s]

iodepth=8 bs=4kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=423MiB/s][w=108k IOPS][eta 00m:00s]

iodepth=16 bs=4kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=592MiB/s][w=152k IOPS][eta 00m:00s]

iodepth=32 bs=4kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=698MiB/s][w=179k IOPS][eta 00m:00s]

iodepth=64 bs=4kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=705MiB/s][w=180k IOPS][eta 00m:00s]

# bs diff
iodepth=1 bs=4kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=82.7MiB/s][w=21.2k IOPS][eta 00m:00s]

iodepth=1 bs=8kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=149MiB/s][w=19.1k IOPS][eta 00m:00s]

iodepth=1 bs=16kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=250MiB/s][w=16.0k IOPS][eta 00m:00s]

iodepth=1 bs=32kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=413MiB/s][w=13.2k IOPS][eta 00m:00s]

iodepth=1 bs=64kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=572MiB/s][w=9149 IOPS][eta 00m:00s] 

iodepth=1 bs=128kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=661MiB/s][w=5285 IOPS][eta 00m:00s]

iodepth=1 bs=256kb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=685MiB/s][w=2739 IOPS][eta 00m:00s]

iodepth=1 bs=1mb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=762MiB/s][w=762 IOPS][eta 00m:00s]

iodepth=4 bs=1mb randwrite
Jobs: 1 (f=1): [w(1)][100.0%][w=759MiB/s][w=759 IOPS][eta 00m:00s]

iodepth=4 bs=1mb write
Jobs: 1 (f=1): [W(1)][100.0%][w=763MiB/s][w=763 IOPS][eta 00m:00s]

EBS's(I just tested gp3) property is different from local NVMe SSD. It has good throughput(up to 1000 MB/s) and bad IOPS(up to 16000). Its write latency is very high(iodepth=1 bs=4kb 1500 IOPS), but the latency doesn't increase a lot with bigger block size(iodepth=1 bs=512kb 300-400 IOPS), so it's very important to batch a lot of data to write.

[huachaohuang]

Hmm, when testing local SSDs, we need to consider the effects of garbage collection. When we start with a fresh disk, all internal blocks in SSD are free, so no garbage collection is necessary for both random writes and sequential writes. Once the disk is filled once, the next time we write, the controller needs to erase an existing block first before overwriting it. In this case, for a random write of 4KB, the controller needs to load the block (e.g. 256KB), overwrite the 4KB in the block, and then write the block back. So we are talking about a 64x write amplification here, which also means a shorter SSD lifetime.
I don't get the time to test that on those AWS instances yet. If you are interested, you can help to verify that :)

[youjiali1995]

Oops, I forget the impact of GC. Sequential writes should have some advantages. Don't know how disks are virtuallized and how FTL works on AWS. Need more tests.

[tisonkun]

I'm trying to test interoperation between sync and async blocks, and the conclusion could be that:

1. Interoperation between sync and async is possible. The major challenge is fighting with trait bounds.
2. Coexisting of async runtime is possible, but it takes more efforts to separating tasks cross the boundary.

Interoperation between sync and async

1. Call sync block from async block is nature and possible.
2. Call async block from sync block simply return a future that one of the sync block should submit to an executor for running (block_on).

#![feature(associated_type_bounds)]

use std::fmt::Debug;
use std::future::Future;

async fn another() -> u64 {
    21
}

fn function() -> impl Future<Output: Debug> {
    // `async fn` desugar to a state machine return a Future
    another()
}

fn main() -> anyhow::Result<()> {
    let rt = tokio::runtime::Builder::new_multi_thread().build()?;
    // you can call `.await` on any function as long as they return Future
    println!("{:?}", rt.block_on(async { function().await }));
    Ok(())
}

actually, #[tokio::main] is a sugar for:

tokio::runtime::Builder::new_current_thread()
    .enable_all()
    .build()
    .unwrap()
    .block_on(async {
        // code in `async fn main`
    })

Coexisting of async runtime

Because of the underneath above, it's possible that we run the storage engine in one async runtime, while when talking via gRPC by tonic, schedule the task on tokio's runtime.

However, it's very subtle to be aware of what tasks run on which runtime and highly possibly require to do bookkeeping on those runtimes.

#![feature(associated_type_bounds)]

use std::fmt::Debug;
use std::future::Future;

struct RuntimeBookKeeper {
    tokio: tokio::runtime::Runtime,
}

impl RuntimeBookKeeper {
    fn block_on_tokio<F, T>(&self, future: F) -> T where F: Future<Output = T> {
        self.tokio.block_on(future)
    }

    fn block_on_async<F, T>(&self, future: F) -> T where F: Future<Output = T> {
        async_std::task::block_on(future)
    }
}

async fn another() -> u64 {
    21
}

fn function() -> impl Future<Output: Debug> {
    another()
}

fn main() -> anyhow::Result<()> {
    let bk = RuntimeBookKeeper { tokio: tokio::runtime::Builder::new_multi_thread().build()? };
    println!("{:?}", bk.block_on_async(async {
        bk.block_on_tokio(async { function().await })
    }));
    Ok(())
}

If the dependency library relies on tokio heavily, said directly talk to the runtime of "current thread", it may fail if the current context doesn't provide the runtime.