Skip to content

Commit

Permalink
Async get out
Browse files Browse the repository at this point in the history
Tokio - and Rust's async model in general - is pretty freaking cool, but
it isn't a perfect fit for everything. After hammering for a few days,
I'm pretty confident that it's not working out here:

- There's no way to enforce scoped async tasks without blocking the
  current thread.[1][2][3] This means that there's no async task
  equivalent to Rayon/Crossbeam-like scopes, and you're back to arcs and
  pins and all sorts of fun boilerplate if you'd like to foster
  parallelism with task::spawn().

- Traits and recursive calls need lots o' boxes, implemented by proc
  macros at best and by hand at worst.

- Since many FS syscalls block, tokio asyncifies them by wrapping each
  in a spawn_blocking(), which spawns a dedicated thread. Of course you
  can wrap chunks of synchronous file I/O in spawn_blocking() if kicking
  off a separate thread for each File::open() call doesn't sound fun,
  but that means you can't interact with async code anywhere inside...

- Add in the usual sprinkling of async/await/join/etc. throughout the
  code base - since anything that awaits needs to be a future itself,
  async code has a habit of bubbling up the stack.

None of these are dealbreakers by themselves, but they can add up to
real overheads. Not just cognitive, but performance too, especially if
you've already got a design with concurrent tasks that do a decent job
of saturating I/O and farming CPU-heavy work out to a thread pool.
< gestures around >

[1]: tokio-rs/tokio#1879
[2]: tokio-rs/tokio#2596
[3]: https://docs.rs/async-scoped/latest/async_scoped/struct.Scope.html#safety-3
  • Loading branch information
mrkline committed Dec 27, 2021
1 parent e10b947 commit c85e7bb
Show file tree
Hide file tree
Showing 23 changed files with 269 additions and 515 deletions.
205 changes: 2 additions & 203 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 0 additions & 7 deletions Cargo.toml
Expand Up @@ -14,10 +14,8 @@ debug = true
[dependencies]
atty = "0.2"
anyhow = "1.0"
async-trait = "0.1"
enum-map = "0.6"
fastcdc = "1.0"
futures = "0.3"
hex = "0.4"
hostname = "0.3"
lazy_static = "1.0"
Expand All @@ -43,11 +41,6 @@ features = ["serde"]
version = "0.10"
features = ["test", "termcolor"]

[dependencies.tokio]
version = "1.0"
default-features = false
features = ["rt-multi-thread", "macros", "sync", "fs", "io-util", "parking_lot" ]

[dependencies.zstd]
version = "0.5"
features = ["zstdmt"]
Expand Down

0 comments on commit c85e7bb

Please sign in to comment.