Folly to Coroutine Migration

HomeBlocks I/O: Folly futures → the v8 coroutine stack

HomeBlocks (and the HomeStore layer under it) removed Folly futures and moved to the C++23 stdexec / sisl::async coroutine stack — the "v8" stack — in commit 5ac2b2c "Remove Folly and redesign the public API onto the v8 coroutine stack."

This page is for engineers who already know the Folly future architecture. The I/O does the same thing; the substrate changed. We focus on the two differences that actually affect how you write and reason about code:

The heap / allocation model — what allocates, where, and what owns what.
The threading / affinity model — which thread you're on, and when that changes under you.

Everything else (the API got reshaped from VolumeManager member methods into free functions, names changed) is cosmetic and lives in a footnote at the bottom.

TL;DR — the mental-model swaps

Folly	v8	The catch
`folly::Future<Result<T>>`	`async_result<T>` = `sisl::async::task<result<T>>`	Lazy. Does nothing until you `co_await` / `sync_get` it. `[[nodiscard]]`.
`.thenValue()` / `.thenError()` chains	linear `co_await` in a coroutine	Reads synchronous; each `co_await` is a yield point.
`folly::Promise` / `SemiFuture`	`sisl::async::value_awaitable<T>`	Cross-thread single-shot cell; must outlive the await.
`folly::collectAll(Unsafe)(futs)`	`co_await sisl::async::when_all(std::move(futs))`	Tasks are lazy — they don't run until `when_all` awaits them.
`folly::Executor` / `.via(InlineExecutor)`	gone	Resumes inline on whatever thread completes the await. Cross-reactor scheduling is now explicit (`run_on_forget`).
`Result<T> = std::expected<T, E>`	`result<T> = std::expected<T, std::error_condition>`	One error surface across HomeStore + HomeBlocks; `status` / `async_status` / `ok()`.

result<T>, async_result<T>, status, async_status, ok() all come straight from <homestore/error.hpp> — HomeBlocks and HomeStore now speak one error vocabulary.

Pillar 1 — Allocation & ownership

The request object: heap + refcount → coroutine frame

Before — every I/O heap-allocated a refcounted request:

struct vol_interface_req : public sisl::ObjLifeCounter< vol_interface_req > {
    uint8_t* buffer{nullptr};
    lba_t lba; lba_count_t nlbas;
    sisl::atomic_counter< int > refcount;   // atomic refcount churn per IO
    bool part_of_batch{false};
    VolumePtr vol{nullptr};                 // holds a ref to the volume
    friend void intrusive_ptr_add_ref(vol_interface_req*);
    friend void intrusive_ptr_release(vol_interface_req*);
    virtual void free_yourself() { delete this; }
};
using vol_interface_req_ptr = boost::intrusive_ptr< vol_interface_req >;

After — the request is a plain struct that lives on the coroutine frame; no heap alloc, no refcount:

struct io_req {
    uint8_t* buffer{nullptr};
    lba_t lba{0}; lba_count_t nlbas{0};
    // ... timing fields ...
};

// In-flight tracking is now an RAII scope guard, not an intrusive refcount:
struct vol_io_guard {
    explicit vol_io_guard(volume_handle const& v) : vol(v) { vol->inc_ref(); }
    ~vol_io_guard() { vol->dec_ref(); }
    volume_handle vol;
};

io_req is constructed in the async_write / async_read coroutine and lives in that frame for the whole operation — its address is stable across every co_await. The volume's in-flight count is a vol_io_guard that releases when the coroutine frame unwinds.

The new unit of allocation is the coroutine frame

A future-chain allocated a folly Future Core per future and per .thenValue. A coroutine allocates one frame per coroutine in the chain (async_write → volume::write → …), taken at the call/suspension boundary. So the request object didn't disappear — it got folded into the frame allocation that the coroutine needs anyway, and the per-.thenValue Core allocations are gone.

Laziness forces you to heap-own what used to be stack-local

This is the subtle one. Because v8 tasks are lazy (they don't start until awaited), anything a not-yet-running task references by pointer/reference must outlive the await — a temp scope will dangle.

The read fan-out is the canonical example. Before, the per-sub-read sg_list was a stack local — fine, because the folly future captured/ran synchronously enough:

sisl::sg_list sgs;                                   // stack-local
sgs.iovs.emplace_back(iovec{.iov_base = read_buf, .iov_len = sgs.size});
futs.emplace_back(rd()->async_read(blkids, sgs, sgs.size, req->part_of_batch));

After, that sg_list must be heap-owned and kept alive until when_all actually runs the tasks:

auto sgs = std::make_unique< sisl::sg_list >();      // heap
sgs->iovs.emplace_back(iovec{.iov_base = read_buf, .iov_len = sgs->size});
futs.emplace_back(rd()->async_read(blkids, *sgs, sgs->size, nullptr));
sgs_keepalive.emplace_back(std::move(sgs));          // owned in the caller's frame

Rule: anything referenced across a co_await must live on the coroutine frame or the heap — never a temp scope. This lifetime discipline is what replaced refcounted request objects.

The cross-thread result cell

repl_result_ctx<T> (the journal-commit handoff) is still heap-allocated via intrusive_ptr — but it now embeds a value_awaitable instead of a folly::Promise:

// Before
struct repl_result_ctx : public vol_repl_ctx {
    folly::Promise< T > promise_;
    folly::SemiFuture< T > result() { return promise_.getSemiFuture(); }
};

// After
struct repl_result_ctx : public vol_repl_ctx {
    sisl::async::value_awaitable< T > promise_;   // co_await it directly; no separate folly Core
};

It stays on the heap because it must outlive the await and be completed from another thread (stable address). value_awaitable has no separate shared-state allocation the way folly::Promise/SemiFuture did.

(detail::detach and when_all allocate too — a wrapper frame + a start_detached op-state, and storage for the N tasks/results, respectively.)

Pillar 2 — Threading & affinity

Folly mediated threading through executors; v8 resumes inline on the completing thread

Folly continuations ran where the executor said (.via(&folly::InlineExecutor::instance()).thenValue(...) ran inline; .get() blocked). The executor was the dial.

There is no executor in v8. sisl::async::task is sticky-affinity: when an awaited operation completes, the coroutine resumes inline on whatever thread completed it. So:

Which thread you are on after a co_await depends on who completed the awaited work.

A single write starts on one thread and finishes on another

Trace volume::write:

It starts on the issuing reactor.
co_await rd()->async_write(...) (the data write) is an io_uring round-trip issued on that reactor; it completes back on that same reactor — you stay home.
write_to_index(...) is synchronous — same thread, no await.
co_await req->promise_ (the journal commit) is completed by HomeBlocksImpl::on_write, which HomeStore calls on the commit thread after the group-commit flush. So the coroutine resumes — and finishes — on the commit thread, not the issuing reactor.

sequenceDiagram
    participant R as Issuing reactor
    participant V as volume::write (coroutine)
    participant D as Data service (io_uring)
    participant J as Journal / CP
    participant C as Commit thread (on_write)
    R->>V: co_await async_write(...)
    V->>D: co_await rd()->async_write (data)
    D-->>V: completes on the issuing reactor (stays home)
    V->>V: write_to_index(...) — synchronous, same thread, no await
    V->>J: async_write_journal(...), then co_await req->promise_
    Note over V: suspends — releases the reactor
    J->>C: group-commit flush → on_write(lsn, ...)
    C-->>V: promise_.complete(ok())  →  RESUME
    Note over V,C: the rest of volume::write now runs on the COMMIT thread
    V-->>R: co_return size

The producer side is a one-liner on the commit thread:

// HomeBlocksImpl::on_write(...) — NOT a coroutine; runs on the commit thread
if (repl_ctx) { repl_ctx->promise_.complete(ok()); }   // was: promise_.setValue(NullResult())

What this means for you

Don't assume "still on my reactor" after a co_await. The thread can change. (co_await is a suspension point; the only code that runs atomically on one thread is the stretch between two awaits. "Non-blocking reactor" ≠ "atomic operation.")
sync_get blocks the calling thread, and is only safe OFF a reactor. Blocking a reactor parks its iomgr loop, and the CP / I/O the task is waiting on runs on reactors → deadlock. On a reactor you co_await; on a test main or control thread you detail::sync_get.
To run work on a reactor from a foreign thread, hop explicitly with iomanager.run_on_forget(iomgr::reactor_regex::random_worker, ...).

⚠️ War story. The HomeBlocks ublk adapter originally co_awaited HomeStore I/O inline on a foreign (non-reactor) thread under deep concurrency. Result: index b-tree corruption and multi-second lost-wakeup stalls. The fix was to run_on_forget each I/O onto a reactor. HomeStore is reactor-affine — the data service's per-reactor io_uring, the shared index write-back cache, and the checkpoint's dirty-state tracking all assume reactor context. Drive HomeStore I/O on reactors. The cross-thread commit hop above is intrinsic to group-commit journaling + replication, not an artifact of the coroutine mechanism.

The ublk adapter handoff: where it hops off and back on

The adapter straddles two threads, each owning a ring: the ublk queue's SINGLE_ISSUER io_uring (only the queue thread may touch it or call ublksrv_complete_io) and HomeBlocks' iomgr reactors (where HomeStore I/O must run). A lone co_await would resume on a reactor and illegally drive the queue ring, so the adapter hops deliberately — off the queue thread to run the I/O, then back onto the queue ring, through the kernel, to complete it.

sequenceDiagram
    participant Q as ublk queue thread<br/>(SINGLE_ISSUER io_uring)
    participant R as HomeBlocks reactors<br/>(iomgr)

    Note over Q: async_iov builds the per-IO cqe_state
    Q->>R: run_on_forget starts run_hb_io<br/>on a worker reactor — HOP OFF
    Note over Q: co_await *state<br/>per-IO coroutine suspends on the queue thread
    Note over R: co_await async_write(...) drives HomeStore,<br/>resumes on the commit reactor when durable
    R->>Q: post_msg_ring posts an IORING_OP_MSG_RING CQE<br/>onto the queue ring — HOP ON
    Note over Q: run_queue_loop reaps the CQE, resumes the<br/>cqe_state, calls ublksrv_complete_io

The hop back is the lock-free part: the kernel delivers the completion straight onto the queue's ring as an ordinary CQE, so the per-IO cqe_state is only ever touched on the queue thread — no eventfd, no service loop, no mutex.

Performance: is the commit-thread hop a regression?

The "finishes on the commit thread" hop is the most striking behavioral statement on this page, so it deserves a direct answer: no, the lift did not introduce it. Folly resumed on the commit thread too. What changed is that the hop is now explicit (a co_await) instead of hidden inside an executor adapter.

Bottom line: the threading affinity is identical before and after; per-hop dispatch is if anything cheaper now. The real thing to scrutinize is a pre-existing property — the commit thread as a shared serialization point — which the coroutine syntax makes easier to overload. Measure it; don't assume it.

The affinity was always the commit thread

The before-code's write tail bound the whole chain to Folly's InlineExecutor:

return req->result()
    .via(&folly::InlineExecutor::instance())   // run the continuation on whoever fulfills the promise
    .thenValue([=](auto&& r) { /* journal result + metrics */ });

req->result()'s promise is fulfilled by on_write (promise_.setValue(...)) on the commit thread, and InlineExecutor means "resume on the fulfilling thread" — so that .thenValue ran on the commit thread. Because the whole future was InlineExecutor-bound, the consumer's .thenValue landed there too. The Folly-era ublk adapter proves it: it .thenValue'd the write and then called ublksrv_queue_send_event(q) to bounce back to its queue thread — i.e. it also woke on the commit thread and immediately hopped off, the same cross-thread shape the new msg_ring bridge has, except msg_ring lets the kernel post the completion CQE straight onto the queue's ring (IORING_OP_MSG_RING) instead of kicking an eventfd into a service loop. v8's co_await req->promise_ (sticky-affinity: resume inline on the completing thread) is the exact same behavior, just written linearly.

Per-hop cost went down, not up

A Folly .thenValue allocated a Core, dispatched through a std::function, and did atomic refcounting on the shared state. A coroutine resume is a direct jump into an already-allocated frame. So the commit-thread hop is cheaper in v8.

What actually deserves scrutiny (and why)

These are real — but they are pre-existing concerns the migration spotlights, not regressions it caused:

The commit thread is a shared serialization point. Everything downstream of a write's durability — the write's own metric tail and the consumer's continuation — runs on it. Under high write IOPS, if that thread is single (or few) and the post-commit work is non-trivial, it can bottleneck write completion. This was true under Folly too.
The ergonomic trap (new). Linear coroutine code reads synchronous, so it's tempting to pile logic "after the co_await" that you'd have been more deliberate about in a .thenValue lambda — and all of it runs on the commit thread. Keep the post-commit tail tiny and hop off the commit thread immediately for any real work. The ublk adapter's msg_ring bridge is the model: the only thing that runs on the commit thread is one post_msg_ring (a single IORING_OP_MSG_RING SQE); the kernel delivers the completion CQE onto the queue's own ring, where the real work — resuming the per-IO coroutine and calling ublksrv_complete_io — runs on the queue thread.
Per-write allocation (see Pillar 1): coroutine frames per write vs Folly Cores per future. Could go either way depending on frame count and HALO elision — worth a malloc-pressure check.

How to actually measure it

Don't trust the static analysis above as the verdict — both tags exist:

Same workload, 5ac2b2c^ (Folly) vs 5ac2b2c (v8): compare write p50/p99 latency and sustained throughput.
Profile the commit thread's CPU occupancy under sustained writes — is it saturated?
Compare per-op allocation counts.

If v8 comes out slower, the likely culprits are allocation pressure or a fattened post-commit tail — not the hop itself — and both are fixable without touching the threading model.

The hot path, annotated (before → after)

Write — `volume::write`

Before (nested .thenValue, alloc result truthy = error):

auto result = rd()->alloc_blks(data_size, hints, new_blkids);
if (result) { return std::unexpected(VolumeError::NO_SPACE_LEFT); }   // truthy == error
return rd()->async_write(new_blkids, data_sgs, vol_req->part_of_batch)
    .thenValue([=](auto&& res) -> NullAsyncResult {
        if (res) { return std::unexpected(VolumeError::DRIVE_WRITE_ERROR); }
        // ... write_to_index ... async_write_journal ...
        return req->result()
            .via(&folly::InlineExecutor::instance())
            .thenValue([=](auto&& r) -> std::expected<void, VolumeError> { /* journal result */ });
    });

After (linear co_await, alloc result falsy = error):

async_status volume::write(io_req& vol_req) {                        // ← vol_req lives on THIS frame
    if (auto const alloc_res = rd()->alloc_blks(data_size, hints, new_blkids); !alloc_res)  // ← flip: !alloc == error
        co_return std::unexpected(std::errc::no_space_on_device);

    if (auto const wr = co_await rd()->async_write(new_blkids, data_sgs, nullptr); !wr)   // ← data; completes on issuing reactor
        co_return std::unexpected(std::errc::io_error);

    // write_to_index(...) — synchronous, same thread

    auto req = repl_result_ctx< status >::make(...);                 // ← heap (intrusive_ptr): outlives the await
    rd()->async_write_journal(new_blkids, req->cheader_buf(), req->ckey_buf(), data_size, req);
    auto const jres = co_await req->promise_;                        // ← RESUMES ON THE COMMIT THREAD
    if (!jres.has_value()) co_return std::unexpected(jres.error());
    co_return homeblocks::ok();
}

Note the two batch-arg differences: vol_req->part_of_batch → nullptr (batching is gone, see below), and the if(result) → if(!result) flip for HomeStore's std::expected-returning calls.

Read — `volume::read`

Before (collectAllUnsafe, futures of error_code, stack-local sg_list):

std::vector< folly::Future< std::error_code > > futs;
submit_read_to_backend(blks_to_read, req, futs);          // sg_list stack-local inside
return folly::collectAllUnsafe(futs).thenValue([=](auto&& vf) -> NullResult {
    for (auto const& err_c : vf) if (err_c.value()) return std::unexpected(to_volume_error(err_c.value()));
    return verify_checksum(read_ctx);
});

After (when_all, tasks of iomgr::io_result, heap sg_list kept alive):

std::vector< sisl::async::task< iomgr::io_result > > futs;
std::vector< std::unique_ptr< sisl::sg_list > >      sgs_keepalive;   // ← heap-own; lazy tasks run later
submit_read_to_backend(blks_to_read, req, futs, sgs_keepalive);

auto const results = co_await sisl::async::when_all(std::move(futs)); // ← fan-in; starts the lazy tasks
for (auto const& r : results)
    if (sisl_unlikely(!r.has_value())) co_return std::unexpected(r.error());
co_return verify_checksum(read_ctx);

Other differences worth internalizing

Lazy tasks. An async_result<T> you don't co_await / sync_get never issues the I/O. ([[nodiscard]] helps, but a dropped task is silent.) Folly futures were already running.
co_await is a yield barrier. Only the code between two awaits is atomic on one thread. Hold no invariant across a co_await that another coroutine on the reactor could violate while you're suspended.
Batching is gone. bool part_of_batch + virtual void submit_io_batch() were removed from the public API. Pass nullptr for the batch arg; fan independent ops out with when_all. (HomeStore's v8 io_batch is a reactor-local RAII scope from data_service().begin_batch(), not the old cross-call accumulator.)
Fire-and-forget needs detail::detach. Because tasks are lazy, dropping one does nothing. To start a task whose result you don't need (e.g. async_free_blks from a non-coroutine on_write):
```
detail::detach(vol_ptr->rd()->async_free_blks(lsn, old_blkid));
```

Cookbook

// On a reactor / inside a coroutine — co_await:
auto r = co_await homeblocks::async_write(vol, addr, sgs);

// Off a reactor (test main, control plane) — block to completion:
auto r = homeblocks::detail::sync_get(homeblocks::async_write(vol, addr, sgs));

// Fan out N independent ops and wait for all:
auto results = co_await sisl::async::when_all(std::move(futs));

// Fire-and-forget a lazy task:
homeblocks::detail::detach(some_async_op());

// Produce a result for a consumer on another thread:
//   consumer:  auto v = co_await ctx->promise_;     (ctx is heap, outlives the await)
//   producer:  ctx->promise_.complete(value);       (called from the completing thread)

// Hop a unit of work onto a reactor from a foreign thread:
iomanager.run_on_forget(iomgr::reactor_regex::random_worker, [=] { detail::detach(do_io(...)); });

Footnote — the cosmetic reshape

Orthogonal to the substance above, the public API was also reshaped (mention only): block I/O moved from VolumeManager member methods (vol_mgr->write(vol, req)) to free functions (async_write(vol, addr, sgs)); HomeBlocks / VolumeManager → home_blocks; consumers hold an opaque volume_handle; bring-up is init_homeblocks(home_blocks_config); I/O is byte-addressed (addr/len) instead of taking vol_interface_req; and the whole public surface is one header, <homeblks/home_blocks.hpp>. None of this changes the runtime model — it's surface.

Where it lives

Piece	File
Public API (one header)	`src/include/homeblks/home_blocks.hpp`
Bridging helpers (`sync_get`, `detach`)	`src/lib/coro_helpers.hpp`
Free-function I/O + `on_write` producer	`src/lib/volume_mgr.cpp`
`volume::write` / `volume::read`	`src/lib/volume/volume.cpp`
`repl_result_ctx` (value_awaitable)	`src/lib/volume/volume.hpp`
`io_req`	`src/lib/volume/io_req.hpp`

Migration commit: 5ac2b2c on dev/v6.x. For any snippet here, the after is the current file and the before is git show 5ac2b2c^:<path>.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Folly to Coroutine Migration

HomeBlocks I/O: Folly futures → the v8 coroutine stack

TL;DR — the mental-model swaps

Pillar 1 — Allocation & ownership

The request object: heap + refcount → coroutine frame

The new unit of allocation is the coroutine frame

Laziness forces you to heap-own what used to be stack-local

The cross-thread result cell

Pillar 2 — Threading & affinity

Folly mediated threading through executors; v8 resumes inline on the completing thread

A single write starts on one thread and finishes on another

What this means for you

The ublk adapter handoff: where it hops off and back on

Performance: is the commit-thread hop a regression?

The affinity was always the commit thread

Per-hop cost went down, not up

What actually deserves scrutiny (and why)

How to actually measure it

The hot path, annotated (before → after)

Write — `volume::write`

Read — `volume::read`

Other differences worth internalizing

Cookbook

Footnote — the cosmetic reshape

Where it lives

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HomeBlocks

CRAFT

Clone this wiki locally

Uh oh!

Folly to Coroutine Migration

HomeBlocks I/O: Folly futures → the v8 coroutine stack

TL;DR — the mental-model swaps

Pillar 1 — Allocation & ownership

The request object: heap + refcount → coroutine frame

The new unit of allocation is the coroutine frame

Laziness forces you to heap-own what used to be stack-local

The cross-thread result cell

Pillar 2 — Threading & affinity

Folly mediated threading through executors; v8 resumes inline on the completing thread

A single write starts on one thread and finishes on another

What this means for you

The ublk adapter handoff: where it hops off and back on

Performance: is the commit-thread hop a regression?

The affinity was always the commit thread

Per-hop cost went down, not up

What actually deserves scrutiny (and why)

How to actually measure it

The hot path, annotated (before → after)

Write — volume::write

Read — volume::read

Other differences worth internalizing

Cookbook

Footnote — the cosmetic reshape

Where it lives

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HomeBlocks

CRAFT

Clone this wiki locally

Write — `volume::write`

Read — `volume::read`