Skip to content

V8_MIGRATION_PLAN

Jump to bottom
Brian Szmyd edited this page Jun 9, 2026 · 1 revision

Lifting homeobject onto HomeStore v8 (off Folly) — migration plan

Hints captured right after doing the same lift for homeblocks (reference implementation: homeblocks commit c43fa2d, "Remove Folly and redesign the public API onto the v8 coroutine stack"). homeobject is the same Folly baseline, so most of it is mechanical reuse — but it has three areas homeblocks didn't, called out under The hard parts.

Baseline → target

now target
homestore ^7.5.2 (folly futures) ^8 (sisl::async coroutines / std::expected)
sisl ^13.2 ^14.6
iomgr transitive (also #include <iomgr/iomgr.hpp> in gc_manager) ^13, make it an explicit requires
nuraft_mesg transitive via homestore ^5 transitive (peer_id_t/replica_id_t/group_id_t come from here; expect renames)
C++ std 20 23 (coroutines)
futures Folly gone

Build all four (sisl/iomgr/nuraft_mesg/homestore) editable from ~/dev/oss, same as the homeblocks lift.

The reusable playbook (mechanical — identical to homeblocks)

Apply these 1:1. The full type/API dictionary lives in the homeblocks memory note folly-to-v8-migration-dict; the short version:

Folly (v7) v8 replacement
Manager<E>::AsyncResult<T> = folly::SemiFuture<folly::Expected<T,E>> task<std::expected<T,E>> (keep the error type — see hard-part #4)
folly::Future<X> / folly::SemiFuture<X> sisl::async::task<X>
folly::Promise<T> (in repl_result_ctx<T>) sisl::async::value_awaitable<T> — .complete() on the commit thread, co_await in the issuing coroutine
folly::collectAllUnsafe(vec) / collectAll sisl::async::when_all(std::vector<task<…>>)
folly::makeFuture/makeSemiFuture(x) co_return x (or a ready task)
folly::makeUnexpected(e) co_return std::unexpected(e)
folly::Unit / NullResult std::monostate / a status-style alias
.get() (tests, control plane) an internal detail::sync_get(...) — off-reactor only
folly::Init delete (sisl options/logging init already present)
folly::InlineExecutor / .via(...) delete — coroutines need no executor; the start/detach shim uses exec::inline_scheduler
folly::ConcurrentHashMap pick one: std::unordered_map + std::shared_mutex (what homeblocks did), or a sisl concurrent map. The mem-backend shard/blob maps + the GC blob index use it.
folly::MPMCQueue (GC) a sisl/std concurrent queue — see hard-part #3
folly::Uri (endpoint parse) boost::urls or a 5-line hand-roll
cast macros r_cast/s_cast/uintptr_cast/... (sisl v14 removed them) real casts; bare Clock → sisl::Clock; MetricsGroupWrapper → MetricsGroup; ReportFormat::kTextFormat → TEXT_FORMAT

homestore v8 renames you'll hit everywhere: ReplDev→repl_dev, ReplDevListener→repl_dev_listener, ReplApplication→repl_application, BlkId→blk_id, MultiBlkId→multi_blk_id, AsyncReplResult→async_status, ReplResult→result, headers .h→.hpp. And the truthiness flip: alloc_blks(...) now returns status where a value means success — invert the old if (result) = error checks.

Keep an internal coro_helpers.hpp (sync_get / detach) like homeblocks — do not publish it.

The hard parts (homeobject-specific — design, not sed)

1. Real multi-member raft listener — replication_state_machine.{hpp,cpp} (critical path)

homeblocks used a solo repl_dev; its listener was a stub. homeobject's ReplicationStateMachine is a full repl_dev_listener: on_commit / on_pre_commit / on_rollback / on_error / on_fetch_data / on_snapshot / create_snapshot / on_start_replace_member / on_remove_member / notify_committed_lsn / .... The v8 repl_dev_listener interface changed (snake_case + signature churn) — re-derive every override against the v8 header, don't assume. The async ones:

  • on_fetch_data(...) → folly::Future<std::error_code> → the v8 task type.
  • create_snapshot(...) → AsyncReplResult<> → async_status (co_return ok()).
  • The repl_result_ctx<T>::promise_ (folly::Promise) → value_awaitable — this is exactly the homeblocks on_write / repl_result_ctx pattern (commit thread calls .complete(ok()), the async_alloc_write coroutine co_awaits it). Lift it verbatim.

This blocks all PG/shard/blob ops — do it first after the public API.

2. Snapshot / baseline-resync — none of this existed in homeblocks

create_snapshot / read_snapshot_obj / write_snapshot_obj + SnapshotReceiveHandler + pg_blob_iterator return Folly futures / AsyncReplResult and walk the blob index. Convert to task; the iterator becomes a coroutine. Budget real time here — it's the second-heaviest area after the managers.

3. GC manager — Folly executors in the work path (the trickiest non-listener piece)

gc_manager runs on two folly::IOThreadPoolExecutor pools (normal + emergent) + a folly::MPMCQueue + collectAllUnsafe over per-chunk tasks. This is executor use in the actual work path (homeblocks's vol_gc was a single reactor timer — nothing like it). Plan:

  • Pools → iomgr reactors (run_on / run_on_forget on worker reactors) or a sisl thread pool; per-chunk fan-out → when_all.
  • MPMCQueue → a sisl/std concurrent queue.
  • Deadlock watch: GC calls blocking-ish control-plane ops (data_service alloc/read/free). If GC coroutines run on reactors and sync_get blocking work that itself needs reactors, you hit the sync_get-on-reactor deadlock (homeblocks memory sync-get-on-reactor-deadlock). Either co_await, or keep the blocking waits off reactors.

4. Error type — keep {code, current_leader}; do NOT flatten to std::error_condition

homeblocks adopted homestore's std::error_condition because VolumeError had no payload. homeobject's ShardError/BlobError carry std::optional<peer_id_t> current_leader (NOT_LEADER client redirect) — that must survive. Recommended shape:

  • Manager surface stays its own type: Result<T> = std::expected<T, ShardError> (resp. BlobError, PGError), AsyncResult<T> = sisl::async::task<Result<T>>.
  • At the homestore boundary, co_await homestore's task<result<T>> (error = std::error_condition) and translate to ShardError/BlobError, filling current_leader from repl_dev->get_leader_id().
  • So only the coroutine type (SemiFuture→task) and the std::expected substrate change; homeobject's public error stays richer than homestore's. This is the main place the homeblocks recipe does not apply.

5. HTTP admin — Pistache → sisl httplib

hs_http_manager (trigger_gc / snapshot / membership / metrics) is on Pistache; homeblocks's HTTP was small iomgr→sisl httplib. Port handlers to sisl's http_server (httplib) as homeblocks did; the collectAllUnsafe / .via(InlineExecutor) inside handlers → when_all / direct co_await.

6. CP callbacks — hs_cp_callbacks.cpp

MyCPCallbacks::cp_flush returns folly::Future<bool> → sisl::async::task<bool> (mirror homestore cp_mgr's cp_start_flush). on_switchover_cp / cp_cleanup / cp_progress_percent are sync. homeblocks didn't implement CPCallbacks (used homestore's) — new but small surface.

Suggested sequence (bottom-up; keep it building green at each step)

  1. conanfile: bump deps (table above), C++23, editable. Get it configuring before touching code.
  2. Public API (common.hpp + pg/shard/blob headers): Manager<E> aliases → task / std::expected; keep the {code, leader} error structs (#4). This is the contract — defer its exact shape to the owner, like homeblocks did.
  3. Memory backend (no homestore deps → cleanest warm-up): makeSemiFuture/makeUnexpected → co_return/std::unexpected; ConcurrentHashMap → unordered_map+shared_mutex.
  4. homestore_backend core: hs_homeobject init (iomgr_params, format_and_start) → the repl_application → the replication_state_machine (#1) → the three managers (co_await repl_dev::async_alloc_write / async_read; repl_result_ctx Promise → value_awaitable).
  5. index_kv (IndexTable wrappers): put/get/query/remove → status; destroy() is async — co_await the forced CP flush, and don't run it on a reactor you then block (the homeblocks IndexTable::destroy deadlock lesson).
  6. CP callbacks (#6) → snapshot/resync (#2) → GC (#3) → HTTP (#5).
  7. Tests (fixture_app): drop folly::Init; .get() → sync_get; collectAll → when_all; build the config the new init takes.
  8. Drop Folly from conanfile + any find_package/includes; grep clean.

Gotchas (carry straight over from homeblocks)

  • sync_get/sync_wait is safe only OFF a reactor — it parks the reactor's iomgr loop. Never sync_get a CP-flush- or repl-awaiting op on a worker reactor; co_await it. homeobject's GC and destroy paths are the risk areas. (homeblocks memory: sync-get-on-reactor-deadlock.)
  • exec::task is lazy — an un-driven task never runs; std::ignore = task is a silent no-op. co_await / sync_get / detach it, and mark async entry points [[nodiscard]].
  • Keep buffers/sg_lists alive across co_await (frame-owned), like homeblocks's sgs_keepalive.
  • alloc_blks truthiness flipped — a value now means success.
  • uintptr_cast expansion trap: when expanding the removed sisl cast macros, uintptr_cast(p) is reinterpret_cast<uint8_t*>(p), not static_cast/<uint32_t*> — homeblocks had 5 latent pointer bugs from a bad expansion (superblock chunk-id pointers). Grep every uintptr_cast site.
  • Coroutine purity: co_await/co_return only; don't re-invert with a detach_then-style callback.

Effort (from the survey; one engineer)

Area Effort
Public API (Manager<E>, 3 manager headers) 2–3 d
Memory backend 1–2 d
replication_state_machine (critical path) 2–3 d
HS BlobManager (heaviest file) 3–4 d
HS ShardManager 2 d
HS PGManager 2–3 d
GC manager (executors → reactors/pool) 3–4 d
Snapshot / baseline-resync (folded into resync work)
HTTP manager (Pistache → httplib) 1–2 d
CP callbacks 1 d
Tests & fixtures 2–3 d

~20–27 person-days. Critical path: Public API → replication_state_machine → BlobManager.

HomeBlocks

CRAFT

Clone this wiki locally