Sharding And Owner Routing

Shard ownership and owner routing

How the server decides which shard owns an execution and what happens when it isn't this one. Both changed for the embedded-state program (noetl/ai-meta#332); both are no-ops at shard_count = 1, which is where they were deliberately landed.

---

Ownership: the partition table

// src/sharding.rs
pub const PARTITION_COUNT: u32 = 4096;
pub fn partition_for(execution_id: i64) -> u32;              // N-independent
pub fn shard_for_partition(partition: u32, n: u32) -> u32;
pub fn shard_for(execution_id: i64, n: u32) -> u32;          // the two composed
pub fn partition_compatible(shard_count: u32) -> bool;

An execution's partition is decided once and never changes. Resizing only reassigns partitions between shards, so a rebalance is a bounded, resumable unit of work rather than a re-hash of every key.

What it prevents. The old mapping was hash(id) % shard_count directly. Fine while the shard map only picks a connection pool; near-worst-case once a shard owns embedded state, because N → N+1 remaps roughly every key — i.e. moves nearly all the data.

⚠⚠ The constraint that shaped it — do not "improve" the mapping

This mapping is implemented three times, in two repos that cannot change in one commit:

server/src/sharding.rs	shard_for(i64, u32)
ehdb-l0/src/dataset.rs	shard_for_execution(&str, u32)
ehdb-reference/src/affinity.rs	shard_for_execution(&str, u32)

Server and command bus disagreeing about who owns an execution is the one disagreement that cannot be tolerated — both would write, and the log forks.

So the two-level form is deliberately bit-identical to the old one: (h % P) % n == h % n exactly when n | P. PARTITION_COUNT = 4096 makes every power of two a legal shard count, and partition_compatible refuses the rest rather than silently splitting the mapping in two.

the_partition_table_is_bit_identical_to_the_ehdb_side_mapping reimplements the other side's formula verbatim and sweeps ids × shard counts. If you change the mapping, all three move together or none do.

---

Routing: ExecutionAffinity, and fail-closed forwarding

src/affinity.rs is a single-hop keyed shuffle for POST /api/events: owns() decides locally; otherwise the request is forwarded to the owner, with AFFINITY_FORWARDED_HEADER as a loop guard (one hop, never a loop).

⚠⚠ A failed forward is a 503 — it must never become a local write

route_event used to degrade to ProcessLocally on every forward failure — owner unreachable, non-2xx, undecodable body.

That was correct while Postgres is the system of record and any replica can write it: a failed forward cost ordering, not data.

Under embedded per-shard state it is a correctness violation. The owning shard holds the only copy of that execution's event log, so handling the write here forks that log — permanently, silently, on a transient network error. It spends exactly the single-writer guarantee owner routing exists to provide.

Failures now return AppError::OwnerUnavailable → 503. A dedicated variant, not AppError::Nats — which happens to map to 503 but says something false about the cause. The 503 hands the retry decision to the caller, the only actor that can make it safely.

This is also the prerequisite for dynamic routing. Once addresses come from a membership table rather than a static template, a stale address must fail closed — with degrade-to-local still in place, a stale hint would silently fork the log.

The routing table

pre_forward(active, already_forwarded, parses, owns, has_owner_url) is a pure function so the decision is testable without a peer, a runtime or a socket. Every Local(..) arm is a case where local handling is correct — none of them is a fallback.

---

At shard_count = 1

active() is false and owns() is unconditionally true, so nothing forwards and nothing can fail closed; every execution is partition-owner 0 either way. Both changes were landed here on purpose: the safest place for a correctness change is where it cannot fire.

Related

• Central record: noetl/ai-meta#332, and docs/rfc/ehdb-embedded-state.md in ai-meta for why the store is moving in-process.
• ⚠ noetl/ai-meta#266 records NOETL_STATE_AFFINITY_ROUTE as inert. That is true of that env flag, not of the affinity layer — owns() and route_event() are both reached.