Delivery Semantics

Felix semantics

The behavioural contract: what an acknowledgement means, what survives what, and where each guarantee stops.

Every normative claim below names the test that holds it. A guarantee with no test behind it is an intention, and this document has been wrong before by describing intentions in the present tense. Where something is not enforced, it says so rather than omitting it.

Written for the replicated, multi-broker system. It replaces the single-node MVP contract, which described a system that no longer exists — at-most-once delivery, no authorization, no durability.

What an acknowledgement means

An acknowledgement is the only promise Felix makes about a record, and what it promises depends on two settings that are chosen separately.

Durability: FsyncMode

Where the record is when the broker acknowledges it.

Mode	An ack means	Lost by
None	The record is in the page cache	Losing the machine
Periodic { interval }	The record is in the page cache, and will be fsynced within interval	Losing the machine within interval of the write
OnCommit	The record is fsynced	Nothing short of the disk failing

Periodic is the default. OnCommit is the only setting under which an acknowledged record survives the loss of the machine, and group commit is what keeps it affordable: one flush serves every waiter queued behind it.

A crash mid-append leaves a torn tail, which recovery repairs by discarding the incomplete record — it was never acknowledged. Corruption in the interior of a segment is refused instead: the broker will not start. Refusing to start beats silently losing an acknowledged record.

Held by crates/felix-storage/src/disk_log/ recovery tests, including a_crash_before_the_header_is_written_leaves_the_log_openable, a_crash_while_preparing_a_rollover_leaves_the_log_openable, and a_crash_while_sealing_a_retired_segment_loses_nothing.

Consistency: how many brokers hold it

Level	An ack means	Loss window
Leader	The shard's leader has it durably	Everything the leader had not yet shipped, if its storage is lost
Quorum	A majority of the replica set, leader included, has it durably	None within the replica set

Quorum waits. A publish is not acknowledged until the leader can show a majority holds the record; if no majority is reachable it is refused, with an error that says this broker cannot vouch for the write rather than one that claims it failed.

a_quorum_publish_without_a_majority_is_refused — freeze every follower and the publish is refused rather than acknowledged. a_frozen_follower_does_not_block_a_quorum — losing a minority does not stop it, which is the case Quorum exists to tolerate. a_quorum_acknowledged_record_survives_its_leader — the acknowledged record is readable after the acknowledging broker is killed.

That last one holds for the faults the suite injects. A model check of the promotion protocol finds an interleaving it does not reach — a leader that acknowledges and then dies before its next report leaves a fresh report naming a replica without the record, and promotion picks from that report. See docs/replication-design.md under "Who may be promoted", and #527 for the rule change that closes it.

The Leader loss window is bounded by replication lag, exported as felix_broker_replication_lag_records. An operator choosing Leader is choosing that window, and a bound nobody can observe is not a bound.

A majority is of the replica set, leader included: a set of three needs two, a set of five needs three, and a set of one needs one — which is why replication_factor: 1 costs nothing.

Failover

A shard's leadership is a time-bounded lease issued by the control plane, with the assignment generation as its epoch. Per-shard Raft was considered and rejected; docs/replication-design.md records why.

Only a replica that holds every record the leader did may be promoted. The catch-up bound is zero. A bound above zero would be a bound on how much a promotion may silently lose, and there is no honest non-zero value that is not a policy decision.

the_promoted_leader_is_one_of_the_replicas and an_unreplicated_shard_does_not_fail_over_to_an_empty_broker — a shard with no qualifying replica is left unavailable rather than served empty. failover_completes_within_the_configured_bound. records_survive_repeated_failovers — two failovers in a row, not just one.

A leader that was frozen past its lease and then resumed cannot acknowledge a write the cluster has lost. It wakes still believing it leads; the lease and the generation are what stop it.

a_resumed_leader_does_not_acknowledge_writes_the_cluster_loses.

The lease depends on bounded process suspension, not on synchronised clocks: each broker measures elapsed time on its own monotonic clock and gives up a quarter of the lease as margin. See "Where the guarantees stop" below.

Routing

A publish and a subscribe make opposite choices, deliberately.

• A publish to a broker that does not own the shard is forwarded to the one that does, and acknowledged only once the owner has answered.
• A subscribe to a broker that does not own the shard is redirected, naming the owner. It is never served locally: that would deliver nothing while looking exactly like a stream with no traffic.

a_publish_to_a_non_owner_is_still_forwarded, a_subscribe_to_a_non_owner_is_redirected, a_cluster_client_follows_the_redirect_to_the_owner. docs/subscribe-routing.md and docs/internal-protocol.md record the two decisions.

A publish may carry a routing key, and the key decides the shard. A stream placed with more than one shard therefore spreads across brokers, which is the mechanism by which one stream scales past a single owner.

a_sharded_stream_is_placed_across_brokers, keys_spread_records_across_shards, a_keyed_publish_is_forwarded_to_the_shards_owner.

A publish with no key lands on shard 0, which is what every record did before keys existed and what a single-shard stream does regardless.

an_unkeyed_publish_still_lands_on_shard_zero.

Keyed publishes use the binary encoding, like unkeyed ones. The key rides in the frame under FLAG_BINARY_PUBLISH_KEYED (0x0040), negotiated on the handshake; a broker that predates the bit gets the JSON encoding instead, which costs throughput rather than correctness.

a_keyed_publish_negotiates_the_binary_frame, an_unacked_keyed_publish_reaches_its_shard.

Delivery to subscribers

• Ordering is per key on a sharded stream, and per stream on a single-shard one. Two records sharing a routing key are ordered with respect to each other, because a key always resolves to the same shard and a shard is one log on one leader. Two records with different keys may be applied by different brokers in either order.

  A stream with one shard keeps total order whatever keys are used, which is what makes routing keys safe to add to an existing stream: the guarantee only weakens when the shard count does the widening.

  There is no ordering across streams.

  one_key_always_lands_on_one_shard.

• A slow subscriber is dropped from, not blocked on. Each subscriber has a bounded queue with an explicit overflow policy, DropNew by default. A publisher never waits for a subscriber.

• A drop is detectable. Delivered events on durable streams carry log offsets, so a gap in offsets is exactly a drop. This is the only way an application can tell, and it is why offsets are on the event rather than inferred.

• A subscription can resume. Subscribe takes latest, earliest, or an offset; stored history joins live delivery with no gap.

• A subscription reads one shard. Shards of a stream can have different owners and a subscription is bound to one connection, so reading a whole multi-shard stream means one subscription per shard. Offsets are per shard, so resuming a multi-shard consumer means carrying one offset per shard.

  ClusterClient::subscribe_sharded opens one per shard and merges them, following each shard's own redirect. It promises per-shard ordering and nothing more: merging cannot restore an order that never existed. An unreachable shard refuses the whole subscription rather than covering three shards of four, and a shard whose owner is lost is re-established on its own while the others keep delivering.

  a_sharded_subscription_receives_every_record, an_unreachable_shard_refuses_the_subscription, one_shard_failing_over_does_not_stop_the_others, a_sharded_subscription_resumes_from_its_per_shard_offsets.

Three fields on a stream are declared and not enforced. The control plane accepts them and stores them; the broker reads only durable, shards and consistency.

Field	What actually decides
delivery (AtMostOnce / AtLeastOnce)	How a client reads: a plain subscription, or a consumer group
retention (max_age_seconds, max_size_bytes)	The broker-wide FELIX_DURABLE_RETENTION_* settings
kind (Stream / Queue / Cache)	Nothing. A queue is a way of reading a stream, not a kind of stream

kind is the one most likely to mislead. Creating a stream with kind: Queue does not make it a queue and does not stop it being subscribed to normally — consumer groups work over any durable stream, and a stream created as Stream serves them just as well. Do not rely on any of the three.

Clients

A client is told which brokers exist and is redirected to the right one, but the retry policy is still the application's. docs/multi-node-client.md is the how-to; this is the contract.

• ClusterClient takes several broker addresses and rebuilds its connection from the rest when the one in use fails.

• It asks a broker which other brokers exist and adds them to what it will try. The configured seeds are never removed, so a wrong or stale answer cannot leave a client with fewer ways in than it started with.

• publish reconnects but does not resend. publish_at_least_once resends, and can therefore duplicate a record whose failure it could not prove was not applied. The names are the contract.

• Retries are bounded by an attempt count and a jittered exponential backoff, and optionally by a deadline across every attempt and sleep. The deadline is off by default: one shorter than a single attempt's own timeout prevents any retry at all, so a useful value depends on the caller's latency budget rather than on a number this library can pick.

• The backoff is full jitter — uniform over [0, ceiling], not the ceiling. Every client notices a failover at the same moment, and an unjittered backoff sends all of them at the freshly promoted broker in step.

• A failure that cannot succeed on another attempt is not retried. That is a short list: a credential without the permission, and an offset retention has passed. Everything else is retried, including errors nobody has classified — the client protocol carries an error as prose with no code, so this is string matching, and a wasted attempt is a cheaper mistake than a lost operation.

  "Not found" is retried, and deliberately. A broker learns its streams from the control plane and opens a shard only once it is given one, so a broker promoted a moment ago reports the stream it is about to serve as missing. Being named leader and being ready to serve are different moments.

a_forbidden_publish_fails_fast_instead_of_retrying, a_client_given_one_seed_learns_the_other_brokers, the_configured_seed_is_never_dropped, a_client_given_one_seed_survives_losing_it, a_publisher_survives_losing_its_broker, records_published_across_a_failover_are_all_readable.

Cache

• Scope: (tenant_id, namespace, cache, key).
• Storage: a cache is a log. A write appends a record; a read consults an in-memory index of key → offset and reads the log. See docs/cache-on-log.md.
• Durability: entries survive a restart when the broker runs with FELIX_DURABLE_STORAGE_DIR. Without one the cache is in memory and is lost, because there is nowhere to write a log.
• TTL: lazy on access, against an absolute expiry time, so an expiry that passes while the process is down is still an expiry.
• Reclamation: compaction rewrites the live set and drops superseded and expired records, without ever rewriting a record in place.

a_cache_survives_a_restart, an_expiry_survives_a_restart, a_later_write_wins, compaction_reclaims_overwritten_records.

Cache operations are routed to the key's owner (#278). A key hashes to a shard, that shard has exactly one leader, and a broker that receives an operation for a key it does not own forwards it there rather than serving a second copy. A value written through any broker is readable through every other, and two brokers cannot both accept a write for one key.

An operation this broker cannot route is refused, and a read it cannot route is an error rather than a miss — reporting a miss would let a client conclude a key does not exist when it does, on the owner.

crates/felix-cluster/tests/cache_routing.rs, including a_value_written_through_one_broker_is_readable_through_every_other and two_brokers_writing_one_key_do_not_diverge.

A cache's shards are replicated. A cache created with a replication factor above one has its log shipped to followers exactly as a stream's is, and a value written before the owning broker dies is readable from the replica promoted in its place.

crates/felix-cluster/tests/cache_failover.rs::a_cache_value_survives_the_loss_of_its_owner.

A key or prefix can be watched (#348). cache_watch — negotiated as FEATURE_CACHE_WATCH, and offered only by a log-backed cache — delivers each applied write in the shard's write order with its log offset: a put with its value, a delete as a change with none. Resume by offset replays from the log and joins live delivery with no gap and no duplicate, by the same register-before-read discipline a stream resume uses. An offset compaction has collapsed is answered with a marked snapshot of current values, never a silent gap; a watch that falls behind is ended with cache_watch_lagged naming the first missed offset, because a filtered watch's offsets are sparse and a drop could not otherwise be seen. TTL expiry appends nothing and so delivers nothing. See docs/cache-on-log.md and docs/protocol.md.

services/broker/tests/cache_watch.rs, including a_key_watch_sees_its_key_and_no_others, a_resumed_watch_is_gapless_under_concurrent_writes and a_watch_from_a_compacted_offset_resnapshots; crates/felix-broker/src/cache_watch_tests.rs::overflow_ends_the_watch_and_names_the_first_missed_offset.

A watch can start from current state (#349). A retained watch delivers each matching key's current value first — at the offset of the write that produced it — then live changes: MQTT's retained message, and the join primitive state-sync applications need. retained_count in the confirmation makes joining an empty key a definite zero rather than silence. Negotiated as FEATURE_CACHE_WATCH_RETAINED, a bit of its own so an older watch-capable broker is never asked for state it would silently not deliver. Survives failover: a promoted replica serves the retained value from its rebuilt index, and the watch is live on it.

services/broker/tests/cache_watch.rs, including a_retained_watch_delivers_current_state_then_live_under_concurrent_writes, a_retained_watch_on_an_empty_key_reports_no_value and a_retained_value_survives_a_restart; crates/felix-cluster/tests/cache_failover.rs::a_retained_watch_survives_the_loss_of_the_owner.

A counter folds deltas into a durable sum (#350). counter_add / counter_get, negotiated as FEATURE_COUNTERS and offered only with durable storage. Scoped and routed exactly as a cache key — same scope, same shard, same owner, same forwarding — but stored beside the cache, so a counter and a cache value sharing a key are unrelated. Each add answers with the sum including itself; never-written is distinct from zero. The sum survives restart, compaction (checkpoints, offsets never renumbered), and leader failover, where the promoted replica folds the true sum from its shipped log and keeps counting. At least once: a retried add after a lost acknowledgement double-counts — deltas carry no dedupe identity, and docs/projections.md records the decision.

services/broker/tests/counters.rs, including an_add_answers_with_the_sum_including_it and a_counter_survives_a_restart; crates/felix-storage/src/counter_log/tests.rs::compaction_moves_neither_the_sum_nor_the_offsets; crates/felix-cluster/tests/cache_failover.rs::a_counter_survives_the_loss_of_its_owner.

What a cache still does not declare is a consistency level. A stream chooses Leader or Quorum; a cache write is acknowledged by its leader once the record is durable there, and replication follows. So losing a leader in the window between the acknowledgement and the ship loses that write, which is the Leader guarantee rather than the Quorum one.

Authorization

Enforced, contrary to what this document said for a long time. Tenant-scoped tokens are verified at the broker, and publish, subscribe and cache operations each check a permission. A forwarded publish is authorized at both the ingress broker and the owner — routing does not launder a credential.

crates/felix-cluster/src/scenarios.rs, including unauthorized_publish_is_refused across both ingress paths.

Per-tenant quotas are not enforced.

Where the guarantees stop

Stated because a guarantee without its failure model is a slogan.

• The failure model is process loss and partition. Kill, graceful stop, freeze, and severing a broker from its peers while it keeps running are all injectable and tested. A partitioned leader keeps heartbeating, so the control plane goes on believing it is healthy while it can reach nobody — and it cannot acknowledge a Quorum publish, because it is not a majority on its own.

  a_partitioned_leader_cannot_reach_a_quorum, a_healed_partition_restores_the_quorum, a_partitioned_leader_still_serves_a_leader_stream.

• Clock skew between brokers cannot affect lease safety, because no lease reads a wall clock. Each broker measures its own elapsed time on a monotonic clock and gives up a quarter of the lease as margin, so two brokers' disagreement about what time it is has nothing to act on. The assumption that does matter is bounded process suspension, and that is injectable — a broker frozen past its lease and resumed is the test above.

  A broker suspended between the commit-time lease check and its write reaching disk is a residual window bounded by the margin. The margin is a choice rather than a proof, and it is the one clock-shaped assumption left.

• Idempotent producers, not exactly-once delivery. A producer that takes an id from the broker and numbers its batches can re-send a batch whose acknowledgement never arrived and have it land once: the shard's leader answers a sequence it already holds from memory rather than appending it. That closes the ambiguous-outcome gap on the publish side, while the leader that took the first copy is the one answering; a new leader knows no producers and says so, so a batch in flight across a failover is reported rather than silently landed or dropped. The consumer side is unchanged: a subscriber can still see a record twice on redelivery, and there are no transactions. See docs/protocol.md, "Idempotent producers".

  a_re_sent_batch_lands_once, a_non_leader_names_the_leader, and a_gap_is_refused_with_the_expected_sequence in crates/felix-cluster/tests/idempotent.rs, on a Quorum stream replicated three ways; racing_re_sends_append_once in crates/felix-broker for the two re-sends that arrive at once.

• No cross-region ordering or routing guarantees.

• A group is bound to the shard the caller names. Consuming a whole multi-shard stream through a group means polling each shard's group separately, for the same reason a subscription reads one shard.

  Delivery is by poll rather than push: a consumer takes work when it has capacity, and the broker cannot know when that is. A poll can ask the broker to wait for work, so an idle consumer costs one open request rather than a round trip per attempt. Only the broker leading a shard serves its groups, and ownership is re-checked while a poll waits — a shard that moves mid-wait ends the wait rather than being served by its former owner.

  The rules are fixed. A group's position on a shard is durable, monotonic, and survives a restart. Above that position the broker tracks what has been handed out: a record claimed by one consumer is not handed to another while the claim stands, a claim that lapses makes the record owed again, and the cursor moves only over a contiguous run of acknowledgements — never past a gap, which would mark a record finished that nobody finished. Two groups over one shard are independent; each sees every record.

  Retention outranks a group. A record removed by retention before a group reached it is skipped, and the group moves past it — leaving it owed would stall the group for ever on a record that exists nowhere. That is the one case where a queue drops work, it is counted rather than silent, and it means a retention window shorter than a group is allowed to fall behind loses work.

  The in-flight set is deliberately not durable. A leader that dies loses it and the group resumes from its cursor, so those records are delivered a second time. That is at-least-once, which is what a queue offers regardless; persisting it would narrow the redelivery window at the cost of a write per delivery and still would not make delivery exactly-once.

  Group state survives failover whole (#314, and now the dead letters too). Both of a shard's group logs — the cursors and the offsets its groups gave up on — replicate beside the shard's records, on the same replica set at the same generation. A promoted replica resumes each group where it had reached, lists what it had abandoned, and serves an operator's redrive.

  crates/felix-cluster/tests/consumer_groups.rs, including a_group_position_survives_a_leader_failover and a_dead_letter_survives_a_leader_failover.

• Retention is per stream and unbounded by default. A stream with no retention policy grows until the disk does not.

---

Generated from docs/semantics.md in the repository. That file is the source of truth; this page may be behind it.