Felix v0.6.0-preview.3

The third preview of 0.6.0. felixctl now manages a cluster as well as using
it: it creates, changes and deletes tenants, namespaces, streams and caches,
works consumer groups, counters and RBAC rules, and shows a shard's or a
subscriber's live state with felixctl inspect. It installs with Homebrew.
Every change of a shard's leader is fenced, and replicas keep a ballot with
each generation they accept. Sealed segments can be copied to an object store
before retention deletes them, the first step toward tiered storage.
felix-capi adds a C ABI over the Rust client, the base for Go and C# SDKs.
Caches gain conditional writes and key-scoped grants, and a client can read a
bounded range of a stream. Development works with Docker or Podman. Several
move, failover and lease-timing bugs are fixed. The Rust API has breaking
changes, listed under Changed. pip will not install it without --pre.

Known issue

On a Quorum stream (replication factor above 1) with a small per-stream retention limit,
retention on the leader can delete records that a follower has not yet stored, so an
acknowledged write can be lost after a failover. Until 0.6.0-preview.4, don't set a small
per-stream retention on Quorum streams. Tracked in #1094; the fix is #1102. Fixed in v0.6.0-preview.4, together with the restart window it left open (#1111).

Added

• A Homebrew formula for felixctl: brew install getfelix/tap/felixctl on
  macOS and Linux, with completions and man pages (#1005). release.yml
  writes it from each release's archive checksums and pushes it to
  <owner>/homebrew-tap when PUBLISH_HOMEBREW is true, using the
  HOMEBREW_TAP_DEPLOY_KEY secret (a write deploy key on the tap); dry runs keep it as an artifact and nightlies
  skip it.

• felixctl inspect subs and the subscriptions_list request (part of
  #1077). A broker advertising FEATURE_INSPECT now also answers
  subscriptions_list with subscriptions_list_info: one page of the
  subscriptions it serves, each with its stream and shard, subscriber and
  subscription ids, connection id, client address and principal, overflow
  policy, queue depth and capacity in batches, records dropped, position and
  the shard's tail. The filter narrows by tenant, namespace, stream, shard,
  principal, or to subscriptions that have dropped records. A page is 100 by
  default and at most 1000, and a cursor of shard and subscriber id continues
  it. It needs node.view:cluster:*. The broker reads each shard's fanout
  snapshot and never touches a queue. On the delivery path, a subscriber's
  receiver now stores one past the offset of each batch it takes (a relaxed
  store), and a drop adds to a per-subscriber count beside the existing
  metric. felixctl asks every broker, or only --node, and prints one table
  or one JSON line per broker, with --principal, --dropping, --limit and
  --cursor. felix-broker adds Broker::list_subscriptions,
  SubscriberStats, SubscriberOwner and Subscription::set_owner;
  felix-client adds Client::list_subscriptions and SubscriptionsPage, and
  its inspect calls now report a broker's unsupported answer as such.
  Diagnosing a cluster covers a subscriber missing records and one falling
  behind.

• Stream logs can copy their sealed segments to an object store before
  retention deletes them, the first part of tiered storage (#172). Off by
  default; FELIX_DURABLE_OFFLOAD_DIR turns it on, and the only backend is a
  directory, through the object_store crate's local filesystem backend. A
  pass on the retention timer uploads each sealed segment, reads it back to
  check size and CRC-32, and records it in a per-shard offload.manifest,
  fsynced. Retention then deletes only recorded segments, so a crash at any
  step leaves the local segment or a recorded copy. Recovery accepts a gap in
  the local chain when the manifest covers it. Nothing reads the copies yet: a
  read below the local head is still Trimmed. New metrics
  felix_storage_offload_segments_total, felix_storage_offload_bytes_total
  and felix_storage_offload_failures_total. Breaking for code that builds
  felix-storage's LogConfig without ..Default::default(): it gains
  offload.

• felixctl inspect shard and the shard_inspect request (part of #1077).
  A broker advertising FEATURE_INSPECT, the first bit of the extended
  feature word (server_features_hi 0x1), answers shard_inspect with
  shard_inspect_info: its own view of one shard, with its phase and
  generation, whether it serves and why not (opening, fencing, failed
  with the open error, draining, lease_lapsed, not_assigned_here,
  behind_generation), which replicas took a promoted leader's fence and
  how many attempts it has made, its lease, tail and commit mark, and as
  leader each follower's next offset, lag and state. It needs
  node.view:cluster:* and may name any tenant. The answer is read from
  snapshots the lifecycle and the replication driver publish (the new
  felix_replication::status::ShardStatusBoard), so it takes neither's lock
  and opens no log. felixctl asks the leader and every replica and prints a
  table, or one JSON line per shard with --json; a broker that cannot be
  reached is listed as unreachable. felix-client adds
  Client::inspect_shard and Client::supports_inspect. A new docs page,
  Diagnosing a cluster, goes through shard, replication, subscriber, auth
  and startup problems by symptom.

• felix-capi, a C ABI over the Rust client and the base for the Go and C#
  SDKs (part of #618). It builds libfelix as a shared and a static library
  with a cbindgen header checked in at crates/sdk/felix-capi/include/felix.h,
  and covers connect, publish and a polled subscribe (felix_client_connect,
  felix_client_publish, felix_client_subscribe,
  felix_subscription_next_event). Handles are opaque with a free function
  each, every call returns a status code whose classes match the other SDKs,
  the message for a failure is per thread (felix_last_error_message), panics
  never cross the boundary, and each client owns its Tokio runtime. A test
  fails while the header is stale, and CI runs a C program against the
  conformance fixture. Not published to crates.io.

• Replicas keep a ballot with each accepted generation (part of #1009): the
  leader they accepted it from, by the node id its Hello gave. At that
  generation a replica refuses a fence, a batch, a bootstrap, a rebuild or a
  tail fetch from any other node with FencedEpoch, and a broker will not
  lead a generation it already accepted from another node. The ballot is a new
  ballot file in the shard directory, fsynced before anything from that
  leader is answered and read back on open; builds without ballots ignore it,
  so rolling back still opens the shard. A new internal capability, BALLOTS
  (1 << 6), is offered with the fence, and FELIX_INTERNAL_FENCE=false
  turns ballots off with it. Nothing changes while the control plane names
  every leader, since it never names two at one generation; this is the
  safety layer replica elections will need. The TLA+ model gains Ballots,
  Elections and OneLeaderPerGeneration, with FelixShardElect.cfg and
  two configurations that must fail: FelixShardElectNoBallot.cfg (two
  leaders at one generation) and FelixShardElectStaleSet.cfg (a replica
  standing on a set it has left). Breaking for callers of felix-storage's
  DiskLog::accept_generation and felix-broker's
  StreamLog::accept_generation, which take the leader, of felix-replication's
  ReplicaHandler entry points, which take the sender, and for code matching
  GenerationCheck, which gains Promised and is no longer Copy.

• felixctl group and felixctl counter (#1005). group create|describe|seek|rm
  manage a consumer group on every shard of a stream, or on --shard.
  group poll claims records and prints each with its claim,
  SHARD:OFFSET:ATTEMPTS, which group ack, nack (with --delay-ms),
  extend and dead-letters add take. group dead-letters ls|redrive|discard
  work the dead-letter list. group rm, dead-letters discard and a seek
  that moves a cursor back over finished records ask [y/N] at a terminal and
  need --yes elsewhere. counter get|add read and add to counters. All go
  through ClusterClient. felix-client now re-exports GroupRecord, which
  group_extend takes.

• Nightly TLC simulation of long random walks over replica-set changes
  (#934). FelixShardWalkSpares, FelixShardWalkMoves and
  FelixShardWalkHandoff lift the bounds the exhaustive configurations
  stop at (time to 40, six writes, six moves) and sample traces up to 300
  steps; each has a negative twin that must find its violation within the
  budget. scripts/check_tla.sh --simulate (task tla:walk) runs them,
  and tla-walk.yml runs them nightly and uploads TLC's trace on a
  failure. Tick in FelixShard.tla enumerates clocks over the drift
  window only, with the same states.

• felixctl writes to the control plane (part of #1005). tenant, namespace,
  stream and cache gain create and rm; stream set changes
  consistency, delivery, durability and retention, keeping a retention bound
  not given, and cache set the display name; each prints what changed.
  node drain and node deregister, shard move NAME SHARD --to NODE (with
  --dry-run and --cache) and shard move cancel, and placement pause,
  resume and abandon drive placement. Deletes, drains and deregistrations
  ask on a terminal and need --yes anywhere else, stopping with status 2
  without it; placement abandon never asks and always needs --yes.

• felixctl rbac policy ls|add|rm and felixctl rbac grouping ls|add|rm list,
  grant and revoke the current tenant's RBAC policies and role assignments
  through the control plane (#1005). policy add checks the object against
  the RBAC grammar first, including cache key and key-prefix objects and their
  cache.read/cache.write-only rule, and stops with status 2 and the reason;
  action names and delegation scope are left to the control plane, whose
  refusal is printed with status 4. rm asks on a terminal and needs --yes
  elsewhere, and exits 5 when the rule does not exist. felix-cluster adds
  Credentials::rbac_admin_token.

• A subscriber can choose its own broker-side queue capacity on a broker
  advertising FEATURE_SUBSCRIBE_QUEUE (0x4000_0000) (#1019). subscribe
  takes an optional queue_capacity, counted in published batches; the broker
  clamps it to 1..=FELIX_SUBSCRIBER_QUEUE_CAPACITY_MAX (new, default 4096)
  and echoes the grant as queue_capacity on subscribed. A subscribe
  without it, and the answer, are byte-identical to before. The overflow policy
  stays the stream's: a subscriber that could choose Block could stall every
  publisher on its shard. felix-client adds
  ClientConfig::broker_sub_queue_capacity and Subscription::queue_capacity;
  felix-broker adds Broker::subscribe_sized and
  Broker::subscribe_from_sized. ClientConfig gains a public field, so code
  building it with a struct literal must set it.

• One Rust Client can act for many users over the same connections (#969).
  Client::with_identity(tenant_id, token_provider) and
  Client::with_identity_token return a client that shares the parent's
  connections and authenticates every stream it opens with the user's token,
  so the broker checks each user's publish, subscribe, cache and group
  requests against that user's grants alone. It opens one publish and one
  cache stream and no connections. One user's token expiring or being revoked
  stops only that user's streams. No wire change: the broker already
  authenticates per stream. A refused publish or cache request still ends the
  stream it came on, so a gateway should build a new identity after one.
  ClusterClient does not offer it yet.

• A write can be made only at the offset its writer expects, on a broker
  advertising FEATURE_PUBLISH_CONDITIONAL (0x1000_0000) (#1017).
  publish_if appends a batch only if it would start at expected_offset, the
  shard's next; commit takes the same optional expected_offset as a
  compare-and-set on the whole shard. The check is made where the log assigns
  offsets, under the same lock, so of two writers at one offset exactly one is
  written. A refusal is publish_refused with the new offset_mismatch reason
  and the shard's tail; it writes nothing, consumes no offset and holds up no
  later publish. The tail counts a new leader's generation-start record, so a
  writer's expected offset goes stale on failover. Served by the shard's
  leader only: a non-leader answers not_leader. An in-memory stream refuses
  it. A commit without expected_offset is byte-identical to before.
  publish_if is charged against the tenant's publish quota and the ingress
  byte budgets like any acked publish, and refused over quota the same way.
  felix-client adds publish_if and commit_if on Client and
  ClusterClient, answering ConditionalWrite; felix-storage adds
  DiskLog::append_claimed_at, and felix-broker Broker::claim_publish_at,
  publish_batch_at and commit_to_handle_at. Per-key version
  preconditions are #1051.

• A second feature word for when the u32 feature set runs out (#1055).
  FEATURE_EXTENDED (0x8000_0000) in client_features or
  server_features says the peer sends and reads client_features_hi /
  server_features_hi. A peer sends the marker and the word only when it
  knows a feature there, and a broker answers with its word only to a client
  that set the marker, so every existing frame is unchanged. No feature uses
  the word yet. felix-wire adds FEATURE_EXTENDED, KNOWN_FEATURES_HI,
  offer_features, answer_features and peer_features_hi. Breaking for
  code that builds Message::Auth or Message::AuthOk: both have a new
  field.

• A client can read a bounded range of a durable stream and stop (#1018). A
  broker advertising FEATURE_STREAM_READ (0x2000_0000) answers
  stream_read (shard, from, optional end, max_records, max_bytes)
  with stream_records: one page of committed records and the next_offset
  to continue from, which steps over generation-start records. No subscriber
  is registered and the read never waits. A page stops at the committed mark
  on a Quorum shard and, under FsyncMode::OnCommit, at the durable offset,
  and is capped at FELIX_DURABLE_MAX_RECORDS_PER_READ records and 4 MiB of
  payload. A from below retention or past the tail gets
  subscribe_cursor_error. Needs stream.subscribe; only the shard's leader
  answers. In felix-client, Client::read and ClusterClient::read return a
  StreamPage; felix-broker has Broker::read_range. Frames to clients that
  do not send the request are unchanged. An end on Subscribe is tracked
  in #1053.

• A consumer can manage its own claims on a broker advertising
  FEATURE_GROUP_CLAIM_CONTROL (0x800_0000) (#974). group_extend keeps a
  claim standing while the work goes on, answered with group_extended; it
  names the delivery by offset and attempts, so once the claim lapses and
  the record goes out again it is refused with stale_claim. delay_ms on
  group_nack makes the record owed only after a delay, holding a place under
  the in-flight cap until then. group_dead_letter lists a record as a dead
  letter and finishes it, written to the dead-letter log first as when the
  broker gives up, so group_redrive works on it. visibility_ms on
  group_poll sets how long that poll's claims stand. All need
  group.consume. Durations are capped by the new
  FELIX_GROUP_MAX_VISIBILITY_MS (12 hours, never below the visibility
  timeout). Both new fields are left out at zero, so nacks and polls that do
  not use them are byte-identical to before. A group's in-flight state is no
  longer evicted while a claim stands. felix-client adds group_poll_with and
  GroupPollOptions, group_extend, group_nack_after, group_dead_letter
  and supports_group_claim_control on Client and ClusterClient, and
  extend, nack_after and dead_letter on ShardedGroup. Breaking for
  callers of felix-broker's GroupReader::poll_below, which takes the claims'
  visibility, and for code matching felix-wire's GroupPoll and GroupNack,
  which gain fields.

• Streams and caches can be created together, all or none (#967).
  POST /v1/tenants/{t}/namespaces/{ns}/resources takes up to 256 stream and
  cache create bodies and writes the new ones in one Postgres transaction or
  one Raft log entry. A missing permission, an invalid item, a name given
  twice, or an item that already exists with a different configuration fails
  the whole batch and creates nothing. An item that already exists as asked is
  reported unchanged, so a batch can be sent again after a failure. Under
  Raft this is metadata level 5; the endpoint answers 503 until every
  control-plane member runs a release that has it.

• Cache grants can name a key or a key prefix (#966). A fourth segment on a
  cache object, cache:{tenant}/{ns}/{cache}/{key} or .../{prefix}*, limits
  cache.read or cache.write to that key, or to keys starting with the
  prefix (a plain string prefix: user:1* covers user:10). The broker checks
  get, put, delete, conditional writes, counters and key watches against the
  key, a prefix watch against its prefix, and a forwarded op again at the
  owner. Grants without a key segment still cover the whole cache. The control
  plane refuses a key with * anywhere but the end, a bare *, a wildcard
  namespace or cache under a key, and any other action on a key object; token
  exchange narrows a whole-cache grant to a key hint. No wire change. Mixed
  versions fail closed: an older broker refuses requests a key grant would
  allow. felix-authz adds PermissionMatcher::allows_cache_keys and
  CacheKeys; the control plane's ParsedObject::Cache gains a key field.

• Token exchange can narrow by action and resource pairs (#968).
  permissions: ["stream.subscribe:stream:t1/ns/a", "stream.publish:stream:t1/ns/b"]
  narrows each pair on its own against the grants with exactly that action,
  where requested and resources apply every action to every resource. It
  never widens what RBAC grants. It must be sent with "requested": [] (a
  400 otherwise), so a control plane that predates the field refuses rather
  than minting full rights, and the refresh record stores the same [] for
  the same reason. Refresh keeps the pairs, and the dev-token route accepts
  them too. Under the Raft store the field is metadata version 5: an exchange
  with pairs is a 409 until every member reports it.

• Delivered events and group records can carry the record's append time, and
  a client can look up an offset by time (#975). A subscriber that offers
  FLAG_EVENT_BATCH_TIMESTAMPS (0x4000) gets a u64 of microseconds before
  each event on batches from a durable stream, the same live and on replay. A
  broker advertising FEATURE_RECORD_TIMESTAMPS (0x400_0000) answers
  offset_for_time with offset_value: the first offset on a shard appended
  at or after a time, found by the binary search the Kafka listener's
  ListOffsets now shares through StreamLog::offset_for_time. A client that
  offers the feature gets timestamp_micros on group records. Nobody else's
  frames change, and no storage format changes. In felix-client, set
  ClientConfig::timestamps and read Event::timestamp_micros; call
  Client::offset_for_time and subscribe at the answer. Times are the clock of
  the broker that led when the record was written, and replicas keep them
  (#1045). Breaking for callers of
  felix-broker's StreamLog::begin_append, begin_append_marked and
  continue_batch, which now take the batch's timestamp_micros; felix-wire's
  EventBatchMeta, EventBatch, SharedEventBatch and GroupRecord gain a
  field.

• felix-loadgen --scenario subscribe (#980): subscribers only, at the live
  tail, counting deliveries for --duration-secs while another generator
  publishes. --fanout subscriptions are spread over --concurrency cluster
  clients, every shard of a sharded stream is read, and the run reports
  delivered events and delivered_throughput_msg_s, offset gaps (records
  dropped) and, with --stamp-send-time, delivery latency. It reports no
  publish throughput, since it publishes nothing. ingest --stamp-send-time
  writes the wall-clock send time into each payload's first 8 bytes for it;
  that latency compares two machines' clocks, so it needs them in sync.

• Nightly builds. nightly.yml builds the newest green commit on main through
  release.yml once a day, smoke-tests the images with a publish and
  subscribe, and publishes to GitHub only: images as nightly and
  nightly-YYYYMMDD on GHCR, and the binaries, wheels and Node addons on a
  rolling nightly pre-release. release.yml can now be called as a reusable
  workflow.

• Delivered events can say who published them (#1010). A subscriber that
  offers FLAG_EVENT_BATCH_PUBLISHER (0x2000) gets the principal (the
  token's sub) on each event batch, and a group consumer that offers
  FEATURE_GROUP_PUBLISHER (0x80_0000) gets it as publisher on each
  record. Nobody else's frames change. In felix-client, set
  ClientConfig::publishers and read Event::publisher. A durable stream
  stores the principal with each record (storage format v6, a byte-length
  trailer on the body), and replication ships it, so replay, promoted replicas
  and consumer groups report the same value. Storing it is off until enabled:
  finalize the publisher_principal fleet feature in a cluster, or set
  FELIX_RECORD_PUBLISHERS=true on a single broker. Either is one-way for the
  data directory, since an older broker cannot open a v6 segment.

• A consumer group has a lifecycle. group_seek moves a group's cursor on one
  shard to earliest, latest or an offset, backwards or forwards; with
  if_new it creates the group there and leaves an existing one alone.
  group_describe reports the committed cursor, the shard's committed tail, and
  what the group has in flight and owed; group_delete removes the cursor and
  dead letters. Advertised as FEATURE_GROUP_ADMIN (0x100_0000). Seek and
  delete need group.manage, describe needs group.consume. A seek voids the
  claims standing when it lands, so a late ack cannot move the new cursor or
  finish a record the group now owes. felix-client adds group_create,
  group_seek, group_describe and group_delete on Client and
  ClusterClient, and ClusterClient::group_*_stream to make the call on
  every shard of a stream. (#973)

• Cache shards are fenced on promotion like stream shards. A promoted cache
  leader fences a majority of the replica set on the cache log, then on the
  counter log, takes the furthest ahead of each by (last generation, length),
  and writes a generation-start record on each before it serves; with
  generation_start finalized, the cache and counter marks count only past
  their own log's record. A new internal capability, CACHE_FENCE (1 << 4),
  says a peer answers Fence and ReplicateFetch for the counter log and
  reads labelled cache and counter batches; a cache shard is fenced only when
  every replica offers it, and opens on the lease otherwise. A failover of a
  Quorum cache keeps its replica set, the dead leader in it, as a Quorum
  stream's does. The new fenced_caches fleet feature, finalized with
  majority_ack and generation_start, has replicated Quorum caches
  acknowledge puts, deletes and counter adds on their followers' answers
  without the report or the lease, and never open a promoted cache shard on
  the lease. Upgrade the control plane before finalizing it. Model-checked in
  FelixShardFencedCache.cfg. (#933)

• Conditional cache writes. cache_put_if stores a value only if the key is
  absent or at a given version, and cache_delete_if removes it only at a
  given version; both answer cache_condition_result with whether the write
  was made and the key's version. A get's cache_value carries the version
  for a client that offered the bit. The condition check and the write are
  atomic per key, including against a write to the key still waiting on its
  fsync. Negotiated as FEATURE_CACHE_CONDITIONAL (0x200_0000); forwarded
  between brokers as a new internal kind that an older owner refuses rather
  than applying unconditionally. felix-client and ClusterClient gain
  cache_put_if, cache_delete_if and cache_get_versioned. (#976)

Changed

• Containers run under Docker or Podman. scripts/container_engine.sh picks
  the engine (CONTAINER_ENGINE, else whichever of docker and podman
  answers) for task test, task coverage, task pg:* and task tla:check; the kcat tests read the same variable and run
  host.containers.internal under Podman. Dockerfiles, the control plane's
  compose file and the docs name images fully qualified
  (docker.io/library/...). A new docs page, Docker or Podman, covers setup
  and the differences. felix_cluster::ports::docker_host is now
  felix_cluster::container::host.
• felix_replication::promotion::Outcome::Pending is a struct variant
  carrying why and took, the replicas that took the fence in that
  attempt, and driver::Published gains status. ShardLifecycle::open_failed
  takes the error. felix_wire::KNOWN_FEATURES_HI is now FEATURE_INSPECT,
  so felix-client sets FEATURE_EXTENDED and sends client_features_hi in
  its auth; an older broker ignores the field.
• Every change of a shard's leader is fenced, not only a promotion (part of
  #1009). A move's cut-over, a failover that names a move's destination, a
  cancelled move's hand-back, and a generation of a shard its leader serves
  that skips one (a promotion elsewhere that the broker only saw coalesced
  away) now wait in fencing until a majority of the replica set takes the
  new generation, and take the answer furthest ahead, before they serve. The
  generation right after one the leader serves (a move's staging or a
  follower replacement step) still opens at once: nobody led in between. Each was a generation the control
  plane picked from its own view, opened at once; a leader it did not know
  about, from a second planner or a later replica election, could keep
  writing beside it. A write that reaches the broker while it fences is held,
  within FELIX_SHARD_MOVE_HOLD_MS, rather than refused. The lease fallback
  is unchanged: when a replica does not offer the fence the shard opens on
  the lease as before. felix_broker_promotions_opened_total{path} now counts
  every new leadership, not only promotions. The TLA+ model gains FenceEveryChange, with
  FelixShardElectHandoff.cfg (passes; an hour, so nightly),
  FelixShardElectHandoffLeaders.cfg (the same without a write, per PR) and
  FelixShardElectHandoffUnfenced.cfg (an unfenced cut-over opens a second
  leader at an elected generation). The CI model check now runs on seven jobs, filled
  by each configuration's measured time, to stay under the hour.
• A publish acknowledged on commit is answered by the task that sees it
  commit (#926). The commit task, or the executor for an in-memory write, puts
  the ack straight onto the control stream's writer queue, instead of
  completing a oneshot that a per-stream ack-waiter task awaited under a
  per-publish timer. The ack timeout is now one deadline sweep per control
  stream, and the writer sends every answer already queued in one write. An
  ack is still never sent before the publish is durable under the stream's
  fsync mode, a pipelining client still gets answers in request order, and a
  failed, dropped or timed-out publish still gets its error. Not yet measured;
  numbers come from an Azure nats-latency.sh run. The test-only subscriber
  event_writer is removed; its byte-cap test moved onto the lane feeder.
  felix_broker_ack_waiter_queue_full_total is gone with the waiter queue.
• A plain Client spreads one stream's shards over its connections, as a
  ClusterClient already did (#724). Its publishers learn a stream's width
  from the broker on the first keyed publish to it (StreamShards, one round
  trip, kept for the client's life) and put each shard on a publish stream of
  its own, placed on the least-loaded connection, so one client's hot
  multi-shard stream no longer rides one connection and one listener. Order is
  kept per shard: every publish a client makes to one shard still goes through
  one writer and one QUIC stream. Unkeyed publishes are shard 0, so a stream
  published without keys, or a single-shard stream, stays on one stream for
  total order. A stream whose width cannot be learned keeps all its keyed
  publishes on shard 0's stream. The writer for a key always comes from the
  width the client keeps, never from a caller's own shard, so plain
  publishes, a ClusterClient and an idempotent producer through one client
  share a key's writer even if they disagree about the width. The kept width
  is dropped when a publish is refused with not_found, so a stream deleted
  and created with another width is routed by its new width. A
  ClusterClient now asks each broker it publishes keyed records through for
  the stream's width once. Unkeyed publishes through a plain
  Client move from a pooled stream to shard 0's own, which costs one stream
  open per stream on the first publish. FELIX_PUB_SHARD_STREAMS now applies
  to plain clients too; 0 restores the old routing.
• A follower stores the leader's append time with each record it replicates,
  instead of stamping it with its own clock, so after a failover the new
  leader reports the same record times, and offset_for_time gives the same
  answer, as before (#1045). Brokers offer the new RECORD_TIMES peer
  capability (1 << 5); a leader sends a follower that offered it
  ReplicateTimedRecords (internal kind 36), a labelled batch with one time
  per record, and a promoted leader taking a replica's tail asks with
  ReplicateTimedFetch (kind 37). An older follower is sent what it reads,
  and a newer follower receiving from an older leader uses its own clock, as
  before. Breaking for callers of felix-broker's
  StreamLog::begin_append_marked_at and replication::apply, which take the
  records' times (empty for this broker's clock), and of felix-wire's
  ReplicateRecords and ReplicateFetch, which gain times and timed.
• Breaking for Rust callers (#1017): felix_wire::Message::Commit has an
  expected_offset field, and PublishRefusalReason and
  felix_broker::BrokerError have new variants, so struct literals and
  exhaustive matches need updating.
• A durable stream no longer reserves a whole 256 MiB segment of disk per
  shard when it is created (#1016). The active segment reserves 1 MiB (or a
  sixteenth of the segment size, if smaller) and doubles the reservation each
  time its records pass half of it, up to the segment size. Each extension runs
  on a blocking thread after the append that earned it and is best effort: a
  failure is logged, counted in felix_storage_segment_reserve_failed_total,
  and never fails an append. Reserving still leaves the file size alone, so
  recovery is unchanged. SegmentWriter::reopen takes a reservation limit.
• Creating a stream or cache that already exists with the same configuration
  answers 200 with the existing one instead of 409 (#967). A different
  configuration under the same name is still 409. A stream's routing only
  counts when the request names it. A caller that took 409 to mean "already
  there" should accept 200 as well.
• Breaking, Rust API: Broker::claim_publish, publish_batch_with_outcome,
  claim_batch_idempotent and commit_to_handle take the publisher;
  ResumedSubscription::backlog is a Vec<RingRecord>; AppendRecord and
  LogRecord have a publisher field; StreamLog's append methods and
  replication::apply take per-record publishers; ReplicateRecords has
  publishers and batch_checksum covers them.
• A compacted cache log can no longer be read by an older broker. Compaction
  now writes each copied value as a version 2 cache record, which carries the
  value's original version; a broker from before this change refuses those
  records as corruption, so a node cannot be downgraded once it has compacted
  a cache. Records a client writes are unchanged. (#976)
• felix_storage::StorageApi has three new required methods (put_if,
  delete_if, get_versioned), and CacheOp::Put a version field. (#976)
• Breaking, Rust API: IdempotentProducer owns its client and has no
  lifetime parameter. Client::idempotent_producer and
  ClusterClient::idempotent_producer take self: &Arc<Self>, so wrap the
  client in an Arc first. The producer can now be stored or moved into a
  task. (#977)

Fixed

• A request the broker refuses and then closes the stream for, such as a
  group describe without permission, reaches the client as the broker's coded
  error (#1078). The broker's response writer could see the stream's
  cancellation before the queued error and finish the stream without it, so the
  client reported "the broker closed the group stream" instead of forbidden.
• A broker notices its lease lapse within a sixteenth of the lease the control
  plane hands out, not of the 10s placeholder it starts on (#1071). The refresh
  that clears the cached admission flag, sets felix_broker_lease_held and
  ends a deposed leader's readers was paced once, at startup, so under a 1s
  lease it ran every 470ms instead of every 47ms. The commit-time lease check
  was always exact; this was admission and the metric running late. It also
  made the cluster harness's clock faults land late, since that refresh is
  what reads the clock on an idle broker, which failed
  a_frozen_leader_is_refused_by_the_majority_its_successor_fenced.
• A write that reaches a shard's new owner while it is still opening the
  shard, or just after its fence settles but before it publishes that it
  serves, is held until it serves instead of being refused with
  shard_unavailable / not_ready (#1085). Publishes, cache writes and
  counter adds were all exposed; a move's cut-over showed it as a refused
  counter add.
• The Rust client's cache worker keeps its stream after the broker refuses a
  cache or counter request (#1085). It used to stop on any error, so one
  refusal turned the requests that later landed on that worker into
  cache worker closed.
• A Quorum publish no longer waits out an unreachable follower when the
  other follower already holds the record but is still answering an earlier
  replication pass (#1080). The pass waiting for a majority had shipped only to
  the unreachable one, and the busy follower's answer reached the driver, not
  that pass, so the mark stayed put until the peer request timed out (5 s by
  default, past most publish timeouts). The answer now ends that wait and the
  next pass counts it.
• felix_broker_shard_phase reports fencing. The gauge left the phase out,
  so a promoted shard waiting for its fence was counted in no phase.
• A broker accepts a token carrying node.view or node.manage. Neither was
  an action it knew, so an operator token that also served the control plane
  failed to authenticate there.
• The docs no longer list DropOld as a working overflow policy (#1019). It is
  accepted at the broker, writer-lane and client stages but behaves as
  DropNew at each: the arriving batch is dropped, not the oldest. The status
  table, configuration reference and client docs now say so.
• JSON numbers survive a decode and re-encode exactly. serde_json's default
  float parser could land a long literal one ulp off, so an extension body the
  broker passed on carried a different number. Nightly fuzzing found it.
• Dropping an IdempotentProducer publish no longer ends the producer. The
  producer sends from a task of its own and a call waits on it, so a call
  dropped by a timeout, a select! or a hung-up handler still runs to its
  answer and the next batch goes out under the next sequence. A batch left in
  doubt with nobody waiting is re-sent by the producer, under its sequence,
  before the next batch on that shard. Dropping the producer lets what it was
  handed finish; IdempotentProducer::close waits for it. (#977)
• The history campaign could report a record delivered past the end of the
  final log (lost-delivery). Since #977 a dropped producer still sends what
  it was handed, so an append recorded as unknown could land after the final
  read. A stopping client now closes its producer and finishes its publishes
  before the final read is taken. IdempotentProducer::close_within(timeout)
  is new: close, but the producer's task is stopped at the timeout, so
  nothing it held is sent later. (#1075)
• felixctl sub, cache get and cache watch escape binary payloads on a
  terminal (\x00, \u{85}) instead of writing raw bytes that garble it;
  piped output is unchanged. felixctl bench latency payloads showed it.
• A log-backed cache's expiry could delete a value a put had just refreshed,
  when the put was waiting on its fsync as the expiry was written. The expiry
  now waits for a write to the key in flight before it checks. (#976)
• felix-cluster up held for more than an hour lost every control-plane call,
  teardown included, because its tokens and the brokers' node tokens were
  minted once with a one-hour life. The harness now mints tokens on use,
  rewrites each broker's node token file and the session file every half hour.
  Cluster's client_token, admin_token, operator_token,
  subscribe_only_token and group_operator_token fields are now methods, and
  ControlPlane's token methods moved to felix_cluster::Credentials.
• A cache follower that dropped a divergent suffix of its cache log, or of its
  counter log, kept the in-memory index or counter sums built from the dropped
  records, so after a promotion a key could read the value of whatever record
  replaced its offset. Both are now rebuilt from the log as it is. (#933)

Documentation

• Leader mode's single writer is stated as resting on the lease: which
  faults let a deposed Leader leader acknowledge a write its successor never
  sees (a pause past the lease margins, a broker clock running slow against
  the control plane's), and that a clock step and Quorum with
  majority_ack are not exposed. A cluster test pins that a Leader stream
  refuses writes on a lapsed lease after majority_ack is finalized. (#1008)
• A publish acknowledged on enqueue (a Leader stream with ack_on_commit
  off, the default) is not readable until it is written, so a group poll,
  history read or replay sent right after the ack can miss it. Stated in the
  semantics and queues pages and on Client::group_poll. The end-to-end group
  tests in cache_durability now publish with commit acks; they assumed
  otherwise and failed intermittently. (#1025)
• Corrected four replication claims. Counter adds on a Quorum cache wait for
  a majority like puts and deletes; the log-backed cache stays Partial for its
  in-memory key index (#1073). A Quorum publish with no reachable majority
  is not refused up front: it waits, then fails with quorum_timeout. Under
  majority_ack a move's destination cut off from its followers at the
  cut-over waits for a majority instead of opening on the lease. A promoted
  leader's fence retries back off from 200 ms to 2 s. (#1074)