Felix v0.6.1

0.6.1 fixes a stall in Quorum publishes. A fresh 0.6.0 cluster finalizes
majority_ack, and under load a Quorum stream on three or more brokers could
then stall for up to the replication tick (2 s by default) before its publishes
were acknowledged (#1479). Upgrading is recommended for anyone running Quorum
on three or more brokers. The release also closes a placement race when a
stream's consistency changes (#1253), makes a forward refuse a stream recreated
under the same name (#1324), and brings back the small-batch fire-and-forget
throughput that 0.6.0 listed as a known issue (#1469).

Upgrading.

• Under a Raft control plane, the consistency-change fence (#1253) takes effect
  once every member runs 0.6.1 (metadata level 19). Until then a change of
  consistency fences nothing, as in 0.6.0.
• A Postgres control plane runs migration 0036 on start.
• The forward incarnation check (#1324) applies between brokers that both run
  0.6.1, which offer the FORWARD_INCARNATION peer capability. A forward to a
  0.6.0 broker is sent as before and checked by generation alone.

Added

• A write can name its shard instead of hashing a key (#1465). publish, publish_batch,
  publish_idempotent, publish_if, commit and state_get take an optional shard,
  negotiated as FEATURE_EXPLICIT_SHARD (bit 0x8 of the extended feature word). A shard the
  stream does not have is refused with invalid_request and nothing is written. The Rust client
  adds publish_on_shard and publish_batch_on_shard (Publisher, ClusterClient), and
  commit_on_shard, commit_if_on_shard, publish_if_on_shard and state_get_on_shard
  (Client, ClusterClient). ClusterClient sends each to the named shard's owner or leader.
  They fail without sending against a broker that did not advertise the bit. Caches and counters
  take no shard.
• Metrics for what a Quorum acknowledgement waits on (#1479):
  felix_broker_publish_quorum_wait_seconds, how long a publish waited for its
  majority, and felix_storage_replica_state_writes_total with
  felix_storage_replica_state_write_duration_seconds, the replica-state writes
  a follower makes for commit offsets.

Fixed

• Changing a stream's consistency or durability now fences placement passes
  that read the stream before the change (#1253). A pass that read a Quorum
  stream could otherwise replace its leader on its followers' word after the
  stream became Leader and the leader was serving on its lease, leaving two
  brokers serving. Postgres gets migration 0036; under Raft it applies once
  every member is at metadata level 19.
• A forwarded publish or cache op now names the incarnation of the stream or
  cache its route was for, and the owner refuses one for another incarnation
  (#1324). Before, a forward routed by a deleted stream's assignment could land
  in a stream recreated at the same generation, which a control-plane restore
  makes possible. Sent only to brokers that offer FORWARD_INCARNATION.
• A leader that crashed under Periodic or None no longer answers its
  successor's fence just because the routes now show a set of two (#1336). The
  set can shrink from three to two under the same leader, and a record the
  three acknowledged may be on neither copy. It answers only when the
  generation it led had those same two copies and its own last record is from
  that generation.
• A subscribe from latest on a durable stream whose log ends in generation-start
  records now reports start_offset and live_offset just past the last event,
  not at the raw tail (#1475). Those records are never delivered, so a client that
  read history up to live_offset - 1 after joining at latest waited forever,
  which on a 0.6.0 cluster is any Quorum shard with no write since its leader
  started. A client that negotiated FLAG_EVENT_BATCH_SKIPPED counts no drop at
  the join. A client with offsets but without it now reads one false drop per
  trailing generation start at a latest join, as it already did mid-stream.
  Count drops before the first event as offset - start_offset - skipped_before:
  start_offset can be 0.
• A Quorum stream on three or more brokers no longer stalls for up to the
  replication tick (2 s by default) under load (#1479). A follower's answer that
  arrived after the pass that asked for it moved no cursor, so no pass counted
  it, and with every publisher waiting on the mark nothing else started one.
  Such an answer now starts a pass, and a pass waiting for a majority counts a
  busy follower's answer itself instead of only giving up on it. The stall was
  most visible once majority_ack is finalized: 256 B durable publishes at 64
  in flight fell 38% with a p999 of 1.86 s.

Performance

• A replicated durable shard no longer writes its replica state on the
  acknowledgement path under fsync_mode = on_commit (#1456). The commit offset
  is written behind, and one write in flight carries every advance made while it
  runs, so a Quorum ack costs one data fsync per replica instead of about
  three. After a crash the commit offset can read back one write behind; the
  accepted generation is still on disk before a fence is answered.
• Fire-and-forget publishes of small batches under pub_ingress_wait are back to the throughput the
  0.6.0 performance runs measured (#1469). A control stream now parks at most 16 unacked publishes while it
  waits for queue room; past that it waits inline. With the acked cap of 1024, a busy broker's
  streams parked until the process-wide byte budget ran out, and every publish then waited on
  that budget: 256 B batch-64 fire-and-forget lost 13%.
  The trade-off: a fire-and-forget stream to one full shard stops reading after 16 frames
  instead of after the byte budget runs out, and with the default stream window it can then
  hold the connection's flow-control credit and stall the connection's other streams.

Felix v0.6.0

0.6.0, the build the release performance runs measured. The previews below
describe most of the series; this section lists what changed since
0.6.0-preview.9. A durable stream now acknowledges a publish only after it
commits, a new cluster finalizes its fleet features at its first broker
registration, ClusterClient publishes batches straight to each shard's
leader, and brokers forwarding Quorum publishes to each other no longer
deadlock their replication. The publish ingress path got a deeper default
queue, one wake per freed slot, and waits that no longer stop a control stream
from reading.

Upgrade notes.

• Acked publishes on durable streams now wait for the commit and cost a write's
  latency. FELIX_ACK_ON_COMMIT=0 restores acknowledging on queue.
• FELIX_BROKER_PUB_QUEUE_DEPTH defaults to 512 (was 64) and
  FELIX_PUB_SHARD_STREAMS to 4 (was 16).
• A Postgres control plane runs migration 0035 on start. Under Raft, the
  recreated-stream fix needs every member at metadata level 17 and the record
  of deleted names needs level 18.
• Creating a stream or cache under a deleted name returns 409 until
  stream_incarnations (or cache_incarnations) is finalized.
• On a new cluster, start every broker with the same FELIX_QUORUM_READS and
  FELIX_INTERNAL_FENCE.

Known issues.

• With pub_ingress_wait on, fire-and-forget batches of 256 B messages run about
  13% slower than on the build the performance runs measured (#1469). Larger
  messages and acked publishes measured the same as that build.

Security

• Updated rustls-webpki to 0.103.16 and webpki-roots and webpki-root-certs to 1.0.10 for RUSTSEC-2026-0382 and
  RUSTSEC-2026-0383, in the workspace and every out-of-workspace lockfile.

Added

• ClusterClient::offset_for_time looks up the offset for a time on one shard of a durable stream, asking whichever broker leads the shard and following NotLeader redirects, as ClusterClient::read does.
• ClusterClient publishes batches owner-direct: publish_batch, publish_batch_keyed, and
  publish_records/publish_records_at_least_once, which split records by shard and send each
  shard's ordered batch to its leader, reporting failures per shard in a BatchReport. Before, only
  single records were leader-direct, so batched publishing on a cluster was forwarded by a
  non-leader. felix-loadgen ingest --cluster uses it, and a loadgen publisher now survives a
  broker loss and reports reconnects, failed_sends and per-shard acked counts.
• ClusterClient now looks up shard leaders from the cluster (FEATURE_SHARD_OWNERS) on a stream's
  first publish, every few seconds, and after a publish finds one wrong, so the first batch and
  publishes that ask for no ack go leader-direct. Before, a leader was learned only from the ack of
  a forwarded batch, so fire-and-forget publishing was forwarded for as long as it ran. A cached
  leader learned under no stream width is no longer trusted for a keyed publish.

Changed

• Behaviour change: a durable stream now acknowledges a publish after it commits by default
  (#1421). Before, the broker answered when the publish was queued, so a broker crash or a lapsed
  lease could lose a record the client had been told was stored. Committed means written to the
  stream's log under FELIX_DURABLE_FSYNC_MODE: with the default periodic mode the bytes are in
  the log file (they survive a broker crash, and a machine crash can lose the last 250 ms), and
  on_commit flushes to the device first. In-memory streams still answer when the publish is
  queued. Acks now carry their offset on durable streams, and acked publishes cost a write's
  latency. FELIX_ACK_ON_COMMIT=0 (or ack_on_commit: false) restores the old behaviour for every
  stream; FELIX_ACK_ON_COMMIT=1 makes in-memory streams wait too. The wire is unchanged, so a
  client that predates commit-ack negotiation works as before.
• FELIX_ACK_ON_COMMIT now reads 1/true/yes/on and 0/false/no/off (trimmed, any case). Any other
  value keeps the default and logs a warning instead of turning commit acks off. A stream with more
  than 1024 answers owed now refuses an acked publish before queueing it, so a retry cannot write
  it twice.
• The control plane refuses with 409 to create a stream or cache under a name that was deleted on
  the cluster until stream_incarnations (or cache_incarnations) is finalized, and the message
  says to finalize first (#1447). Without an incarnation, a stream created again opened the
  deleted stream's logs on every broker, so a recreated Quorum stream's followers held old
  committed records that the new leader's writes disagreed with, and they halted as diverged.
  Every store records deleted names (Postgres migration 0035; on Raft once every member is at
  metadata level 18), and they are part of the exported state.
• A new cluster finalizes stream_incarnations and cache_incarnations at its first broker
  registration, so a stream or cache deleted and created again there always starts empty on every
  broker (#1447). The control plane treats a cluster as new when no stream or cache was ever
  created and no broker has registered twice, and finalizes through the same checks as an
  operator's finalize, so a broker without the features keeps them off or is refused afterwards.
  A cluster upgraded from an older release still waits for the operator to finalize.
• The one-command compose.yaml stack now runs its broker with FELIX_DURABLE_STORAGE_DIR, so
  orders is a durable stream, consumer groups work, and cache watches work. It also creates the
  caches config and stats, and its development tokens can use caches, counters and groups.
  CI exercises these commands through the stack.
• A new cluster finalizes every fleet feature its brokers report at its first
  broker registration, not only the incarnations: generation_start,
  majority_ack, lease_free_reads, fenced_caches, jump_hash_routing,
  atomic_commit, group_cursor_dependencies, stream_incarnations and
  cache_incarnations, in that order. A fresh RF=3 cluster used to acknowledge
  Quorum writes on the control plane's replica report until an operator
  finalized majority_ack; it now acknowledges on its followers from the
  first stream. publisher_principal stays an operator step, since it grows
  every record by its publisher's id. Each finalize goes through the
  operator's checks, so a feature some broker does not report stays off with
  the features that need it, and under the Raft backend nothing is finalized
  until every member runs this release; a finalize held back that way is
  retried in the background for up to ten minutes. Start every broker with the same
  FELIX_QUORUM_READS and FELIX_INTERNAL_FENCE: a broker that lacks a
  feature the first one finalized is refused, and the refusal now names
  FELIX_INTERNAL_FENCE as well as FELIX_QUORUM_READS. An upgraded cluster
  still waits for the operator.

Fixed

• Placement now spreads each stream's and cache's leaders evenly across brokers, not only the
  cluster-wide total (#1429). A 12-shard stream on three brokers could be placed 0/0/12 beside
  other streams that already filled two brokers to their share, and rebalancing left it at 1/1/10
  because the totals were even, so the stream's writes ran on one broker. New shards now go to
  the broker leading fewest of their stream first, a broker leading two or more of a stream's
  shards more than another hands one over, and a move for the cluster-wide balance never makes a
  stream's spread wider. The totals are now evened to within one, where before a broker was only
  moved off when it led more than its rounded-up share. Existing clusters even out after the upgrade, one move at a time under
  the usual move limits.
• finish() and broker connects no longer hang a client forever when a broker stalls (#1433).
  Publisher::finish and ClusterClient::finish give up after finish_timeout
  (FELIX_FINISH_TIMEOUT_MS, default 60 s) and abort the streams, and each connect, including a
  cluster reconnect, fails after connect_timeout (FELIX_CONNECT_TIMEOUT_MS, default 10 s).
  A cluster reconnect now dials outside the client lock and within three times connect_timeout,
  ClusterClient::finish finishes brokers concurrently, and the topology, shard-count and subscribe
  exchanges give up after connect_timeout.
• A fenced shard move whose leader is live but never reports drained no longer holds the shard
  without writes forever (#1434). After FELIX_SHARD_FENCE_TIMEOUT_MS (two minutes by default,
  shard_fence_timeout_ms in the config file) placement gives the move up the way an operator's
  cancel does: the leader serves again at a new generation, holding every write it accepted, and
  the move counts in felix_shard_moves_timed_out_total. The shard's next move waits the
  timeout, doubling with each give-up in a row up to an hour, so a leader that cannot drain does
  not cut its shard's writes every few minutes; repeated give-ups log a warning. 0 keeps the old
  behaviour.
• A stream deleted and created again under the same name without incarnations no longer leaves a
  zombie leader behind (#1435). The delete now puts an unassigned change for each shard on the
  shard change log, so a broker leading one lets it go. The recreated stream's shards start at the
  change seq instead of generation 0, because brokers keep an unincarnated stream's logs and the
  generations those logs accepted: a broker that had taken a shard by a move refused to lead or
  follow the new shard at generation 0. Under Raft both need every member at metadata level 17. A
  broker also takes a shard's first assignment over whatever it still holds, which covers control
  planes that don't record the unassignments yet.
• Brokers forwarding durable Quorum publishes to each other no longer deadlock their
  replication (#1458). Replic...

Nightly 0.6.0-nightly.20261024+c5c2ad9

Nightly build 0.6.0-nightly.20261024+c5c2ad9 of c5c2ad9, built 2026-10-24.

Built from the newest commit on main whose CI run passed, then smoke-tested. That is all the testing it has had: do not run it in production.

docker pull ghcr.io/getfelix/felix-broker:nightly-20261024
docker pull ghcr.io/getfelix/felix-controlplane:nightly-20261024
docker pull ghcr.io/getfelix/felixctl:nightly-20261024

The binaries report the version of the last release, not this build's.

Unreleased changes

Added

• A write can name its shard instead of hashing a key (#1465). publish, publish_batch,
  publish_idempotent, publish_if, commit and state_get take an optional shard,
  negotiated as FEATURE_EXPLICIT_SHARD (bit 0x8 of the extended feature word). A shard the
  stream does not have is refused with invalid_request and nothing is written. The Rust client
  adds publish_on_shard and publish_batch_on_shard (Publisher, ClusterClient), and
  commit_on_shard, commit_if_on_shard, publish_if_on_shard and state_get_on_shard
  (Client, ClusterClient). ClusterClient sends each to the named shard's owner or leader.
  They fail without sending against a broker that did not advertise the bit. Caches and counters
  take no shard.

Fixed

• A subscribe from latest on a durable stream whose log ends in generation-start
  records now reports start_offset and live_offset just past the last event,
  not at the raw tail (#1475). Those records are never delivered, so a client that
  read history up to live_offset - 1 after joining at latest waited forever,
  which on a 0.6.0 cluster is any Quorum shard with no write since its leader
  started. A client that negotiated FLAG_EVENT_BATCH_SKIPPED counts no drop at
  the join. A client with offsets but without it now reads one false drop per
  trailing generation start at a latest join, as it already did mid-stream.
  Count drops before the first event as offset - start_offset - skipped_before:
  start_offset can be 0.

Performance

• Fire-and-forget publishes of small batches under pub_ingress_wait are back to their v0.6.0
  campaign throughput (#1469). A control stream now parks at most 16 unacked publishes while it
  waits for queue room; past that it waits inline. With the acked cap of 1024, a busy broker's
  streams parked until the process-wide byte budget ran out, and every publish then waited on
  that budget: 256 B batch-64 fire-and-forget lost 13%.
  The trade-off: a fire-and-forget stream to one full shard stops reading after 16 frames
  instead of after the byte budget runs out, and with the default stream window it can then
  hold the connection's flow-control credit and stall the connection's other streams.

Felix v0.6.0-preview.9

The ninth preview of 0.6.0, mostly about what a crash does to a broker's
storage identity. A process crash keeps the storage id and only a machine crash
rotates it, a volume restored from before the node's last registration rotates
it, a wiped node can no longer vote for a deposed leader, and a fence counts an
answer only from the copy the failover's report named. docker compose up now
starts a local stack, and the control plane no longer serves a Swagger UI.

Upgrade notes.

• A Raft control plane writes a fence answer's storage identity only once every
  member is at metadata level 16. Upgrade every member first.
• A Postgres control plane runs migration 0034 on start.
• A broker now writes <data dir>/registration and, after a start under a
  relaxed fsync mode, run-state version 2. A downgrade after a crash rotates
  the storage id.
• /docs (Swagger UI) on the control plane is gone; use /v1/openapi.json.

Added

• compose.yaml at the repository root runs a control plane, a broker and
  development credentials from the released images, so docker compose up (or
  podman compose up) is enough to publish and subscribe with felixctl
  (#1345). The credentials are dev-only and every port is bound to loopback. CI
  does a round trip through the stack on every change.

Changed

• On Linux, under periodic or none fsync, a broker whose process crashed (SIGKILL,
  an OOM kill, a panic) keeps its storage id, so its followers' copies and
  its own leadership survive the restart without a re-copy (#1367). Only a
  machine crash rotates it: run-state now records the machine's boot id
  (/proc/sys/kernel/random/boot_id), and a start in a different boot, or where either boot is unknown,
  takes a new id as before. Keeping the id first calls syncfs on every
  filesystem under the data directory and rotates if that fails. macOS and
  other platforms still rotate after every relaxed crash. A failed fsync forgets the boot, so a crash after one still
  rotates. A VM or container memory-snapshot restore keeps the boot id; use
  on_commit or delete storage-id for those workflows. A clean stop is written in the layout older releases
  read, so a downgrade after one keeps the id.
• A release now asks getfelix.dev to show it as soon as it is published, through a
  release-published dispatch that needs the new SITE_DISPATCH_TOKEN secret.

Removed

• The control plane no longer serves a Swagger UI at /docs. /v1/openapi.json is unchanged; point any OpenAPI viewer at it (docs: Viewing the API document). This removes the zip build dependency behind RUSTSEC-2026-0367 and RUSTSEC-2026-0368 and the Swagger UI download at build time.

Fixed

• A replica holding no fence promise on disk, such as one on a wiped data
  directory, no longer answers a fence for a shard its routing view does not have
  yet, or a leader other than the one the view names at that generation (#1267).
  A wipe took the promises it had given, so a deposed leader could count it toward
  a majority its successor had already fenced, and acknowledge a write the
  successor never held. A new shard's first fence may now wait for a follower's
  assignment watch.
• A broker whose data directory was restored from a volume snapshot or copied in
  no longer keeps its storage id when the node has registered since the copy
  was taken, under any fsync mode (#1334). The broker records each
  registration's node incarnation in <data dir>/registration, and a cluster
  member asks the control plane for its current incarnation before any store
  opens: when the control plane is ahead, it writes a new storage id, so peers
  and placement stop counting the copy on positions it no longer holds. A
  cluster member with durable storage now waits for the control plane to
  answer before it opens storage. A copy taken during the node's current
  registration is not detected, so the backup runbook still deletes
  storage-id for one.
• A promoted leader's fence no longer counts a replica that came back under a
  new storage identity (#1335). Under Periodic or None fsync, a follower that
  crashed lost records it had already answered for. If the leader was then
  replaced, the fence could count that short copy toward its majority, and a
  record acknowledged on the old leader and that follower was lost. A failover's
  assignment now carries the storage identity the replica report named for each
  replica (fence_storage_ids). The fence counts an answer only when it comes
  over a connection whose handshake gave that identity. Any other answer still
  takes the promise, but counts for nothing, and the fence waits for a copy that
  can vouch for what it holds. Under Raft the field waits for metadata level 16.
  A member of the old set the report could not vouch for is named with an empty
  identity and does not count. A replica outside the old set, or one from a
  build that predates identities, counts as before. With the old leader
  gone for good and no majority that counts, the shard waits. The way out is
  to stop the promoted broker and run felixctl placement abandon. That keeps
  every copy in the new replica set and takes the furthest of them.
  Under Periodic, the default, this includes a leader and a follower whose
  machines go down together. The follower comes back under a new identity even
  when nothing was lost, so the shard no longer serves when it returns. It
  waits for the operator instead. A process killed on Linux keeps its identity
  in the same boot (#1367), so its answer still counts.

Documentation

• The TLA+ model now checks wipes, follower crashes and leader crashes on the
  fenced, follower-acked path the broker ships, not only on the report-gated
  one (#1333). Three new configurations pass (FelixShardFencedWipe,
  FelixShardFencedCrash, FelixShardFencedLeaderCrash), and each has a twin
  that drops one fix and finds the lost record. FelixShardFencedCrash is the
  first passing check of a follower crash with a rotated storage identity.

Felix v0.6.0-preview.8

The eighth preview of 0.6.0. A broker now has a storage identity that its node
registers, and placement counts a follower toward promotion only while the
identity its leader reported matches, so a follower that lost its disk is
never promoted on an old report. Streams get incarnations, and caches get them
too, so a stream or cache deleted and created again under the same name starts
empty on every broker; both are behind finalized features. A leader that
crashed under FsyncMode::Periodic or None no longer reopens its old
generation with a log that may be short of what it acknowledged (#1294,
#1305). The single-publisher fanout path is back to preview.7's throughput,
with large-payload fanout faster (#1309).

Upgrade notes.

• A Raft control plane needs every member upgraded before these take effect,
  each at its own metadata level. Until then the proposer leaves the new data
  out, the same on every member:
  • Level 9: an auth change and its revocation, and a refresh rotation, are
    single entries. Below it they are written as the two older entries they
    combine.
  • Level 10: stream incarnations (an upgraded member refuses the
    stream_incarnations finalize with 409), and deletes that take shard
    assignments and replica reports.
  • Level 11: a replica report's by_round. Below it, leaders are failed over
    on their lease.
  • Level 12: storage identity. Below it, followers read as unverified.
  • Level 13: a report's leader and node_incarnations. Below it, placement
    reads reports as at level 12.
  • Level 14: cache incarnations (an upgraded member refuses the
    cache_incarnations finalize with 409). Below it, caches are created at
    incarnation 0 and no cache delete records a tombstone.
  • Level 15: a report's given_up. Below it, placement replaces a leader that
    gave a shard up only on its storage identity.
• A Postgres control plane runs migrations 0027 through 0033 on start.
• stream_incarnations and cache_incarnations are finalized by an operator
  after the upgrade: every control-plane member at level 10 and 14, and every
  broker on this release, first. With Postgres, upgrade every instance before
  finalizing, since an older instance still running after the finalize cannot
  create streams or caches. See the upgrades page.
• A cluster that ran an unreleased build at metadata level 10 to 13 and logged
  a stream create that names no incarnation, or its delete, replays it
  differently now and must rebuild those members from a snapshot (#1287).

Security

• The Python and TypeScript SDKs read their CA file with rustls-pki-types
  instead of the unmaintained rustls-pemfile (RUSTSEC-2025-0134), and
  deny.toml no longer ignores that advisory (#1255).
• A revoked gateway can no longer delegate with a felix-controlplane token
  it got before the revocation (#1205). /token/delegate checked the
  revocation cutoffs against the user's token only, and trusted
  token.delegate from the caller's token, which stays valid for its TTL. Now
  /token/delegate and actor_token on /token/exchange refuse, with
  actor_refused, a gateway whose token predates a revocation of it or that
  current RBAC no longer grants token.delegate. A refresh of a user chain
  bound to a gateway is refused, and the chain ended, once RBAC no longer
  grants the gateway token.delegate.
• Revocation cutoffs are read on the Raft leader when minting or delegating,
  so a member that has not applied a revocation yet no longer mints past it.
• Deleting an IdP issuer, or narrowing or replacing what it trusts (an
  audience dropped, the subject or groups claim changed, or jwks_url or
  discovery_url changed), revokes the tenant's sessions, as removing a rule
  that reached an IdP group does. Adding an audience revokes nobody.

Added

• felix_client::Seed and ClusterClient::connect_seeds (#1227): a cluster
  client's seeds can be host:port names as well as addresses. connect and
  connect_with_policy still take SocketAddrs and are unchanged.
• Brokers export felix_revocation_feed_last_success_seconds (set when they
  start following the feed, then at each good poll) and
  felix_revocation_feed_poll_errors_total, and log a warning, repeated every
  minute, while they cannot poll the revocation feed. docs/auth.md now states
  how revocation fails open meanwhile, and the clock-skew bound on revocation
  cutoffs across control-plane members.
• Replay rings and cache watch queues are bounded by bytes as well as entries
  (#1201). FELIX_REPLAY_RING_MAX_BYTES (64 MiB per stream shard) drops a
  ring's oldest records first; FELIX_CACHE_WATCH_MAX_BYTES (64 MiB per watch)
  ends a watch that would go over as lagged, as a full queue does. Each has an
  optional broker-wide _TOTAL. New gauges: felix_replay_ring_bytes,
  felix_cache_watch_queue_bytes.
• Brokers exchange storage identities in the peer handshake, behind a new
  capability bit, STORAGE_IDENTITY (1 << 10), so an older peer sees the
  frames it always did (#1266). A leader asks for each follower's identity at
  the start of every exchange and starts the follower's cursors over when it
  changes, so a follower that lost its disk is no longer reported caught up
  on an idle shard. Replica reports carry the identity each follower was
  counted under as replica_storage_ids, omitted when empty. Placement does
  not read it yet.
• Each broker data directory now has a storage identity (#1266): 16 random
  bytes kept in <FELIX_DURABLE_STORAGE_DIR>/storage-id, created on the first
  start, kept across restarts, and replaced when the file is missing or does
  not decode. A broker sends it as storage_id with its registration; a
  broker without durable storage sends a new one each start.
  felix-broker restore-point writes a new one.
• Changelog entries are now collected from changelog.d/ fragments, one file per
  change, so pull requests no longer conflict on CHANGELOG.md.
  scripts/changelog_assemble.py folds them into the release section, and the
  release workflow refuses a tag while fragments remain. Entries written directly
  in CHANGELOG.md still work. See changelog.d/README.md.

Changed

• A subscription's feeder takes room in its writer-lane share without a lock
  or a wakeup registration, and a written frame notifies only when the feeder
  is waiting. The byte gauges (felix_sub_egress_bytes,
  felix_sub_queue_bytes, felix_cache_watch_queue_bytes,
  felix_replay_ring_bytes) are looked up once per subscription or ring
  instead of on every batch, and a replay ring reports once per publish.
• YAML parser: serde_yaml_ng replaced by serde-saphyr (#1211). The broker,
  control plane and felix-client read their config files with
  serde-saphyr 1.3, a pure-Rust parser. serde_yaml_ng pulled in
  unsafe-libyaml, which is unmaintained (RUSTSEC-2024-0449). felix-client
  users get the new dependency in place of serde_yaml_ng. A config file that
  loaded before loads to the same values, and the 1.1 spellings that were
  refused still are: yes/no/on/off for a boolean, 0777, a << merge
  key, a duplicate key, an unknown key. Some inputs behave differently:
  • Now accepted: a quoted number or boolean (max_frame_bytes: "1024",
    ack_on_commit: "true"), 1_000, a file that is only ~ or null, and
    a tab before the first key.
  • Now refused: an explicit tag that contradicts the setting's type
    (!!str 5 for a number, !!int "5" for a string); a tag the parser does
    not know (!custom x, which used to read as x); and a bare or
    null tls.cert_path/key_path in the control plane file (cert_path: or
    cert_path: ~), which used to be read as the path "" or ~. A quoted
    cert_path: '' is still accepted, as before. !!binary on a string
    setting is still kept as written, not decoded.
  • Parse errors name the key and give a line and column. A wrong-typed
    value is no longer quoted, since it could be a token; an unknown enum
    value still is.
  • --print-config puts double quotes, not single quotes, around strings
    that need them, and also quotes yes and 1_000. The output still reads
    back to the same values.
• An auth change that takes rights back (an RBAC rule removed, an issuer
  deleted or narrowed) and its revocation are one store write: one Raft entry
  (metadata level 9) or one Postgres transaction. A failed request changed
  nothing and is safe to retry. Below level 9 the Raft proposer writes them
  separately, as before.
• /token/refresh does its store reads before spending the refresh token, and
  spends it in the same write that stores the replacement (Raft level 9), so a
  failed refresh no longer turns the client's retry into a replay that ends
  the chain.
• /token/delegate reads the subject's and the caller's revocation cutoffs in
  one leader read.
• A spent refresh token presented again with the right secret within
  FELIX_REFRESH_REUSE_GRACE_SECONDS (30 by default; 0 turns it off) of its
  rotation, while its replacement is unused, gets a working token instead of
  revoking the chain. This covers a refresh whose answer was lost and two
  concurrent refreshes. Replay detection is blind for that window. Postgres
  migration 0027_refresh_token_rotation; under Raft, metadata level 9.
• Deleting an IdP issuer the tenant does not have answers 404 and revokes
  nobody.

Upgrade note. A Raft control plane needs every member at metadata level 9
before auth changes and their revocations, and refresh rotations, are single
entries.

• FELIX_SUB_QUEUE_MAX_BYTES now counts only what waits in a subscriber's
  broker queue, and its default drops from 256 MiB to 128 MiB (#1195). Frames in
  the writer lanes count against a new per-connection bound,
  FELIX_SUB_EGRESS_MAX_BYTES (256 MiB), shared out equally among the
  connection's subscriptions, so one whose stream...

Felix v0.6.0-preview.7

The seventh preview of 0.6.0. Subscriber memory on the broker is now bounded
by bytes end to end: each subscriber queue, and the writer lanes behind it,
stop at FELIX_SUB_QUEUE_MAX_BYTES (256 MiB by default) as well as at their
entry count. The bound is sized so a subscriber keeping up at line rate with
4 MiB batches never reaches it. Under the block policy, one full subscriber
no longer holds up a batch's fanout to the others. Revoking a principal now
ends the streams its tokens already opened, with the new revoked error code,
and a subscription whose stream is deleted ends with stream_deleted for
clients that offer it. ClusterClient::with_identity lets a gateway act as
another user over shared cluster connections, and a delegating gateway can
hold a connection-only broker token instead of a grant it never uses.
Replication and placement fixes: group and dead-letter logs ride each
follower's exchange, a new Quorum copy is seated only once the leader names
it caught up, and a counters-only cache restore is no longer dropped as
stalled.

Upgrade notes.

• A Raft control plane needs every member upgraded before revocation (metadata
  level 7) and counter-offset reports (level 8) take effect. Until then the
  revoke endpoint answers 409 after revoking the refresh tokens, and reports
  are recorded without counter offsets.
• A Postgres control plane runs migrations 0025 and 0026 on start.

Added

• Subscriber queues on the broker are bounded by bytes as well as entries.
  An entry is a whole published batch, so the count alone let one slow
  subscriber hold up to 4096 batches of up to 16 MiB each. Each queue now also
  stops at FELIX_SUB_QUEUE_MAX_BYTES of payload (256 MiB by default, 0 for
  none), and the queue policy handles a full byte bound as it handles a full
  count: drop_new drops the batch for that subscriber, with the same offset
  gap and drop counters, and block waits. A batch larger than the bound
  still goes into an empty queue. FELIX_SUB_QUEUE_MAX_BYTES_TOTAL (off by
  default) caps the bytes held across all subscriber queues, at the cost of
  isolation between subscribers. New metrics: felix_sub_queue_bytes and
  felix_sub_queue_byte_dropped_total{bound}. Broker gains
  with_subscriber_queue_max_bytes and with_subscriber_queue_max_bytes_total
  (#1157, #1189).
• A delegating gateway no longer needs a broker grant it never uses. A
  principal holding token.delegate:tenant:{tenant_id} can exchange for a
  connection-only broker token, its own subject with no permissions, by
  sending "requested": [], "permissions": [] to /token/exchange (or the
  dev-token bootstrap endpoint). Its shared connections authenticate with it,
  and every request still runs under the user's delegated token. Refresh
  checks token.delegate again and ends the chain once it is removed. Anyone
  else asking for no permissions still gets 403 no permissions (#1130).
• ClusterClient::with_identity(tenant_id, token_provider) and
  ClusterClient::with_identity_token act as another user over the cluster
  client's connections (#1129), as Client::with_identity does for one
  broker. The result is a ClusterClient with its own routing state that
  follows moved shards, redirects and failovers, reaching each broker over the
  parent's connection to it, so a gateway sharing connections keeps a user's
  subscription across a shard move and still holds one connection per broker.
  The Python Client.with_identity and AsyncClient.with_identity and the
  TypeScript Client.withIdentity wrap it with a fixed token. No wire change.
• Revoking a principal ends what its tokens already opened, on every broker,
  within about one control-plane sync interval (#1132). Until now a revoked
  principal kept its open subscriptions, cache watches, publish streams and
  Kafka connections until its token expired. POST /v1/tenants/{t}/refresh-tokens/revoke now also records a revocation, and so
  does removing an RBAC policy or role assignment, for everyone the rule
  reached (the whole tenant when it reached an IdP group). Brokers poll the new
  GET /v1/principal-revocations/changes feed, refuse broker tokens issued at
  or before a revocation, and end the streams they opened with the new
  revoked error code (16, fatal): an error on a control stream, or as the
  last frame on a subscription's or cache watch's event stream. A Kafka
  connection is closed at its next request. Under the Raft store this is
  metadata version 7, and until every member has it the revoke endpoint
  answers 409 after revoking the refresh tokens. See
  docs/auth.md.
• CacheWatch::refusal in the Rust client says why the broker ended a watch
  with an error, such as revoked. A subscription ended that way returns the
  BrokerError from next_event instead of an "unexpected message" error
  (#1132).
• The subscriber byte bounds also cover the broker's writer lanes. A batch's
  bytes stay held until the frame carrying it is written to the subscriber's
  QUIC stream, so frames queued behind a flow-controlled subscription count
  toward FELIX_SUB_QUEUE_MAX_BYTES instead of piling up in the lanes by entry
  count alone (up to 64 frames of up to 16 MiB each per subscription). Under
  drop_new the next batch is dropped at the subscriber queue as before;
  publishers still never wait. SubscriptionReceiver gains recv_held and
  try_recv_held, and HeldBytes is public (#1163).
• A subscription ended because its stream was deleted now says so. The broker
  sends stream_deleted as its last frame to a client that offered the new
  extended feature bit FEATURE_STREAM_DELETED; any other client sees the
  stream end as before. Removing a stream used to leave its subscribers
  waiting on a feed nothing would end. The Rust client reports it as
  Subscription::stream_deleted and ClusterSubscription::stream_deleted,
  and a sharded subscription yields ShardEvent::StreamDeleted for each shard
  instead of trying to resubscribe it (StreamDeleted in Python,
  streamDeleted in TypeScript) (#1119).

Changed

• A broker closing a shard it no longer has any role in tells a reader still
  attached that the shard moved, naming its current leader, rather than just
  ending the stream. Broker::close_stream_shard takes the handoff (#1119).

Fixed

• Under the block queue policy, one subscriber at its byte or entry bound no
  longer holds up the rest of a batch's fanout. The broker queues the batch
  for every subscriber with room first and waits only on the full ones, so a
  fast subscriber gets each batch as soon as it is durable. Each subscriber
  still receives batches in publish order. The latency demo now passes
  FELIX_SUB_QUEUE_MAX_BYTES and FELIX_SUB_QUEUE_MAX_BYTES_TOTAL to the
  broker it builds (#1179).
• The Node client returns null, as index.d.ts says, for an absent field on
  a delivered item: laggedResumeFrom, change and shardMoved on a cache
  watch item, value on a cache change, offset on an event, event,
  lostError and recovered on a sharded item, and resumeFrom, nodeId and
  addr on a shard move. They came back undefined, so a caller testing
  !== null read every cache watch item as a lag notice (#1140).
• The cluster test harness's wait_for_replication now waits until the
  control plane holds a report naming a follower at the leader's current tail.
  It used to return once the leader had shipped anything, which the harness's
  start-up publish already guaranteed, so a test that killed the leader next
  could fail over without the records it had just published (#1136).
• Right after a broker died, a stream with consumer groups could hold each
  replication pass on it for a request timeout or a dial: the group cursor and
  dead-letter logs shipped to every follower at the end of the pass, and the
  next publish waited behind that pass. They now ride each follower's
  exchange, as a Quorum cache's counters do since #1137, and so do a
  non-Quorum cache's counters (#1143).
• A new copy of a Quorum cache is no longer seated before it holds the
  cache's counter log. Seating checked only the cache log's offsets, so the
  set could count toward its majority a copy that promotion would refuse.
  A seat on a Quorum stream or cache now also waits until the leader's
  report names the copy caught up (#1139).
• A restore of a cache used only for counters is no longer dropped as stalled
  while it copies a large counter log. The stall window
  (FELIX_SHARD_RESTORE_STALL_MS) watched only the cache log's offset, which
  stays at 0 for such a cache. Replica reports now carry each replica's
  counter-log position (replica_counter_offsets, omitted for streams and by
  older brokers), and placement counts a move there as progress. The control
  plane stores it in the report (Postgres migration
  0026_replica_report_counter_offsets, Raft metadata level 8; below level 8
  reports are recorded without it) (#1164).

Felix v0.6.0-preview.6

The sixth preview of 0.6.0, fixes to shard placement, broker readiness and the
control plane. A cluster started all at once now places its shards on all its
brokers: while its first broker registered less than
FELIX_PLACEMENT_SETTLE_MS ago (10 s by default), a new shard waits for as
many live brokers as its factor, and a restore that only tops up a nearly
empty shard no longer takes the move slot. A shard placed while its leader was
the only live broker now gets its other copies once more brokers join. A
broker fetches every tenant's signing keys before it reports ready, so after a
restart it no longer refuses forwarded writes, such as cache puts, while the
control plane is down. A stopped control-plane Raft member no longer claims
leadership, and a second SIGTERM sent just as the control plane starts
draining is no longer missed. The docs now give failover timings per case with
the default settings.

Fixed

• A cluster started all at once now places its new shards on all the brokers
  it starts with. Placement ran as soon as the first one or two brokers had
  registered, so each shard got fewer copies than its replication factor, and
  restores added the rest one shard at a time behind the single move slot:
  eleven shards spent about 90 s short, and stopping a broker then could leave
  some without a majority. While the cluster's first broker registered less
  than FELIX_PLACEMENT_SETTLE_MS ago (10 s by default), a new shard waits for
  as many live brokers as its factor, and a broker registering runs placement
  at once. A restore that only tops up a short shard whose leader holds at
  most FELIX_SHARD_MOVE_FENCE_MAX_LAG_RECORDS records no longer takes a move
  slot, so a cluster whose shards did go short fills them together. The control
  plane logs when shards go short of their factor and when they are full again
  (#1151).
• A second SIGTERM sent to the control plane just after its readiness flipped
  to draining could be missed, so the pre-drain hold-off ran in full. The
  listener for it is now installed before readiness changes.
• A shard placed while its leader was the only live broker now gets the
  copies its replication factor asks for once other brokers join. Placement
  adds a copy only after the leader has reported at the shard's generation,
  and a leader with no followers never reported, so the shard stayed at one
  copy with under_replicated: true and nothing logged. A leader with no
  followers now reports itself, after its promotion fence as before (#1153).
• A broker fetches the JWKS of every tenant in its catalog when its first
  catalog sync lands, before it reports ready. It used to fetch them only when
  a client of that tenant connected, so a broker that restarted and then lost
  the control plane refused every write a peer forwarded to it, such as a
  cache put sent to a broker that does not own the key, until the control
  plane came back (#1148).
• A control-plane Raft member that had shut down still reported itself as
  leader and answered leader-only requests (heartbeats, expiry) from its last
  term's view. It now reports no leadership and refuses them with the same
  unavailable answer an instance without a leader gives, so callers move on
  to a live member (#1144).

Documentation

• The failover docs said a lost leader is replaced in about a second, which is
  the cluster harness's 1 s expiry window. They now give each case with the
  defaults: a crashed Leader-stream or cache leader is marked down 18.75 s
  after its last heartbeat and replaced on the next placement pass, a SIGTERM
  hands leaderships off as planned moves, and a lease-free Quorum stream fails
  over about 5 s after its leader goes quiet (#1152).

Felix v0.6.0-preview.5

The fifth preview of 0.6.0, mostly fixes to shard restores and replication.
A restore that could not finish no longer holds the only move slot for the
30-minute move timeout: one whose new copy stops moving for
FELIX_SHARD_RESTORE_STALL_MS (default two minutes) is dropped, and dropped
shards take turns at the slot. A restore of an empty shard is now seated, so a
cluster whose brokers registered a moment apart no longer stays on two of
three brokers. A Quorum cache's writes no longer stall for seconds after one
follower dies. Token delegation now takes only tokens minted for the
delegating gateway: token exchange accepts an RFC 8693 actor_token, and
gateways must send it on the exchanges whose tokens they delegate
(FELIX_CONTROLPLANE_DELEGATE_UNBOUND_TOKENS=true keeps the old behaviour
during an upgrade).

Changed

• POST /v1/tenants/{tenant_id}/token/exchange takes an optional RFC 8693
  actor_token (with actor_token_type): a gateway's own felix-controlplane
  token holding token.delegate on the tenant. The broker token it mints, and
  every token refreshed from it, carries may_act: {"sub": "<gateway>"}. The
  refresh record keeps the actor as a Raft metadata version 6 field, so under
  the Raft store an exchange with actor_token is a 409 until every member
  is upgraded.
• An exchange whose actor_token does not verify for the tenant or lacks
  token.delegate answers 403 with code actor_refused instead of
  forbidden, and counts as reason="actor_refused" in
  felix_controlplane_auth_rejected_total. A gateway can now tell its own
  token being refused from a user without permissions, which stays forbidden.
• POST /v1/tenants/{tenant_id}/token/delegate refuses a subject token whose
  may_act does not name the caller. A token with no may_act is refused too
  unless FELIX_CONTROLPLANE_DELEGATE_UNBOUND_TOKENS=true. Gateways must send
  actor_token on the exchanges whose tokens they delegate.

Fixed

• A restore of a shard whose log was empty, such as a cache used only for
  counters or one nothing had written to yet, was never seated: with nothing
  to ship, the leader never reported the new copy's position. It held the
  cluster's move slot until the 30-minute move timeout, so every other restore
  waited behind it. A cluster whose brokers registered a moment apart placed
  its shards on two of three brokers and stayed that way, and a Quorum cache
  on such a pair refused every write once either broker died (#1133). A
  follower of an empty log is now reported level at offset zero.
• A Quorum cache's writes stalled for seconds after one of its followers
  died, even with a majority alive: the leader shipped the counter log to
  every follower and waited for all of them before shipping the cache log,
  and again at the end of each pass. The counters now ride each follower's
  exchange, so a follower that does not answer holds neither mark (#1137).
• One restore that could not finish held the only move slot for the whole
  30-minute move timeout, so every other shard short of its replication
  factor stayed short (#1138). A restore whose new copy's reported position
  has not moved for FELIX_SHARD_RESTORE_STALL_MS (shard_restore_stall_ms,
  default two minutes, 0 waits for the move timeout) is now dropped as
  stalled and its slot goes to the next shard. Shards whose restores were
  dropped take turns at the slot, oldest attempt first, instead of the first in
  key order retaking it every time.
• Any holder of token.delegate could delegate any broker token of the tenant
  it got hold of, a stolen one included, and present it over its own
  certificate. Delegation now takes only tokens minted for the caller.

Documentation

• The shared-connection docs said each user's limits kept one user at a limit
  from blocking the others. The users share a connection ceiling of four
  users' worth by default, so four at their limits can starve a fifth; the
  docs now say so, and a test covers the per-user publish byte budget and the
  ceiling over QUIC.
• docs/security/rbac.md says what token.delegate lets its holder do.

Felix v0.6.0-preview.4

The fourth preview of 0.6.0, mostly fixes. A Quorum stream with a tight
retention bound could acknowledge a publish whose records no broker still
held; retention now stops at the commit offset on every replica, and that hold
survives a restart. A promoted cache or counter shard whose log fails to open
no longer opens unfenced. A consumer group's cursor no longer moves backwards
after a cancelled commit, and a late nack can no longer take a record from the
next consumer. An offload outage no longer fails durable logs at open, and
offload keys now start with the node id, so two brokers sharing a directory
cannot overwrite each other's copies. A shared connection's subscription and
publish limits now apply per identity, and a gateway can trade a user's token
for a delegated one carrying act, which subject binding accepts. A durable
Quorum stream with majority_ack and lease_free_reads finalized fails
over on its followers' word in about 5 s instead of waiting out the lease. A
subscriber that registers just as its shard moves is refused rather than left
attached to nothing, and a publish forwarded to a shard its new owner is still
fencing is held rather than refused. felix-broker inspect segments checks a
data directory offline and reports what startup would do. The docs site covers
felix-gateway and the projects built on Felix. The Rust API has breaking
changes, listed under Changed. pip will not install it without --pre.

Added

• Failover on the followers' word (part of #1009). Each broker pings the
  leaders of the shards it follows (new internal kinds Ping 38 and Pong
  39, capability PING, 1 << 7) and names in its heartbeat the ones that
  have not answered for FELIX_LEADER_SUSPECT_AFTER_MS (5 s; 0 turns it
  off). Once the fleet has finalized majority_ack and lease_free_reads,
  placement promotes a reported follower of a durable Quorum stream as soon
  as a majority of its set names the leader, without waiting for the leader
  to be marked down, so failover takes about 5 s instead of L + margin, and
  a leader cut off from its followers while still heartbeating is replaced
  too. The promotion keeps the set and fences as before. Leader streams and
  caches still fail over on the lease, and with the control plane down a dead
  leader is still not replaced. The heartbeat request gains an optional
  suspects list; the store gains record_suspicion/list_suspicions
  (Postgres migration 0024_node_suspicions, Raft leader soft state).
  run_heartbeat and membership::spawn take the broker's Suspects.
  TLA+: Suspicions constant, FelixShardSuspicion.cfg (pass) and
  FelixShardSuspicionLease.cfg (violates AtMostOneServing).
• felix-broker inspect segments <data-dir> [TENANT/NAMESPACE/NAME/SHARD]
  (part of #1077): a broker's data directory read from disk, by the broker
  binary without starting a broker; it binds nothing and reads no
  configuration. For every shard of every store (streams, caches/,
  groups/, dead-letters/, counters/) it lists the segments, checks every
  record's checksum and each index against one rebuilt from its segment, and
  gives the verdict startup would reach: clean, repair (and what it would cut
  or remove), or refuse (and the segment and byte). Strictly read-only: files
  are opened for reading and nothing is created, cut, removed or re-indexed.
  --repair-checksum-tail, --index-spacing and --verify-all-on-open match
  the broker's settings, --kind narrows to one store, --segments lists
  every shard's segments and --json prints one line per shard. It exits 0
  when every shard would open as it is, 6 when startup would repair one, and 7
  when it would refuse one or a record fails its checksum. Diagnosing a
  cluster covers a broker that will not start, on Kubernetes, Docker and
  Podman, and on a copy. Decoding records offline (inspect records) comes
  next.
• A cache shard's compaction pass now writes its key index to keys.idx in
  the shard directory, and an open loads it and replays only the log past it
  instead of the whole log (part of #1073). The snapshot is checked against
  the log (CRC-32C, the offset range it covers, the checksum of the last
  record it saw) and ignored for a full replay when anything disagrees; a log
  truncation and forget_index remove it. Memory is unchanged: the index is
  still held whole in memory. Format in docs/storage-format.md.

Changed

• FELIX_BROKER_PUBLISH_CONN_INFLIGHT_BYTES and
  FELIX_MAX_SUBSCRIPTIONS_PER_CONN are now per identity (tenant and token
  subject) on a connection, which for a plain client is the whole connection.
  Two new settings cap a connection across its identities:
  FELIX_BROKER_PUBLISH_CONN_TOTAL_INFLIGHT_BYTES (default four times the
  per-identity budget, at most FELIX_BROKER_PUBLISH_INFLIGHT_BYTES) and
  FELIX_MAX_SUBSCRIPTIONS_PER_CONN_TOTAL (default four times the
  per-identity cap), also settable in the config file. Startup refuses either
  below its per-identity value. A subscription past the connection's ceiling
  is refused with max subscriptions per connection exceeded across identities.

• Client::group_nack and ClusterClient::group_nack in felix-client take the
  GroupRecord instead of its offset, and send record.attempts to a broker
  advertising FEATURE_GROUP_CLAIM_CONTROL. The Python group_nack takes the
  GroupRecord and the Node groupNack takes the record (any object with
  offset and attempts). felixctl's group nack sends the attempt count of
  a SHARD:OFFSET:ATTEMPTS claim with or without --delay-ms.

• felix-storage: OffloadTarget::LocalDir is now LocalDir { dir, node },
  and LogConfig::validate refuses a node id that is not one path segment
  (letters, digits, -, _, ., not starting with .). In
  felix-broker-service, DurableStorageConfig::stream_log takes the cluster
  node id and returns a Result (#1109).

• Startup recovery of a shard now plans every write before it makes any
  (plan_recovery in felix-storage's disk_log/recovery/plan.rs), then makes
  them in the same order as before. What recovery does is unchanged; the plan
  is what felix_storage::inspect and felix-broker inspect segments report.
  An I/O error while planning now stops before any repair is written, where it
  used to stop partway. The segment_recovery fuzz target also checks that
  plan and recovery agree.

Fixed

• A publish forwarded to a moved shard's new owner while it was still fencing
  the replicas was refused as not_ready instead of held, so a planned move or
  shutdown handoff whose fence took longer than the forwarding broker's retries
  could refuse a publish (#1120). The owner now holds it until the shard
  serves, as it already did for a publish sent to it directly.
• A subscription or cache watch that registered just after a lease lapse,
  deposal or release ended its shard's readers is now refused instead of being
  left attached to a shard the broker no longer serves, where it got nothing
  and never heard the shard moved (#1117). The broker checks again after
  registering, and a broker deposed at a shard's generation now refuses new
  readers of it even while its lease holds.
• A retention sweep after a restart could delete records above a Quorum
  shard's commit offset, on an idle leader or a follower cut off from its
  leader, until replication reached the shard again (#1109). The hold at the
  commit offset is now kept in the shard's replica file (a flag in what was
  a reserved field, which older builds ignore) and restored at open, before
  retention starts. A stream or cache raised to Quorum is held as the
  broker applies the change, not from the first replication pass. A leader
  whose shard has no followers left lifts the hold, and so does a broker with
  neither membership nor a peer transport, at open, since nothing there
  advances a commit offset. In felix-storage and felix-broker,
  hold_retention_at_commit is now async and returns whether the change was
  written; felix-storage adds LogConfig::retention_hold,
  DiskLogProvider::set_stream_retention_hold and
  StorageApi::set_retention_hold.
• A consumer-group commit cancelled after it handed its write to a task, but
  before that task first ran, could still let the next commit read the old
  cursor and move the group backwards (part of #1109; the case #1097 left
  open). A log-backed cache put or delete that checks no condition now marks
  its key in flight before it spawns the write, so a settled read waits for it.
• A plain nack can no longer take a record from the consumer it was handed to
  next (part of #1109). group_nack names the delivery by its attempt count,
  as group_nack_after does, so a late one is refused with stale_claim.
• Two brokers offloading to one directory could write the same object key,
  because keys held only the shard directory name, which every replica of a
  shard shares, so one broker could replace a copy the other had recorded
  (#1109). Keys now start with the node id:
  <node id>/<shard directory>/<base>-<segment id>.segment. The node id is
  FELIX_NODE_ID, or, on a broker without one, an id generated once and kept
  in node-id under FELIX_DURABLE_STORAGE_DIR.
  Copies made by 0.6.0-preview.3 keep their old keys, which their manifest
  entries still name.
• One client acting for many users over a shared connection
  (Client::with_identity) no longer lets one user starve the rest (part of
  #1098). The broker's subscription cap (FELIX_MAX_SUBSCRIPTIONS_PER_CONN)
  and publish byte budget (FELIX_BROKER_PUBLISH_CONN_INFLIGHT_BYTES) now
  apply per tenant and token subject on a connection, so one user at either
  limit is refused or slowed while the others keep their room. A user's share
  is dropped once nothing of theirs is open or in flight on the connection. A
  client that authenticates every stream as one ident...

Felix v0.6.0-preview.3

The third preview of 0.6.0. felixctl now manages a cluster as well as using
it: it creates, changes and deletes tenants, namespaces, streams and caches,
works consumer groups, counters and RBAC rules, and shows a shard's or a
subscriber's live state with felixctl inspect. It installs with Homebrew.
Every change of a shard's leader is fenced, and replicas keep a ballot with
each generation they accept. Sealed segments can be copied to an object store
before retention deletes them, the first step toward tiered storage.
felix-capi adds a C ABI over the Rust client, the base for Go and C# SDKs.
Caches gain conditional writes and key-scoped grants, and a client can read a
bounded range of a stream. Development works with Docker or Podman. Several
move, failover and lease-timing bugs are fixed. The Rust API has breaking
changes, listed under Changed. pip will not install it without --pre.

Known issue

On a Quorum stream (replication factor above 1) with a small per-stream retention limit,
retention on the leader can delete records that a follower has not yet stored, so an
acknowledged write can be lost after a failover. Until 0.6.0-preview.4, don't set a small
per-stream retention on Quorum streams. Tracked in #1094; the fix is #1102. Fixed in v0.6.0-preview.4, together with the restart window it left open (#1111).

Added

• A Homebrew formula for felixctl: brew install getfelix/tap/felixctl on
  macOS and Linux, with completions and man pages (#1005). release.yml
  writes it from each release's archive checksums and pushes it to
  <owner>/homebrew-tap when PUBLISH_HOMEBREW is true, using the
  HOMEBREW_TAP_DEPLOY_KEY secret (a write deploy key on the tap); dry runs keep it as an artifact and nightlies
  skip it.
• felixctl inspect subs and the subscriptions_list request (part of
  #1077). A broker advertising FEATURE_INSPECT now also answers
  subscriptions_list with subscriptions_list_info: one page of the
  subscriptions it serves, each with its stream and shard, subscriber and
  subscription ids, connection id, client address and principal, overflow
  policy, queue depth and capacity in batches, records dropped, position and
  the shard's tail. The filter narrows by tenant, namespace, stream, shard,
  principal, or to subscriptions that have dropped records. A page is 100 by
  default and at most 1000, and a cursor of shard and subscriber id continues
  it. It needs node.view:cluster:*. The broker reads each shard's fanout
  snapshot and never touches a queue. On the delivery path, a subscriber's
  receiver now stores one past the offset of each batch it takes (a relaxed
  store), and a drop adds to a per-subscriber count beside the existing
  metric. felixctl asks every broker, or only --node, and prints one table
  or one JSON line per broker, with --principal, --dropping, --limit and
  --cursor. felix-broker adds Broker::list_subscriptions,
  SubscriberStats, SubscriberOwner and Subscription::set_owner;
  felix-client adds Client::list_subscriptions and SubscriptionsPage, and
  its inspect calls now report a broker's unsupported answer as such.
  Diagnosing a cluster covers a subscriber missing records and one falling
  behind.
• Stream logs can copy their sealed segments to an object store before
  retention deletes them, the first part of tiered storage (#172). Off by
  default; FELIX_DURABLE_OFFLOAD_DIR turns it on, and the only backend is a
  directory, through the object_store crate's local filesystem backend. A
  pass on the retention timer uploads each sealed segment, reads it back to
  check size and CRC-32, and records it in a per-shard offload.manifest,
  fsynced. Retention then deletes only recorded segments, so a crash at any
  step leaves the local segment or a recorded copy. Recovery accepts a gap in
  the local chain when the manifest covers it. Nothing reads the copies yet: a
  read below the local head is still Trimmed. New metrics
  felix_storage_offload_segments_total, felix_storage_offload_bytes_total
  and felix_storage_offload_failures_total. Breaking for code that builds
  felix-storage's LogConfig without ..Default::default(): it gains
  offload.
• felixctl inspect shard and the shard_inspect request (part of #1077).
  A broker advertising FEATURE_INSPECT, the first bit of the extended
  feature word (server_features_hi 0x1), answers shard_inspect with
  shard_inspect_info: its own view of one shard, with its phase and
  generation, whether it serves and why not (opening, fencing, failed
  with the open error, draining, lease_lapsed, not_assigned_here,
  behind_generation), which replicas took a promoted leader's fence and
  how many attempts it has made, its lease, tail and commit mark, and as
  leader each follower's next offset, lag and state. It needs
  node.view:cluster:* and may name any tenant. The answer is read from
  snapshots the lifecycle and the replication driver publish (the new
  felix_replication::status::ShardStatusBoard), so it takes neither's lock
  and opens no log. felixctl asks the leader and every replica and prints a
  table, or one JSON line per shard with --json; a broker that cannot be
  reached is listed as unreachable. felix-client adds
  Client::inspect_shard and Client::supports_inspect. A new docs page,
  Diagnosing a cluster, goes through shard, replication, subscriber, auth
  and startup problems by symptom.
• felix-capi, a C ABI over the Rust client and the base for the Go and C#
  SDKs (part of #618). It builds libfelix as a shared and a static library
  with a cbindgen header checked in at crates/sdk/felix-capi/include/felix.h,
  and covers connect, publish and a polled subscribe (felix_client_connect,
  felix_client_publish, felix_client_subscribe,
  felix_subscription_next_event). Handles are opaque with a free function
  each, every call returns a status code whose classes match the other SDKs,
  the message for a failure is per thread (felix_last_error_message), panics
  never cross the boundary, and each client owns its Tokio runtime. A test
  fails while the header is stale, and CI runs a C program against the
  conformance fixture. Not published to crates.io.
• Replicas keep a ballot with each accepted generation (part of #1009): the
  leader they accepted it from, by the node id its Hello gave. At that
  generation a replica refuses a fence, a batch, a bootstrap, a rebuild or a
  tail fetch from any other node with FencedEpoch, and a broker will not
  lead a generation it already accepted from another node. The ballot is a new
  ballot file in the shard directory, fsynced before anything from that
  leader is answered and read back on open; builds without ballots ignore it,
  so rolling back still opens the shard. A new internal capability, BALLOTS
  (1 << 6), is offered with the fence, and FELIX_INTERNAL_FENCE=false
  turns ballots off with it. Nothing changes while the control plane names
  every leader, since it never names two at one generation; this is the
  safety layer replica elections will need. The TLA+ model gains Ballots,
  Elections and OneLeaderPerGeneration, with FelixShardElect.cfg and
  two configurations that must fail: FelixShardElectNoBallot.cfg (two
  leaders at one generation) and FelixShardElectStaleSet.cfg (a replica
  standing on a set it has left). Breaking for callers of felix-storage's
  DiskLog::accept_generation and felix-broker's
  StreamLog::accept_generation, which take the leader, of felix-replication's
  ReplicaHandler entry points, which take the sender, and for code matching
  GenerationCheck, which gains Promised and is no longer Copy.
• felixctl group and felixctl counter (#1005). group create|describe|seek|rm
  manage a consumer group on every shard of a stream, or on --shard.
  group poll claims records and prints each with its claim,
  SHARD:OFFSET:ATTEMPTS, which group ack, nack (with --delay-ms),
  extend and dead-letters add take. group dead-letters ls|redrive|discard
  work the dead-letter list. group rm, dead-letters discard and a seek
  that moves a cursor back over finished records ask [y/N] at a terminal and
  need --yes elsewhere. counter get|add read and add to counters. All go
  through ClusterClient. felix-client now re-exports GroupRecord, which
  group_extend takes.
• Nightly TLC simulation of long random walks over replica-set changes
  (#934). FelixShardWalkSpares, FelixShardWalkMoves and
  FelixShardWalkHandoff lift the bounds the exhaustive configurations
  stop at (time to 40, six writes, six moves) and sample traces up to 300
  steps; each has a negative twin that must find its violation within the
  budget. scripts/check_tla.sh --simulate (task tla:walk) runs them,
  and tla-walk.yml runs them nightly and uploads TLC's trace on a
  failure. Tick in FelixShard.tla enumerates clocks over the drift
  window only, with the same states.
• felixctl writes to the control plane (part of #1005). tenant, namespace,
  stream and cache gain create and rm; stream set changes
  consistency, delivery, durability and retention, keeping a retention bound
  not given, and cache set the display name; each prints what changed.
  node drain and node deregister, shard move NAME SHARD --to NODE (with
  --dry-run and --cache) and shard move cancel, and placement pause,
  resume and abandon drive placement. Deletes, drains and deregistrations
  ask on a terminal and need --yes anywhere else, stopping with status 2
  without it; placement abandon never asks and always needs --yes.
• felixctl rbac policy ls|add|rm and felixctl rbac grouping ls|add|rm list,
  grant and revoke the current tenant's RBAC policies and role assignments
  through the control plane (#1005)....