Felix v0.6.0-preview.4

The fourth preview of 0.6.0, mostly fixes. A Quorum stream with a tight
retention bound could acknowledge a publish whose records no broker still
held; retention now stops at the commit offset on every replica, and that hold
survives a restart. A promoted cache or counter shard whose log fails to open
no longer opens unfenced. A consumer group's cursor no longer moves backwards
after a cancelled commit, and a late nack can no longer take a record from the
next consumer. An offload outage no longer fails durable logs at open, and
offload keys now start with the node id, so two brokers sharing a directory
cannot overwrite each other's copies. A shared connection's subscription and
publish limits now apply per identity, and a gateway can trade a user's token
for a delegated one carrying act, which subject binding accepts. A durable
Quorum stream with majority_ack and lease_free_reads finalized fails
over on its followers' word in about 5 s instead of waiting out the lease. A
subscriber that registers just as its shard moves is refused rather than left
attached to nothing, and a publish forwarded to a shard its new owner is still
fencing is held rather than refused. felix-broker inspect segments checks a
data directory offline and reports what startup would do. The docs site covers
felix-gateway and the projects built on Felix. The Rust API has breaking
changes, listed under Changed. pip will not install it without --pre.

Added

• Failover on the followers' word (part of #1009). Each broker pings the
  leaders of the shards it follows (new internal kinds Ping 38 and Pong
  39, capability PING, 1 << 7) and names in its heartbeat the ones that
  have not answered for FELIX_LEADER_SUSPECT_AFTER_MS (5 s; 0 turns it
  off). Once the fleet has finalized majority_ack and lease_free_reads,
  placement promotes a reported follower of a durable Quorum stream as soon
  as a majority of its set names the leader, without waiting for the leader
  to be marked down, so failover takes about 5 s instead of L + margin, and
  a leader cut off from its followers while still heartbeating is replaced
  too. The promotion keeps the set and fences as before. Leader streams and
  caches still fail over on the lease, and with the control plane down a dead
  leader is still not replaced. The heartbeat request gains an optional
  suspects list; the store gains record_suspicion/list_suspicions
  (Postgres migration 0024_node_suspicions, Raft leader soft state).
  run_heartbeat and membership::spawn take the broker's Suspects.
  TLA+: Suspicions constant, FelixShardSuspicion.cfg (pass) and
  FelixShardSuspicionLease.cfg (violates AtMostOneServing).
• felix-broker inspect segments <data-dir> [TENANT/NAMESPACE/NAME/SHARD]
  (part of #1077): a broker's data directory read from disk, by the broker
  binary without starting a broker; it binds nothing and reads no
  configuration. For every shard of every store (streams, caches/,
  groups/, dead-letters/, counters/) it lists the segments, checks every
  record's checksum and each index against one rebuilt from its segment, and
  gives the verdict startup would reach: clean, repair (and what it would cut
  or remove), or refuse (and the segment and byte). Strictly read-only: files
  are opened for reading and nothing is created, cut, removed or re-indexed.
  --repair-checksum-tail, --index-spacing and --verify-all-on-open match
  the broker's settings, --kind narrows to one store, --segments lists
  every shard's segments and --json prints one line per shard. It exits 0
  when every shard would open as it is, 6 when startup would repair one, and 7
  when it would refuse one or a record fails its checksum. Diagnosing a
  cluster covers a broker that will not start, on Kubernetes, Docker and
  Podman, and on a copy. Decoding records offline (inspect records) comes
  next.
• A cache shard's compaction pass now writes its key index to keys.idx in
  the shard directory, and an open loads it and replays only the log past it
  instead of the whole log (part of #1073). The snapshot is checked against
  the log (CRC-32C, the offset range it covers, the checksum of the last
  record it saw) and ignored for a full replay when anything disagrees; a log
  truncation and forget_index remove it. Memory is unchanged: the index is
  still held whole in memory. Format in docs/storage-format.md.

Changed

• FELIX_BROKER_PUBLISH_CONN_INFLIGHT_BYTES and
  FELIX_MAX_SUBSCRIPTIONS_PER_CONN are now per identity (tenant and token
  subject) on a connection, which for a plain client is the whole connection.
  Two new settings cap a connection across its identities:
  FELIX_BROKER_PUBLISH_CONN_TOTAL_INFLIGHT_BYTES (default four times the
  per-identity budget, at most FELIX_BROKER_PUBLISH_INFLIGHT_BYTES) and
  FELIX_MAX_SUBSCRIPTIONS_PER_CONN_TOTAL (default four times the
  per-identity cap), also settable in the config file. Startup refuses either
  below its per-identity value. A subscription past the connection's ceiling
  is refused with max subscriptions per connection exceeded across identities.

• Client::group_nack and ClusterClient::group_nack in felix-client take the
  GroupRecord instead of its offset, and send record.attempts to a broker
  advertising FEATURE_GROUP_CLAIM_CONTROL. The Python group_nack takes the
  GroupRecord and the Node groupNack takes the record (any object with
  offset and attempts). felixctl's group nack sends the attempt count of
  a SHARD:OFFSET:ATTEMPTS claim with or without --delay-ms.

• felix-storage: OffloadTarget::LocalDir is now LocalDir { dir, node },
  and LogConfig::validate refuses a node id that is not one path segment
  (letters, digits, -, _, ., not starting with .). In
  felix-broker-service, DurableStorageConfig::stream_log takes the cluster
  node id and returns a Result (#1109).

• Startup recovery of a shard now plans every write before it makes any
  (plan_recovery in felix-storage's disk_log/recovery/plan.rs), then makes
  them in the same order as before. What recovery does is unchanged; the plan
  is what felix_storage::inspect and felix-broker inspect segments report.
  An I/O error while planning now stops before any repair is written, where it
  used to stop partway. The segment_recovery fuzz target also checks that
  plan and recovery agree.

Fixed

• A publish forwarded to a moved shard's new owner while it was still fencing
  the replicas was refused as not_ready instead of held, so a planned move or
  shutdown handoff whose fence took longer than the forwarding broker's retries
  could refuse a publish (#1120). The owner now holds it until the shard
  serves, as it already did for a publish sent to it directly.
• A subscription or cache watch that registered just after a lease lapse,
  deposal or release ended its shard's readers is now refused instead of being
  left attached to a shard the broker no longer serves, where it got nothing
  and never heard the shard moved (#1117). The broker checks again after
  registering, and a broker deposed at a shard's generation now refuses new
  readers of it even while its lease holds.
• A retention sweep after a restart could delete records above a Quorum
  shard's commit offset, on an idle leader or a follower cut off from its
  leader, until replication reached the shard again (#1109). The hold at the
  commit offset is now kept in the shard's replica file (a flag in what was
  a reserved field, which older builds ignore) and restored at open, before
  retention starts. A stream or cache raised to Quorum is held as the
  broker applies the change, not from the first replication pass. A leader
  whose shard has no followers left lifts the hold, and so does a broker with
  neither membership nor a peer transport, at open, since nothing there
  advances a commit offset. In felix-storage and felix-broker,
  hold_retention_at_commit is now async and returns whether the change was
  written; felix-storage adds LogConfig::retention_hold,
  DiskLogProvider::set_stream_retention_hold and
  StorageApi::set_retention_hold.
• A consumer-group commit cancelled after it handed its write to a task, but
  before that task first ran, could still let the next commit read the old
  cursor and move the group backwards (part of #1109; the case #1097 left
  open). A log-backed cache put or delete that checks no condition now marks
  its key in flight before it spawns the write, so a settled read waits for it.
• A plain nack can no longer take a record from the consumer it was handed to
  next (part of #1109). group_nack names the delivery by its attempt count,
  as group_nack_after does, so a late one is refused with stale_claim.
• Two brokers offloading to one directory could write the same object key,
  because keys held only the shard directory name, which every replica of a
  shard shares, so one broker could replace a copy the other had recorded
  (#1109). Keys now start with the node id:
  <node id>/<shard directory>/<base>-<segment id>.segment. The node id is
  FELIX_NODE_ID, or, on a broker without one, an id generated once and kept
  in node-id under FELIX_DURABLE_STORAGE_DIR.
  Copies made by 0.6.0-preview.3 keep their old keys, which their manifest
  entries still name.
• One client acting for many users over a shared connection
  (Client::with_identity) no longer lets one user starve the rest (part of
  #1098). The broker's subscription cap (FELIX_MAX_SUBSCRIPTIONS_PER_CONN)
  and publish byte budget (FELIX_BROKER_PUBLISH_CONN_INFLIGHT_BYTES) now
  apply per tenant and token subject on a connection, so one user at either
  limit is refused or slowed while the others keep their room. A user's share
  is dropped once nothing of theirs is open or in flight on the connection. A
  client that authenticates every stream as one identity sees the same limits,
  refusals and frames as before.
• Subject binding (FELIX_TLS_CLIENT_CERT_BIND_SUBJECT) refused every user's
  token a gateway presented over its own certificate (part of #1098). The
  control plane has a new RFC 8693 token exchange,
  POST /v1/tenants/{tenant_id}/token/delegate: a caller holding the new
  token.delegate:tenant:{tenant_id} permission trades a user's broker token
  for one with the same subject, no wider permissions and no later expiry, and
  act: {"sub": "<caller>"}. The broker binds such a token to a certificate
  issued to its actor, on QUIC and Kafka SASL alike; a token without act
  binds only to its subject, as before. token.delegate is never put in a
  broker token, so brokers that predate it are unaffected.
• The docs said a delegated identity had its own publish budget when only the
  client's was per identity; they now describe what the broker enforces.
• An in-memory cache's versions restarted at 0 with the broker, so a
  cache_put_if or cache_delete_if holding a version read before a restart
  could apply against a value written after it (#1098). Versions now start
  each run at the wall-clock time in microseconds. Log-backed caches were
  not affected: their versions are log offsets.
• A late nack can no longer take a record from the consumer it was handed to
  next (#1098). group_nack takes an optional attempts that names the
  delivery, as on group_extend, and is refused with stale_claim once that
  claim has lapsed or the record has gone out again. Without it, a delayed
  nack arriving after a redelivery parked the new delivery behind its delay.
  Left out at 0, so a nack that does not name a delivery is the frame it
  always was; a broker that predates the field ignores it. felix-client's
  group_nack_after now takes the GroupRecord rather than its offset (on
  Client and ClusterClient), and sends attempts to a broker advertising
  FEATURE_GROUP_CLAIM_CONTROL. felixctl sends it for a nack --delay-ms
  claim given as SHARD:OFFSET:ATTEMPTS. In felix-broker,
  GroupReader::nack_after takes the attempt count to check and nacks at once
  for a zero delay.
• felixctl saved its config through a temporary file with a predictable
  name, opened without O_EXCL, so a symlink planted at that path in a
  shared directory was followed and the target overwritten (#1098). The
  temporary file now gets a random name and is created exclusively,
  owner-only on Unix, before it is renamed over the config.
• A first segment whose header and records had all been zeroed was deleted at
  startup as a segment whose creation never finished, and the log came back
  empty (part of #1098). Creation syncs the header before anything appends,
  so only a file no longer than a header can be an unfinished creation.
  Recovery now refuses a longer one as corruption.
• Retention could acknowledge a Quorum publish for records no broker held
  (#1094). With both followers of a shard down, a tight retention bound
  deleted records only the leader had, including ones publishes were still
  waiting on. The followers came back below the leader's new base, were
  rebuilt there, and their answers moved the quorum mark over the deleted
  records. Retention and compaction's head trims now stop at the commit
  offset on every replica of a Quorum shard, and the leader will not
  bootstrap or rebuild a follower at a base above it. A stream whose
  followers are down can therefore grow past its retention bound until a
  majority is back. DiskLog::hold_retention_at_commit and
  StreamLog::hold_retention_at_commit are new.
• A consumer group's cursor can no longer move backwards after a cancelled
  commit (#1097). The cancelled commit's write still finished on its own task,
  but the next commit read the position from before it and wrote over it. A
  commit now waits for a write to the cursor that is still applying. Finishing
  a redrive had the same race against a cancelled return to dead and could
  drop the dead letter; it waits the same way.
• A missing or read-only offload directory no longer fails every durable
  stream log at open. The directory is opened by the offload pass, which
  reports a failure through felix_storage_offload_failures_total, the new
  felix_storage_offload_failing_logs and felix_storage_offload_held_bytes
  gauges and a backed-off log line, and retries on the next tick. Retention
  still never deletes a segment without a recorded copy, so an outage grows
  local disk; deletion resumes once the directory is back (#1096).
• A promoted cache shard whose cache log or counter log failed to open (the
  shard closing, too many open files, a failed recovery) opened on the lease
  without a fence, even once caches acknowledge by their followers and the
  lease no longer stops the old leader. It now stays closed and the next pass
  tries again (#1095).

Documentation

• The docs said an in-memory cache's versions never repeat across a restart.
  They can if the clock steps back across the restart or the last run averaged
  more than one write per microsecond; docs/protocol.md and the cache page
  now say so (#1109). That cache runs only on a broker with no data directory,
  so there is nowhere to record where its counter stopped.
• The docs site documents felix-gateway, the WebSocket relay that lets
  browsers use Felix, under Clients, and has a Built on Felix section for
  felix-canvas, felix-webhook-relay and felix-arena that says which Felix
  features each one uses and what it relies on Felix to guarantee.