Felix v0.6.0-preview.7

The seventh preview of 0.6.0. Subscriber memory on the broker is now bounded
by bytes end to end: each subscriber queue, and the writer lanes behind it,
stop at FELIX_SUB_QUEUE_MAX_BYTES (256 MiB by default) as well as at their
entry count. The bound is sized so a subscriber keeping up at line rate with
4 MiB batches never reaches it. Under the block policy, one full subscriber
no longer holds up a batch's fanout to the others. Revoking a principal now
ends the streams its tokens already opened, with the new revoked error code,
and a subscription whose stream is deleted ends with stream_deleted for
clients that offer it. ClusterClient::with_identity lets a gateway act as
another user over shared cluster connections, and a delegating gateway can
hold a connection-only broker token instead of a grant it never uses.
Replication and placement fixes: group and dead-letter logs ride each
follower's exchange, a new Quorum copy is seated only once the leader names
it caught up, and a counters-only cache restore is no longer dropped as
stalled.

Upgrade notes.

• A Raft control plane needs every member upgraded before revocation (metadata
  level 7) and counter-offset reports (level 8) take effect. Until then the
  revoke endpoint answers 409 after revoking the refresh tokens, and reports
  are recorded without counter offsets.
• A Postgres control plane runs migrations 0025 and 0026 on start.

Added

• Subscriber queues on the broker are bounded by bytes as well as entries.
  An entry is a whole published batch, so the count alone let one slow
  subscriber hold up to 4096 batches of up to 16 MiB each. Each queue now also
  stops at FELIX_SUB_QUEUE_MAX_BYTES of payload (256 MiB by default, 0 for
  none), and the queue policy handles a full byte bound as it handles a full
  count: drop_new drops the batch for that subscriber, with the same offset
  gap and drop counters, and block waits. A batch larger than the bound
  still goes into an empty queue. FELIX_SUB_QUEUE_MAX_BYTES_TOTAL (off by
  default) caps the bytes held across all subscriber queues, at the cost of
  isolation between subscribers. New metrics: felix_sub_queue_bytes and
  felix_sub_queue_byte_dropped_total{bound}. Broker gains
  with_subscriber_queue_max_bytes and with_subscriber_queue_max_bytes_total
  (#1157, #1189).
• A delegating gateway no longer needs a broker grant it never uses. A
  principal holding token.delegate:tenant:{tenant_id} can exchange for a
  connection-only broker token, its own subject with no permissions, by
  sending "requested": [], "permissions": [] to /token/exchange (or the
  dev-token bootstrap endpoint). Its shared connections authenticate with it,
  and every request still runs under the user's delegated token. Refresh
  checks token.delegate again and ends the chain once it is removed. Anyone
  else asking for no permissions still gets 403 no permissions (#1130).
• ClusterClient::with_identity(tenant_id, token_provider) and
  ClusterClient::with_identity_token act as another user over the cluster
  client's connections (#1129), as Client::with_identity does for one
  broker. The result is a ClusterClient with its own routing state that
  follows moved shards, redirects and failovers, reaching each broker over the
  parent's connection to it, so a gateway sharing connections keeps a user's
  subscription across a shard move and still holds one connection per broker.
  The Python Client.with_identity and AsyncClient.with_identity and the
  TypeScript Client.withIdentity wrap it with a fixed token. No wire change.
• Revoking a principal ends what its tokens already opened, on every broker,
  within about one control-plane sync interval (#1132). Until now a revoked
  principal kept its open subscriptions, cache watches, publish streams and
  Kafka connections until its token expired. POST /v1/tenants/{t}/refresh-tokens/revoke now also records a revocation, and so
  does removing an RBAC policy or role assignment, for everyone the rule
  reached (the whole tenant when it reached an IdP group). Brokers poll the new
  GET /v1/principal-revocations/changes feed, refuse broker tokens issued at
  or before a revocation, and end the streams they opened with the new
  revoked error code (16, fatal): an error on a control stream, or as the
  last frame on a subscription's or cache watch's event stream. A Kafka
  connection is closed at its next request. Under the Raft store this is
  metadata version 7, and until every member has it the revoke endpoint
  answers 409 after revoking the refresh tokens. See
  docs/auth.md.
• CacheWatch::refusal in the Rust client says why the broker ended a watch
  with an error, such as revoked. A subscription ended that way returns the
  BrokerError from next_event instead of an "unexpected message" error
  (#1132).
• The subscriber byte bounds also cover the broker's writer lanes. A batch's
  bytes stay held until the frame carrying it is written to the subscriber's
  QUIC stream, so frames queued behind a flow-controlled subscription count
  toward FELIX_SUB_QUEUE_MAX_BYTES instead of piling up in the lanes by entry
  count alone (up to 64 frames of up to 16 MiB each per subscription). Under
  drop_new the next batch is dropped at the subscriber queue as before;
  publishers still never wait. SubscriptionReceiver gains recv_held and
  try_recv_held, and HeldBytes is public (#1163).
• A subscription ended because its stream was deleted now says so. The broker
  sends stream_deleted as its last frame to a client that offered the new
  extended feature bit FEATURE_STREAM_DELETED; any other client sees the
  stream end as before. Removing a stream used to leave its subscribers
  waiting on a feed nothing would end. The Rust client reports it as
  Subscription::stream_deleted and ClusterSubscription::stream_deleted,
  and a sharded subscription yields ShardEvent::StreamDeleted for each shard
  instead of trying to resubscribe it (StreamDeleted in Python,
  streamDeleted in TypeScript) (#1119).

Changed

• A broker closing a shard it no longer has any role in tells a reader still
  attached that the shard moved, naming its current leader, rather than just
  ending the stream. Broker::close_stream_shard takes the handoff (#1119).

Fixed

• Under the block queue policy, one subscriber at its byte or entry bound no
  longer holds up the rest of a batch's fanout. The broker queues the batch
  for every subscriber with room first and waits only on the full ones, so a
  fast subscriber gets each batch as soon as it is durable. Each subscriber
  still receives batches in publish order. The latency demo now passes
  FELIX_SUB_QUEUE_MAX_BYTES and FELIX_SUB_QUEUE_MAX_BYTES_TOTAL to the
  broker it builds (#1179).
• The Node client returns null, as index.d.ts says, for an absent field on
  a delivered item: laggedResumeFrom, change and shardMoved on a cache
  watch item, value on a cache change, offset on an event, event,
  lostError and recovered on a sharded item, and resumeFrom, nodeId and
  addr on a shard move. They came back undefined, so a caller testing
  !== null read every cache watch item as a lag notice (#1140).
• The cluster test harness's wait_for_replication now waits until the
  control plane holds a report naming a follower at the leader's current tail.
  It used to return once the leader had shipped anything, which the harness's
  start-up publish already guaranteed, so a test that killed the leader next
  could fail over without the records it had just published (#1136).
• Right after a broker died, a stream with consumer groups could hold each
  replication pass on it for a request timeout or a dial: the group cursor and
  dead-letter logs shipped to every follower at the end of the pass, and the
  next publish waited behind that pass. They now ride each follower's
  exchange, as a Quorum cache's counters do since #1137, and so do a
  non-Quorum cache's counters (#1143).
• A new copy of a Quorum cache is no longer seated before it holds the
  cache's counter log. Seating checked only the cache log's offsets, so the
  set could count toward its majority a copy that promotion would refuse.
  A seat on a Quorum stream or cache now also waits until the leader's
  report names the copy caught up (#1139).
• A restore of a cache used only for counters is no longer dropped as stalled
  while it copies a large counter log. The stall window
  (FELIX_SHARD_RESTORE_STALL_MS) watched only the cache log's offset, which
  stays at 0 for such a cache. Replica reports now carry each replica's
  counter-log position (replica_counter_offsets, omitted for streams and by
  older brokers), and placement counts a move there as progress. The control
  plane stores it in the report (Postgres migration
  0026_replica_report_counter_offsets, Raft metadata level 8; below level 8
  reports are recorded without it) (#1164).