Felix v0.6.0-preview.6

The sixth preview of 0.6.0, fixes to shard placement, broker readiness and the
control plane. A cluster started all at once now places its shards on all its
brokers: while its first broker registered less than
FELIX_PLACEMENT_SETTLE_MS ago (10 s by default), a new shard waits for as
many live brokers as its factor, and a restore that only tops up a nearly
empty shard no longer takes the move slot. A shard placed while its leader was
the only live broker now gets its other copies once more brokers join. A
broker fetches every tenant's signing keys before it reports ready, so after a
restart it no longer refuses forwarded writes, such as cache puts, while the
control plane is down. A stopped control-plane Raft member no longer claims
leadership, and a second SIGTERM sent just as the control plane starts
draining is no longer missed. The docs now give failover timings per case with
the default settings.

Fixed

• A cluster started all at once now places its new shards on all the brokers
  it starts with. Placement ran as soon as the first one or two brokers had
  registered, so each shard got fewer copies than its replication factor, and
  restores added the rest one shard at a time behind the single move slot:
  eleven shards spent about 90 s short, and stopping a broker then could leave
  some without a majority. While the cluster's first broker registered less
  than FELIX_PLACEMENT_SETTLE_MS ago (10 s by default), a new shard waits for
  as many live brokers as its factor, and a broker registering runs placement
  at once. A restore that only tops up a short shard whose leader holds at
  most FELIX_SHARD_MOVE_FENCE_MAX_LAG_RECORDS records no longer takes a move
  slot, so a cluster whose shards did go short fills them together. The control
  plane logs when shards go short of their factor and when they are full again
  (#1151).
• A second SIGTERM sent to the control plane just after its readiness flipped
  to draining could be missed, so the pre-drain hold-off ran in full. The
  listener for it is now installed before readiness changes.
• A shard placed while its leader was the only live broker now gets the
  copies its replication factor asks for once other brokers join. Placement
  adds a copy only after the leader has reported at the shard's generation,
  and a leader with no followers never reported, so the shard stayed at one
  copy with under_replicated: true and nothing logged. A leader with no
  followers now reports itself, after its promotion fence as before (#1153).
• A broker fetches the JWKS of every tenant in its catalog when its first
  catalog sync lands, before it reports ready. It used to fetch them only when
  a client of that tenant connected, so a broker that restarted and then lost
  the control plane refused every write a peer forwarded to it, such as a
  cache put sent to a broker that does not own the key, until the control
  plane came back (#1148).
• A control-plane Raft member that had shut down still reported itself as
  leader and answered leader-only requests (heartbeats, expiry) from its last
  term's view. It now reports no leadership and refuses them with the same
  unavailable answer an instance without a leader gives, so callers move on
  to a live member (#1144).

Documentation

• The failover docs said a lost leader is replaced in about a second, which is
  the cluster harness's 1 s expiry window. They now give each case with the
  defaults: a crashed Leader-stream or cache leader is marked down 18.75 s
  after its last heartbeat and replaced on the next placement pass, a SIGTERM
  hands leaderships off as planned moves, and a lease-free Quorum stream fails
  over about 5 s after its leader goes quiet (#1152).