Skip to content

v0.1.0-preview.2

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 09 Nov 21:00
· 28 commits to main since this release
ac34afa

This release uses Felix 0.7.0 and needs a Felix 0.7 broker and control plane. The compose
files pin the 0.7.0 images and felixd builds against felix-client 0.7.0. Upgrade every node
before using --upgrade-at (see below).

  • Module upgrades at a recorded input (#64). felixd run --id <id> --upgrade-at <input> <module>
    makes the new module the workload's from that input on. The spec keeps the module the workload
    began with and lists each upgrade as an input and a module hash, so the lineage rules that
    compare the module are unchanged and a spec without upgrades is written as before. The owner
    hands its state to the new module through a snapshot and a restore in memory at the upgrade's
    input, a node that resumes does the same, and felixd check and felixd replay switch at the
    same inputs. felixd run refuses an input the workload has already run, and a module that does
    not declare it can restore the old one's snapshot layout. It writes an upgrade record to .snap
    with publish_if at the tail it checked before the spec, and an owner on an older spec writes no
    snapshot once it has seen one, so the old module cannot leave a snapshot past the upgrade.
    A later spec that drops a recorded upgrade is refused. Upgrade every node before using
    --upgrade-at: older builds cannot read the upgrade record. Upgrades are an operator action and
    admission rejects requests that name them.

  • The state classes table in docs/design.md covers every cache and system stream felixd uses (#81), and a test fails when a new name has no row. It now lists the leases, epochs, workloads and modules caches and the sys.intake stream.

  • The scale test measures failover without waiting on its own harness. It scanned every output stream
    in one pass with the whole phase as its budget, so one stuck read made leaders_moved_secs and
    without_output measure the harness. Now a scan round takes at most 10 s, each read at most 3 s,
    and the leaders are checked every round. The 300 s phase limit still bounds the whole wait.

  • A value over Felix's frame limit fails with a clear error (#196). A 20 MiB cache value used to fail
    as write frame payload: sending stopped by peer and a reconnect. felixd now measures cache puts
    and publishes before sending and returns TooLarge, naming the key or stream, the size and the
    most Felix takes (about 12 MiB for a cache value, which travels as base64). An output append that
    gets it marks the workload failed. Felix is unchanged.

  • Agents degrade under load instead of tipping into lease churn (GetFelix/felix#1760). On a slow
    cluster, leases lapsed together, every agent claimed them back at once, and the claims slowed
    renewals further until nothing settled. Now a holder whose renewal is late pauses its writes at
    two thirds of the TTL and carries on if that renewal is applied, instead of dropping the
    workload. A renewal unanswered for a TTL counts as failed. A node makes at most 8 claims at
    once, holds new claims back for up to TTL/3 while its renewals are slow, and waits before
    claiming a workload it lost again (TTL/12 to TTL/6, doubling per loss in a row up to one TTL).
    Lease and epoch rules are unchanged; these only delay claims and writes.

  • Nightly scale run against Felix main (GetFelix/felix#1760). scale-nightly.yml runs the scale test at 1000
    workloads on the :nightly Felix images every night and opens an issue labelled felix-nightly-scale when they
    do not all start. Scale takes a felix_image_tag input and can be called from another workflow.

  • Fewer control plane requests per felixd run (GetFelix/felix#1760). Since the read back
    after a 409 (#149), each submit sent about nine extra requests to the control plane. A 409
    that says the stream or cache already exists is no longer read back; any other 409 still
    is. The control plane calls also share one connection.

  • Operator docs (#24). docs/operating.md covers running a cluster: what to start, the
    configuration of every command, the grants each role needs, sizing the lease TTL, what failover
    and replay guarantee, a three-node failover walkthrough, what each ps state asks of an
    operator, and how to upgrade a node.

  • Scale test (#20). The manually run Scale workflow starts three Felix brokers and three agents, creates N
    workloads and reports create time, memory, and how long recovery takes when an agent and then a broker is
    killed (crates/felixd/tests/scale.rs, .github/workflows/scale.yml; see CONTRIBUTING.md). The numbers
    are in design.md under "Scale": hundreds of workloads work on Felix 0.6.2 and 0.7.0-preview.2, 1000 often do not
    settle on 0.7.0-preview.2 (about 17,000 claims in the 20 minute window, against about 1,100 over the 7 to 8
    minutes 0.6.2 needed to get all running) and first output at 500 took 282 s there against 6.9 s on 0.6.2.
    Creating streams (about 1.5 to 7 a second) and a broker failure are slow
    (felix#1760,
    felix#1761,
    felix#1009).

  • Divergence records (#73). A workload fails as poison when a node replays its
    inputs and gets an output other than the one the stream holds. The node now
    first appends a divergence record (output record kind 9: spec record,
    input offset and index, the input of the snapshot the replay started from,
    node id, contract, wasmtime and felixd versions, the sha256 of the output
    the stream holds and of the one the replay produced, detail), fenced by epoch
    like the failure record that follows it, which keeps its class. It is best
    effort, and a deposed node writes neither record. felixd ps prints it under
    a failed workload, felixd replay --diff names it when it finds the same
    output differing, and felixd_divergences_total counts them. Every reader
    passes over it like a claim, and a node that cannot read one that is last on
    the stream does not take the workload. A build from before kind 9 reads it
    as an unknown kind: an old node that claims a failed workload after its spec
    is submitted again fails the new spec, an old ps, check or replay
    reports the stream unreadable, and a downstream workload run by an old node
    needs an operator. Finish a rolling upgrade before submitting a failed
    workload again. There is no switch. This closes #73;
    defining the records in WIT waits for the effect records, and felixd does
    not push OTLP.

  • felixd agent shuts down cleanly on SIGTERM as well as Ctrl-C (#203). It stops its workloads and deletes
    their leases, so docker stop and systemctl stop hand work to another node at once instead of after
    the lease TTL. As PID 1 in a container it no longer ignores SIGTERM.

  • Degraded workloads (#73). A workload that keeps failing on a transient
    error, such as a stream its node cannot read, used to be retried for ever
    on that node, out of sight. Now felixd ps shows it as degraded with the
    error, and another node takes it. The holder gives up when attempts have
    failed for --stuck-after-ms (default 60000, or the lease TTL if longer;
    the agent refuses a value between 0 and the TTL) with no progress in
    between, meaning no new input stepped and no record appended. It appends a
    degraded record (output record kind 8: spec record, error class, node id,
    detail) fenced by epoch, deletes its lease so another
    node can claim it, and leaves the workload alone for
    --degraded-backoff-ms (default 120000). The spec stays running, so
    nothing has to be done. felixd_degraded_total{class} counts releases.
    A build from before kind 8 refuses the record: it fails a workload whose
    last record it is, never claims one whose first record it is, and a
    downstream workload reading the stream needs an operator. During a rolling
    upgrade run the new build with --stuck-after-ms 0, which turns the bound
    off, until no old node is left. Counting degraded records across nodes and
    blocking claims is #99.

  • Modules over 8 MiB (#21). A module over 8 MiB is stored in the modules
    cache as 4 MiB chunks under <sha256>/<n> with a JSON manifest under the
    module's own key, chunks first and manifest last. A module of 8 MiB or less,
    which is all a build before this could hold, is stored as before. Readers
    (nodes, felixd check, felixd replay, admission's projector) read four
    chunks at a time, check each chunk and then the whole against the hash; a
    missing chunk is the module missing needs-operator reason and a wrong one
    fails the workload as poison. Admission's default max_module_bytes and
    felixd run's limit are now 64 MiB, up from 8 MiB. New
    felixd_agent::module_store and felixd_core::module items;
    felixd-testkit gains padded.

  • Upgrading for module chunks (#21): upgrade every node before submitting a
    module over 8 MiB, since an older node reads the manifest as a module that
    does not match its hash. An admission module already running keeps the
    8 MiB limit it was built with; to raise it, build the new admission module,
    check it with felixd replay --id admission --diff --all --module <hash> and
    run felixd init with it.

  • Refused stream creates (#149). felixd run, felixd init and felixd drain
    read a stream or cache back after Felix answers 409 (unless it says the
    resource already exists), and now exit with an
    error carrying Felix's reason where they used to succeed and leave the next
    read waiting for a stream that was never created. Felix refuses a stream
    under the name of a deleted one until the cluster has finalized stream
    incarnations; a control plane no broker registers with, like the
    single-broker test stack, never finalizes them. Admission's projector hits the
    same refusal when it creates an accepted workload's streams: it logs Felix's
    reason and retries every second until the name can be reused, instead of
    recording a spec whose streams do not exist.

  • Felix 0.7.0-preview.2. felixd builds on felix-client 0.7.0-preview.2 and the
    compose files and test stacks default to the 0.7.0-preview.2 images. Nothing
    changes for users. This release is the first with conditional head trims,
    which later changes use to delete history. The client's cluster type is now
    called Client.

  • Upgrade check from a snapshot (#29). felixd replay --diff --module and
    replay --to ... --module start the new module from the workload's nearest
    snapshot (before --from for a diff, before the end of the range for --to)
    when the new module's felixd.snapshot custom section says it reads the
    layout the workload's module wrote, and from the initial state when it does
    not, saying which and why on stderr. The section is optional JSON,
    {"format": "<id>", "reads": ["<id>", ...]}, additive to contract 0.2
    (docs/contract.md, "Snapshot layouts"); a module without one is readable only
    by itself. A new module that declares the layout and traps or runs out of
    fuel restoring it is reported as "snapshot not readable by the new module",
    not as a divergence. A section that does not parse is an error when it is
    read, and the message points at --from-start. New --from-start always
    replays from the initial state. --module takes a component file as well as
    the hash of one in the modules cache. The --to form now prints two lines
    on stderr, started from ... and ran N more of the inputs below E, where
    it printed one. In felixd_agent::replay, replay and diff take
    from_start, Replayed carries a Start in place of restored and Diffed
    gains start (breaking for library users). The counter example declares
    its layout and has cargo features that build the variants the tests replay
    against it; felixd_testkit::workload_with builds one.

  • Broker host names and token rotation (#160, #159). --brokers takes
    host:port as well as ip:port; names are resolved on connect and again
    when no known broker answers. --token-file and
    --controlplane-token-file are read again each time the token is used, so a
    token rotated in place keeps the agent's leases; a missing or half-written
    file keeps the last token. felixd_agent::settings::Settings holds
    Seed brokers and TokenSource tokens (breaking for library users).

  • Grants (#117). felixd init writes Felix RBAC policies for three roles
    per namespace, role:felixd.<namespace>.node, .admission and
    .submitter, and writes nothing it finds already there; the operator
    running it needs rbac.view and rbac.policy.manage. Only admission's
    credential may write sys.specs and create workload streams; a submitter
    publishes requests and puts modules; a node holds no control plane rights.
    Breaking: the sys.* streams move to the namespace <namespace>.sys, so a
    node's grant on every workload stream does not reach them; felixd run and
    felixd init create it. felixd init, which grants nodes publish on
    <namespace>/*, refuses while an old sys.* stream is in the workload
    namespace, so that check is what protects an upgrade; agents do not check.
    The migration steps are in docs/design.md ("Migrating the system streams"). init also
    refuses, writing nothing, when a felixd role holds a rule beyond its
    grants or one principal holds two roles, and needs rbac.view over the
    whole tenant. Namespaces ending in .sys are refused. New felixd agent --admission-token-file
    (FELIXD_ADMISSION_TOKEN_FILE), the broker token admission's projector
    writes with. A node reading a spec record past the tail of sys.specs no
    longer needs publish there.

  • Admission (#114). felixd init admission.wasm brings up the admission
    workload, reading the new sys.intake stream; it is the one spec still
    written directly. From then on felixd run uploads the module, publishes a
    request on sys.intake and prints admission's decision with how long it
    took, exiting 1 on a rejection, and needs no control plane token. Admission
    outputs one decision per request on wl.admission.out, accepting the spec
    or rejecting it with a reason code (limits, module size, ids, inputs), and
    decides from the request alone, so felixd replay --id admission --diff --all --module <hash> lists the recorded requests a new policy decides
    otherwise (--all is new). The node holding admission's lease records
    each accepted decision on sys.specs, naming it, after checking the
    module's hash and length and creating the streams; only nodes with a control
    plane token hold admission. A node then runs a spec record written after
    admission's own only if it names a decision that accepted exactly that
    spec; any other needs an operator with the new reason spec_unadmitted and
    never claims. New fault crash-after-projection-sent=N for tests.

  • Resubmitting a workload a newer build left to an operator (#136). felixd run
    from an older build submits a new spec over a needs-operator record whose
    reason it cannot read, which is how the workload is released. Nodes on the
    older build leave the workload alone until a node that can read the record
    takes it. Such a record also raises the highest spec claimed, as one it can
    read does.

  • System workloads (admission, scheduler) are refused through intake;
    felixd run writes their specs directly with the control plane token as
    before. A spec record runs only if no record of the workload anywhere on
    sys.specs names a later accepting decision, so an old acceptance cannot
    run again, whether it is appended again or workloads is pointed back at
    it. An accepted decision whose module does not match gets a skip record
    (skipped), and a projector resumes after the newest decision any verified
    record names, so an acceptance one passed over is never recorded later.

  • Deleting snapshots and the merge log (part of #19). After the trim record
    lands, the holder trims the workload's .snap and merge log with Felix's
    conditional head trim, up to the record's floors, and a holder that resumes
    deletes whatever a crash left. Inputs and .out follow below (#150).
    Needs the Felix fleet to have finalized head_trim, which is a one-way step
    (back up first), and stream.trim on the node's token, which is off by
    default: felixd init --grant-trim grants it to the node role. Without
    either, the node logs one warning and the workload runs on without deleting.
    felixd check and felixd replay read .snap from the oldest offset Felix
    holds, and stop with the stream and the oldest offset when they need a merge
    log entry that was deleted. The compose files and the cluster test stack
    finalize head_trim.

  • felixd init --grant-trim (part of #19). Grants the node role stream.trim
    on the namespace's streams, so nodes can delete history. Felix grants a
    stream by its exact name or a whole namespace, so a node's credential can
    then trim any stream in it, inputs and other workloads' outputs included,
    and a trim erases history for every reader and cannot be undone. Without the
    flag init grants no trim and nothing is deleted. Re-mint node tokens once
    the fleet has finalized head_trim, since Felix puts the grant in tokens
    only after that. Running init without the flag later keeps the rules;
    remove them with the control plane's RBAC API. The flag also grants the node
    role stream.trim on sys.facts by exact name (#164), so the scheduler's
    run can delete facts; without it the facts are kept. A namespace that ran
    init --grant-trim before this needs it run again and the nodes' tokens
    re-minted, and an older felixd's init refuses a namespace that holds the
    new rule.

  • Trimming sys.facts (#164). The first input that is deleted: a scheduler run
    with --keep-snapshots also deletes sys.facts below the floor its trim
    record names for that input, the same step that deletes .snap and the merge
    log. Only the scheduler reads sys.facts, and felixd run now refuses a
    system stream as the input of any other workload, as admission already did
    (breaking for a workload submitted without admission: one already running
    with a system stream is not rechecked, and parks as needing an operator once
    the scheduler trims). Every node appends to the stream, so a
    refused tail is asked again at the one Felix reports, up to three more
    times, where .snap and the merge log give up. About keep-snapshots x snapshot-every facts stay, each about 150 bytes plus the ids of the
    workloads its node holds (3 bytes and the id each): about 1 MB for
    --keep-snapshots 3 at the default when nodes hold 8 workloads.
    felixd replay of the scheduler starts from a snapshot and is unaffected. felixd check of a scheduler that has
    deleted history says which stream it needed and the oldest offset Felix
    holds, as it does for the merge log. wl.scheduler.out is never trimmed
    (#150).

  • Deleting inputs and .out (the deletion half of #150). With
    --keep-snapshots and felixd init --grant-trim, a holder now also deletes
    its inputs and its own .out, once its trim record has landed. A stream
    other workloads read goes only as far as the lowest floor of all of them:
    every workload in the workloads cache that lists it counts, in any state,
    by the input floor in the last trim record on its .out. A reader with no
    trim record, a spec that does not decode, or a failed read counts as 0 and
    holds the stream where it is. A workload's .out goes below the lower of its
    oldest kept snapshot's output tail and its readers' lowest floor, and only
    once its last trim record carries the first claim's spec; wl.admission.out
    and wl.scheduler.out are never cut. .out is trimmed at the tail the
    holder expects, so a superseded holder is refused; inputs are asked again at
    the tail Felix reports. A node whose token cannot trim these streams logs one
    warning of its own and still deletes .snap, the merge log and sys.facts.
    No record format changes. Breaking, and the risk of --grant-trim: a program
    outside felixd that reads an input or a workload's .out is not known to
    this rule and can find its history gone; a conditional publisher to an input
    is refused once per trim; and a workload submitted after the history was
    deleted starts at a record that is gone and is left to an operator
    (reason 9). felixd replay --diff of a cut .out compares from the input
    after the oldest kept snapshot; --from-start stops. A felixd replay to an
    offset older than the oldest snapshot it can restore, once the input before it is gone,
    says so and names the oldest offset it can reach, where it said it needed the
    stream from offset 0. Test agents add the new
    skip-shared-deletion fault by default. New felixd_core::readers.

  • Upgrading for deleted outputs and inputs (#150): upgrade every node before
    setting --keep-snapshots with the grant on. A node from before trim records
    carried the stream's lineage cannot read a .out whose head is cut, and
    leaves the workload to an operator.

  • If felixd init dies between appending admission's spec record and
    writing workloads, run it again: it writes the spec again and nodes pick
    admission up.

  • Upgrading for admission: run init only once every node runs this build,
    since an older node ignores admission. After init, every workload but the
    system ones needs a decision, so resubmit existing workloads through
    intake. Without init nothing changes. The
    requester, grants (#117) and quotas are not built; see
    Submitting work.

  • felixd keeps its Felix and control plane tokens out of what it writes
    (part of #74). They are held in a type whose Debug and Display print a
    placeholder, and a new integration test runs a workload through a failover
    with every log at trace level, then scans every record, cache entry, metric,
    log line and refused-credential error for the token bytes. Per-lease tokens
    wait on scoped token minting in Felix (GetFelix/felix#1625).

  • The requester (#182). Workload contract 0.3 adds publisher to record-in:
    the principal the broker authenticated an input record's write from, none
    when the broker stored none and for another workload's output. felixd runs
    0.3 and 0.2 workloads side by side, chosen by the interfaces a component
    exports; a 0.2 workload is unchanged and never sees the publisher, and
    wit/workload.wit moved to wit/0.2/workload.wit beside the new
    wit/0.3/workload.wit. The agent asks the broker for publishers and passes
    the stored value on every path (live, takeover, resume, merge, check,
    replay). Admission is built on 0.3 and records the requester in each
    decision (requester, left out when none). Breaking: a request whose
    publisher the broker did not record is rejected as no_requester, so a
    deployment that brings up the 0.3 admission needs a broker that stores
    publishers, FELIX_RECORD_PUBLISHERS=true on a lone broker or the
    publisher_principal fleet feature finalized in a cluster (one-way: storage
    format v6, which older brokers refuse), before felixd init. A deployment
    that already runs admission keeps its 0.2 module, which records no requester,
    until module upgrades exist: a workload's module cannot change once it has
    written output, and admission's id is fixed. Upgrade every node and the CLI
    before submitting a 0.3 module or running the 0.3 admission: a felixd from
    this release on leaves a module of a newer contract to an operator, and
    earlier builds fail the workload; an older felixd run cannot read the new
    reasons. Decision reason now tolerates a code a newer build wrote
    (unknown). New Policy::requesters limits who may ask (not_permitted; the
    admission-requesters example). felixd_runtime::Input and
    felixd_core::admission::decide take the publisher. The test stacks turn
    recording on.

  • A scheduler that is itself a felixd workload (#60). Submit
    scheduler.wasm (in the container image) as felixd run --id scheduler --input sys.specs --input sys.facts. It reads spec records and node facts
    and writes an assignment, workload to node, to wl.scheduler.out each time
    one changes, a progress record for a spec record that changes nothing, and
    its whole table every ten minutes of input time. Its clocks are its inputs'
    append times, so a replay assigns the same way, and it fails over like any
    workload. While it runs, each node appends a fact to the new sys.facts
    stream every --facts-every-ms (three lease TTLs) and within half a second
    of a change. A node follows assignments only while the scheduler's holder
    renews its lease: it claims what is assigned to it at once, leaves a
    workload to an assignee whose recent nodes entry says it can take it for
    at most --assignment-bound-ms (15 s), and waits for a scheduler that has
    not read a new spec yet for at most --assignment-grace-ms (2 s).
    Anything else is first claim wins at once. Claims still go through the lease and the
    fence. A drained node's workloads move to where the scheduler assigns them,
    and felixd ps shows an ASSIGNED column. Without the scheduler nothing
    changes. nodes entries gain blocked, set while a node cannot read its
    drain mark.

  • Upgrading: felixd run creates sys.facts. To use the scheduler, grant
    nodes publish on sys.facts and read on wl.scheduler.out. An older node
    ignores assignments and claims by first claim wins, which is safe; upgrade
    every node to get the placement. sys.facts is trimmed only with
    --keep-snapshots on the scheduler (#164, below).

  • felixd agent --metrics-bind <addr> (or FELIXD_METRICS_BIND) serves
    Prometheus metrics at /metrics on that address, and /live. Off unless
    given. Per node: leases held, claims by result, handoffs, workloads that
    need an operator by reason, failures, snapshots and their time. Per
    workload: held, steps and step latency, input lag per input, output
    records and fenced refusals, merge log appends, restarts and snapshot
    size. Workloads past --metrics-max-workloads (100) are reported together
    as workload="_other". The endpoint is plain HTTP with no authentication
    (#18, docs/metrics.md).

  • felixd ps shows each workload's state (running, waiting, needs operator
    with the reason, failed, stopped), holder, epoch, output stream tail and how
    far it is along each input, and each node, all read from records (#17).

  • Trim records carry the stream's lineage, and felixd check starts from a
    snapshot (#150). A trim record on the output stream now also holds the spec
    record of the stream's first claim, copied from trim to trim, and the highest
    spec record claimed so far. Once a cut head has taken the first claim, the
    agent, felixd run and felixd check read what the stream was written under
    from the oldest record Felix keeps, and the stale-spec check stops at a trim
    record. Trim records without the new fields still read. felixd check
    still replays from the first input; once the head of the output stream is
    deleted it restores the oldest snapshot the last trim record keeps and checks
    what follows it, the merge log from the entry at its step. --from-snapshot
    asks for that earlier and --from-start refuses it. A snapshot's state is
    taken as given and outputs below its tail are not compared.

  • felixd drain <node> makes a node release its leases, so other nodes take
    its workloads over without waiting for them to expire, and claim nothing
    until felixd undrain <node> (#17). felixd run creates the new drains
    cache. --wait waits until the node holds nothing and says it claims
    nothing, up to --timeout-secs.

  • Upgrading: a node must be able to read the drains cache, or it claims
    nothing. Grant node credentials read on drains before upgrading the
    nodes; felixd ps then shows a drain mark as pending on any node that
    has not said it read it.

  • felixd run --keep-snapshots K keeps a workload's last K snapshots. After
    each snapshot of its own, the holder appends a trim record to the output
    stream naming the floors below which this workload's resume reads nothing:
    on the snapshot stream, the merge log and each input. It is fenced by the
    lease epoch like an output. A shared input is trimmed only below every
    reader's floor (#150).

  • Upgrading: a trim record is a new output record kind (7). A node from an
    earlier build that takes over a workload with one fails it as poison, and
    one whose workload reads that output stream leaves it to an operator. Upgrade
    every node before setting --keep-snapshots.

  • Reads that walk back from a stream's tail stop at the oldest offset Felix
    holds instead of failing with CursorTooOld. With age or size retention on
    an input, a resume that finds its last stepped record gone has no append
    time for it, so the next snapshot_every_ms snapshot can fall on a
    different input than before. Outputs are unaffected.

  • A record a running workload needs that Felix no longer holds, after
    retention passed it, now leaves the workload to an operator (reason 9,
    history trimmed) instead of retrying. felixd replay and felixd check
    say which stream and offset were trimmed.

Added

  • The scale workflow takes cp_down=true (with election=on): it pauses the control plane, kills the broker leading the most shards, restarts it, and resumes the control plane, reporting cp_down_failover, cp_down_after_restart and cp_back (GetFelix/felix#1914).
  • felixd run --max-takeovers N --takeover-window-ms MS sets a restart intensity in the spec (#99).
    Every holder counts the takeovers in a row with no output between them from the claim records on
    wl.<id>.out, so every node agrees on the count. Past the limit the workload is recorded as
    needing an operator, reason 11 restart intensity exceeded, fenced by epoch, until a new spec
    record. Unset specs encode and run as before.

Changed

  • The scale workflow uploads each broker's and the control plane's whole log, gzipped, instead of the last 300 lines of each (felix #1914).

Fixed

  • A node reads a workload's output and snapshot streams before it takes a claim permit, so a few streams that are slow to be served after a failover no longer hold up every other claim.