Skip to content

Releases

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 15 Nov 00:12
a376215

The first full release (#231). It uses Felix 0.7.2 and needs a Felix 0.7 broker and control plane. The
compose files pin the 0.7.2 images and felixd builds against felix-client 0.7.2 (#227, #228). The code is
0.1.0-preview.2's, on Felix 0.7.2 instead of 0.7.0.

  • On Felix 0.7.2 three brokers and three agents settle 1000 workloads: all running in 655 s with
    one claim each, and a dead agent's workloads move in 29 s. On Felix 0.7.0-preview.2, 1000 did
    not settle. A broker failure at that size is still slow. The numbers are in
    docs/design.md.

v0.1.0-preview.2

v0.1.0-preview.2 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 09 Nov 21:00
ac34afa

This release uses Felix 0.7.0 and needs a Felix 0.7 broker and control plane. The compose
files pin the 0.7.0 images and felixd builds against felix-client 0.7.0. Upgrade every node
before using --upgrade-at (see below).

  • Module upgrades at a recorded input (#64). felixd run --id <id> --upgrade-at <input> <module>
    makes the new module the workload's from that input on. The spec keeps the module the workload
    began with and lists each upgrade as an input and a module hash, so the lineage rules that
    compare the module are unchanged and a spec without upgrades is written as before. The owner
    hands its state to the new module through a snapshot and a restore in memory at the upgrade's
    input, a node that resumes does the same, and felixd check and felixd replay switch at the
    same inputs. felixd run refuses an input the workload has already run, and a module that does
    not declare it can restore the old one's snapshot layout. It writes an upgrade record to .snap
    with publish_if at the tail it checked before the spec, and an owner on an older spec writes no
    snapshot once it has seen one, so the old module cannot leave a snapshot past the upgrade.
    A later spec that drops a recorded upgrade is refused. Upgrade every node before using
    --upgrade-at: older builds cannot read the upgrade record. Upgrades are an operator action and
    admission rejects requests that name them.

  • The state classes table in docs/design.md covers every cache and system stream felixd uses (#81), and a test fails when a new name has no row. It now lists the leases, epochs, workloads and modules caches and the sys.intake stream.

  • The scale test measures failover without waiting on its own harness. It scanned every output stream
    in one pass with the whole phase as its budget, so one stuck read made leaders_moved_secs and
    without_output measure the harness. Now a scan round takes at most 10 s, each read at most 3 s,
    and the leaders are checked every round. The 300 s phase limit still bounds the whole wait.

  • A value over Felix's frame limit fails with a clear error (#196). A 20 MiB cache value used to fail
    as write frame payload: sending stopped by peer and a reconnect. felixd now measures cache puts
    and publishes before sending and returns TooLarge, naming the key or stream, the size and the
    most Felix takes (about 12 MiB for a cache value, which travels as base64). An output append that
    gets it marks the workload failed. Felix is unchanged.

  • Agents degrade under load instead of tipping into lease churn (GetFelix/felix#1760). On a slow
    cluster, leases lapsed together, every agent claimed them back at once, and the claims slowed
    renewals further until nothing settled. Now a holder whose renewal is late pauses its writes at
    two thirds of the TTL and carries on if that renewal is applied, instead of dropping the
    workload. A renewal unanswered for a TTL counts as failed. A node makes at most 8 claims at
    once, holds new claims back for up to TTL/3 while its renewals are slow, and waits before
    claiming a workload it lost again (TTL/12 to TTL/6, doubling per loss in a row up to one TTL).
    Lease and epoch rules are unchanged; these only delay claims and writes.

  • Nightly scale run against Felix main (GetFelix/felix#1760). scale-nightly.yml runs the scale test at 1000
    workloads on the :nightly Felix images every night and opens an issue labelled felix-nightly-scale when they
    do not all start. Scale takes a felix_image_tag input and can be called from another workflow.

  • Fewer control plane requests per felixd run (GetFelix/felix#1760). Since the read back
    after a 409 (#149), each submit sent about nine extra requests to the control plane. A 409
    that says the stream or cache already exists is no longer read back; any other 409 still
    is. The control plane calls also share one connection.

  • Operator docs (#24). docs/operating.md covers running a cluster: what to start, the
    configuration of every command, the grants each role needs, sizing the lease TTL, what failover
    and replay guarantee, a three-node failover walkthrough, what each ps state asks of an
    operator, and how to upgrade a node.

  • Scale test (#20). The manually run Scale workflow starts three Felix brokers and three agents, creates N
    workloads and reports create time, memory, and how long recovery takes when an agent and then a broker is
    killed (crates/felixd/tests/scale.rs, .github/workflows/scale.yml; see CONTRIBUTING.md). The numbers
    are in design.md under "Scale": hundreds of workloads work on Felix 0.6.2 and 0.7.0-preview.2, 1000 often do not
    settle on 0.7.0-preview.2 (about 17,000 claims in the 20 minute window, against about 1,100 over the 7 to 8
    minutes 0.6.2 needed to get all running) and first output at 500 took 282 s there against 6.9 s on 0.6.2.
    Creating streams (about 1.5 to 7 a second) and a broker failure are slow
    (felix#1760,
    felix#1761,
    felix#1009).

  • Divergence records (#73). A workload fails as poison when a node replays its
    inputs and gets an output other than the one the stream holds. The node now
    first appends a divergence record (output record kind 9: spec record,
    input offset and index, the input of the snapshot the replay started from,
    node id, contract, wasmtime and felixd versions, the sha256 of the output
    the stream holds and of the one the replay produced, detail), fenced by epoch
    like the failure record that follows it, which keeps its class. It is best
    effort, and a deposed node writes neither record. felixd ps prints it under
    a failed workload, felixd replay --diff names it when it finds the same
    output differing, and felixd_divergences_total counts them. Every reader
    passes over it like a claim, and a node that cannot read one that is last on
    the stream does not take the workload. A build from before kind 9 reads it
    as an unknown kind: an old node that claims a failed workload after its spec
    is submitted again fails the new spec, an old ps, check or replay
    reports the stream unreadable, and a downstream workload run by an old node
    needs an operator. Finish a rolling upgrade before submitting a failed
    workload again. There is no switch. This closes #73;
    defining the records in WIT waits for the effect records, and felixd does
    not push OTLP.

  • felixd agent shuts down cleanly on SIGTERM as well as Ctrl-C (#203). It stops its workloads and deletes
    their leases, so docker stop and systemctl stop hand work to another node at once instead of after
    the lease TTL. As PID 1 in a container it no longer ignores SIGTERM.

  • Degraded workloads (#73). A workload that keeps failing on a transient
    error, such as a stream its node cannot read, used to be retried for ever
    on that node, out of sight. Now felixd ps shows it as degraded with the
    error, and another node takes it. The holder gives up when attempts have
    failed for --stuck-after-ms (default 60000, or the lease TTL if longer;
    the agent refuses a value between 0 and the TTL) with no progress in
    between, meaning no new input stepped and no record appended. It appends a
    degraded record (output record kind 8: spec record, error class, node id,
    detail) fenced by epoch, deletes its lease so another
    node can claim it, and leaves the workload alone for
    --degraded-backoff-ms (default 120000). The spec stays running, so
    nothing has to be done. felixd_degraded_total{class} counts releases.
    A build from before kind 8 refuses the record: it fails a workload whose
    last record it is, never claims one whose first record it is, and a
    downstream workload reading the stream needs an operator. During a rolling
    upgrade run the new build with --stuck-after-ms 0, which turns the bound
    off, until no old node is left. Counting degraded records across nodes and
    blocking claims is #99.

  • Modules over 8 MiB (#21). A module over 8 MiB is stored in the modules
    cache as 4 MiB chunks under <sha256>/<n> with a JSON manifest under the
    module's own key, chunks first and manifest last. A module of 8 MiB or less,
    which is all a build before this could hold, is stored as before. Readers
    (nodes, felixd check, felixd replay, admission's projector) read four
    chunks at a time, check each chunk and then the whole against the hash; a
    missing chunk is the module missing needs-operator reason and a wrong one
    fails the workload as poison. Admission's default max_module_bytes and
    felixd run's limit are now 64 MiB, up from 8 MiB. New
    felixd_agent::module_store and felixd_core::module items;
    felixd-testkit gains padded.

  • Upgrading for module chunks (#21): upgrade every node before submitting a
    module over 8 MiB, since an older node reads the manifest as a module that
    does not match its hash. An admission module already running keeps the
    8 MiB limit it was built with; to raise it, build the new admission module,
    check it with felixd replay --id admission --diff --all --module <hash> and
    run felixd init with it.

  • Refused stream creates (#149). felixd run, felixd init and felixd drain
    read a stream or cache back after Felix answers 409 (unless it says the
    resource already exists), and now exit with an
    error carrying Felix's reason where they used to succeed and leave the next
    read waiting for a stream that was never created. Felix refuses a stream
    under the name of a deleted one until the cluster has finalized stream
    incarnations; a control plane no broker registers with, like the
    single-broker test stack, never finalizes them. Admission's projector hits the
    same refusal when it creates an accepted workload's streams: it logs Felix's
    reason and retries every second until the name can be reused, inste...

Read more

v0.1.0-preview.1

v0.1.0-preview.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 30 Oct 03:54
14e43b2

The first release. Tested against Felix 0.6.2, the version the compose files
pin.

What it does:

  • Runs WebAssembly workloads on a node agent, with Felix as the only backend.
  • Each workload has one owner at a time through a lease. A new owner fences the
    output stream with a claim record, so a node frozen past its lease writes
    nothing when it wakes, and each output lands once through lost answers,
    partial batches and crashes.
  • Spec history, felixd replay to rebuild a workload as of any input offset or
    time, and --diff to check a replay or a new module against recorded outputs.
  • Pipelines, where one workload reads another's output, survive failover at
    every stage.
  • Compose files for Docker and Podman.
  • Reproducible release binaries for Linux x86_64, Linux aarch64 and macOS
    aarch64, with a signed SHA256SUMS.

Not there yet:

  • A workload reads one input only (#58).
  • No ps or drain (#17).
  • No snapshot or input trimming, so streams grow without bound (#19).
  • No metrics (#18).
  • Operator docs are still to come (#24).