Repository navigation
Releases
v0.1.0
The first full release (#231). It uses Felix 0.7.2 and needs a Felix 0.7 broker and control plane. The
compose files pin the 0.7.2 images and felixd builds against felix-client 0.7.2 (#227, #228). The code is
0.1.0-preview.2's, on Felix 0.7.2 instead of 0.7.0.
- On Felix 0.7.2 three brokers and three agents settle 1000 workloads: all running in 655 s with
one claim each, and a dead agent's workloads move in 29 s. On Felix 0.7.0-preview.2, 1000 did
not settle. A broker failure at that size is still slow. The numbers are in
docs/design.md.
v0.1.0-preview.2
This release uses Felix 0.7.0 and needs a Felix 0.7 broker and control plane. The compose
files pin the 0.7.0 images and felixd builds against felix-client 0.7.0. Upgrade every node
before using --upgrade-at (see below).
-
Module upgrades at a recorded input (#64).
felixd run --id <id> --upgrade-at <input> <module>
makes the new module the workload's from that input on. The spec keeps the module the workload
began with and lists each upgrade as an input and a module hash, so the lineage rules that
compare the module are unchanged and a spec without upgrades is written as before. The owner
hands its state to the new module through a snapshot and a restore in memory at the upgrade's
input, a node that resumes does the same, andfelixd checkandfelixd replayswitch at the
same inputs.felixd runrefuses an input the workload has already run, and a module that does
not declare it can restore the old one's snapshot layout. It writes an upgrade record to.snap
withpublish_ifat the tail it checked before the spec, and an owner on an older spec writes no
snapshot once it has seen one, so the old module cannot leave a snapshot past the upgrade.
A later spec that drops a recorded upgrade is refused. Upgrade every node before using
--upgrade-at: older builds cannot read the upgrade record. Upgrades are an operator action and
admission rejects requests that name them. -
The state classes table in
docs/design.mdcovers every cache and system stream felixd uses (#81), and a test fails when a new name has no row. It now lists theleases,epochs,workloadsandmodulescaches and thesys.intakestream. -
The scale test measures failover without waiting on its own harness. It scanned every output stream
in one pass with the whole phase as its budget, so one stuck read madeleaders_moved_secsand
without_outputmeasure the harness. Now a scan round takes at most 10 s, each read at most 3 s,
and the leaders are checked every round. The 300 s phase limit still bounds the whole wait. -
A value over Felix's frame limit fails with a clear error (#196). A 20 MiB cache value used to fail
aswrite frame payload: sending stopped by peerand a reconnect. felixd now measures cache puts
and publishes before sending and returnsTooLarge, naming the key or stream, the size and the
most Felix takes (about 12 MiB for a cache value, which travels as base64). An output append that
gets it marks the workload failed. Felix is unchanged. -
Agents degrade under load instead of tipping into lease churn (GetFelix/felix#1760). On a slow
cluster, leases lapsed together, every agent claimed them back at once, and the claims slowed
renewals further until nothing settled. Now a holder whose renewal is late pauses its writes at
two thirds of the TTL and carries on if that renewal is applied, instead of dropping the
workload. A renewal unanswered for a TTL counts as failed. A node makes at most 8 claims at
once, holds new claims back for up to TTL/3 while its renewals are slow, and waits before
claiming a workload it lost again (TTL/12 to TTL/6, doubling per loss in a row up to one TTL).
Lease and epoch rules are unchanged; these only delay claims and writes. -
Nightly scale run against Felix main (GetFelix/felix#1760).
scale-nightly.ymlruns the scale test at 1000
workloads on the:nightlyFelix images every night and opens an issue labelledfelix-nightly-scalewhen they
do not all start.Scaletakes afelix_image_taginput and can be called from another workflow. -
Fewer control plane requests per
felixd run(GetFelix/felix#1760). Since the read back
after a 409 (#149), each submit sent about nine extra requests to the control plane. A 409
that says the stream or cache already exists is no longer read back; any other 409 still
is. The control plane calls also share one connection. -
Operator docs (#24).
docs/operating.mdcovers running a cluster: what to start, the
configuration of every command, the grants each role needs, sizing the lease TTL, what failover
and replay guarantee, a three-node failover walkthrough, what eachpsstate asks of an
operator, and how to upgrade a node. -
Scale test (#20). The manually run
Scaleworkflow starts three Felix brokers and three agents, creates N
workloads and reports create time, memory, and how long recovery takes when an agent and then a broker is
killed (crates/felixd/tests/scale.rs,.github/workflows/scale.yml; see CONTRIBUTING.md). The numbers
are in design.md under "Scale": hundreds of workloads work on Felix 0.6.2 and 0.7.0-preview.2, 1000 often do not
settle on 0.7.0-preview.2 (about 17,000 claims in the 20 minute window, against about 1,100 over the 7 to 8
minutes 0.6.2 needed to get all running) and first output at 500 took 282 s there against 6.9 s on 0.6.2.
Creating streams (about 1.5 to 7 a second) and a broker failure are slow
(felix#1760,
felix#1761,
felix#1009). -
Divergence records (#73). A workload fails as poison when a node replays its
inputs and gets an output other than the one the stream holds. The node now
first appends adivergencerecord (output record kind 9: spec record,
input offset and index, the input of the snapshot the replay started from,
node id, contract, wasmtime and felixd versions, the sha256 of the output
the stream holds and of the one the replay produced, detail), fenced by epoch
like the failure record that follows it, which keeps its class. It is best
effort, and a deposed node writes neither record.felixd psprints it under
a failed workload,felixd replay --diffnames it when it finds the same
output differing, andfelixd_divergences_totalcounts them. Every reader
passes over it like a claim, and a node that cannot read one that is last on
the stream does not take the workload. A build from before kind 9 reads it
as an unknown kind: an old node that claims a failed workload after its spec
is submitted again fails the new spec, an oldps,checkorreplay
reports the stream unreadable, and a downstream workload run by an old node
needs an operator. Finish a rolling upgrade before submitting a failed
workload again. There is no switch. This closes #73;
defining the records in WIT waits for the effect records, and felixd does
not push OTLP. -
felixd agentshuts down cleanly on SIGTERM as well as Ctrl-C (#203). It stops its workloads and deletes
their leases, sodocker stopandsystemctl stophand work to another node at once instead of after
the lease TTL. As PID 1 in a container it no longer ignores SIGTERM. -
Degraded workloads (#73). A workload that keeps failing on a transient
error, such as a stream its node cannot read, used to be retried for ever
on that node, out of sight. Nowfelixd psshows it asdegradedwith the
error, and another node takes it. The holder gives up when attempts have
failed for--stuck-after-ms(default 60000, or the lease TTL if longer;
the agent refuses a value between 0 and the TTL) with no progress in
between, meaning no new input stepped and no record appended. It appends a
degradedrecord (output record kind 8: spec record, error class, node id,
detail) fenced by epoch, deletes its lease so another
node can claim it, and leaves the workload alone for
--degraded-backoff-ms(default 120000). The spec staysrunning, so
nothing has to be done.felixd_degraded_total{class}counts releases.
A build from before kind 8 refuses the record: it fails a workload whose
last record it is, never claims one whose first record it is, and a
downstream workload reading the stream needs an operator. During a rolling
upgrade run the new build with--stuck-after-ms 0, which turns the bound
off, until no old node is left. Counting degraded records across nodes and
blocking claims is #99. -
Modules over 8 MiB (#21). A module over 8 MiB is stored in the
modules
cache as 4 MiB chunks under<sha256>/<n>with a JSON manifest under the
module's own key, chunks first and manifest last. A module of 8 MiB or less,
which is all a build before this could hold, is stored as before. Readers
(nodes,felixd check,felixd replay, admission's projector) read four
chunks at a time, check each chunk and then the whole against the hash; a
missing chunk is themodule missingneeds-operator reason and a wrong one
fails the workload as poison. Admission's defaultmax_module_bytesand
felixd run's limit are now 64 MiB, up from 8 MiB. New
felixd_agent::module_storeandfelixd_core::moduleitems;
felixd-testkitgainspadded. -
Upgrading for module chunks (#21): upgrade every node before submitting a
module over 8 MiB, since an older node reads the manifest as a module that
does not match its hash. An admission module already running keeps the
8 MiB limit it was built with; to raise it, build the new admission module,
check it withfelixd replay --id admission --diff --all --module <hash>and
runfelixd initwith it. -
Refused stream creates (#149).
felixd run,felixd initandfelixd drain
read a stream or cache back after Felix answers 409 (unless it says the
resource already exists), and now exit with an
error carrying Felix's reason where they used to succeed and leave the next
read waiting for a stream that was never created. Felix refuses a stream
under the name of a deleted one until the cluster has finalized stream
incarnations; a control plane no broker registers with, like the
single-broker test stack, never finalizes them. Admission's projector hits the
same refusal when it creates an accepted workload's streams: it logs Felix's
reason and retries every second until the name can be reused, inste...
v0.1.0-preview.1
The first release. Tested against Felix 0.6.2, the version the compose files
pin.
What it does:
- Runs WebAssembly workloads on a node agent, with Felix as the only backend.
- Each workload has one owner at a time through a lease. A new owner fences the
output stream with a claim record, so a node frozen past its lease writes
nothing when it wakes, and each output lands once through lost answers,
partial batches and crashes. - Spec history,
felixd replayto rebuild a workload as of any input offset or
time, and--diffto check a replay or a new module against recorded outputs. - Pipelines, where one workload reads another's output, survive failover at
every stage. - Compose files for Docker and Podman.
- Reproducible release binaries for Linux x86_64, Linux aarch64 and macOS
aarch64, with a signedSHA256SUMS.
Not there yet: