Repository navigation
v0.1.0-preview.2
Pre-releaseThis release uses Felix 0.7.0 and needs a Felix 0.7 broker and control plane. The compose
files pin the 0.7.0 images and felixd builds against felix-client 0.7.0. Upgrade every node
before using --upgrade-at (see below).
-
Module upgrades at a recorded input (#64).
felixd run --id <id> --upgrade-at <input> <module>
makes the new module the workload's from that input on. The spec keeps the module the workload
began with and lists each upgrade as an input and a module hash, so the lineage rules that
compare the module are unchanged and a spec without upgrades is written as before. The owner
hands its state to the new module through a snapshot and a restore in memory at the upgrade's
input, a node that resumes does the same, andfelixd checkandfelixd replayswitch at the
same inputs.felixd runrefuses an input the workload has already run, and a module that does
not declare it can restore the old one's snapshot layout. It writes an upgrade record to.snap
withpublish_ifat the tail it checked before the spec, and an owner on an older spec writes no
snapshot once it has seen one, so the old module cannot leave a snapshot past the upgrade.
A later spec that drops a recorded upgrade is refused. Upgrade every node before using
--upgrade-at: older builds cannot read the upgrade record. Upgrades are an operator action and
admission rejects requests that name them. -
The state classes table in
docs/design.mdcovers every cache and system stream felixd uses (#81), and a test fails when a new name has no row. It now lists theleases,epochs,workloadsandmodulescaches and thesys.intakestream. -
The scale test measures failover without waiting on its own harness. It scanned every output stream
in one pass with the whole phase as its budget, so one stuck read madeleaders_moved_secsand
without_outputmeasure the harness. Now a scan round takes at most 10 s, each read at most 3 s,
and the leaders are checked every round. The 300 s phase limit still bounds the whole wait. -
A value over Felix's frame limit fails with a clear error (#196). A 20 MiB cache value used to fail
aswrite frame payload: sending stopped by peerand a reconnect. felixd now measures cache puts
and publishes before sending and returnsTooLarge, naming the key or stream, the size and the
most Felix takes (about 12 MiB for a cache value, which travels as base64). An output append that
gets it marks the workload failed. Felix is unchanged. -
Agents degrade under load instead of tipping into lease churn (GetFelix/felix#1760). On a slow
cluster, leases lapsed together, every agent claimed them back at once, and the claims slowed
renewals further until nothing settled. Now a holder whose renewal is late pauses its writes at
two thirds of the TTL and carries on if that renewal is applied, instead of dropping the
workload. A renewal unanswered for a TTL counts as failed. A node makes at most 8 claims at
once, holds new claims back for up to TTL/3 while its renewals are slow, and waits before
claiming a workload it lost again (TTL/12 to TTL/6, doubling per loss in a row up to one TTL).
Lease and epoch rules are unchanged; these only delay claims and writes. -
Nightly scale run against Felix main (GetFelix/felix#1760).
scale-nightly.ymlruns the scale test at 1000
workloads on the:nightlyFelix images every night and opens an issue labelledfelix-nightly-scalewhen they
do not all start.Scaletakes afelix_image_taginput and can be called from another workflow. -
Fewer control plane requests per
felixd run(GetFelix/felix#1760). Since the read back
after a 409 (#149), each submit sent about nine extra requests to the control plane. A 409
that says the stream or cache already exists is no longer read back; any other 409 still
is. The control plane calls also share one connection. -
Operator docs (#24).
docs/operating.mdcovers running a cluster: what to start, the
configuration of every command, the grants each role needs, sizing the lease TTL, what failover
and replay guarantee, a three-node failover walkthrough, what eachpsstate asks of an
operator, and how to upgrade a node. -
Scale test (#20). The manually run
Scaleworkflow starts three Felix brokers and three agents, creates N
workloads and reports create time, memory, and how long recovery takes when an agent and then a broker is
killed (crates/felixd/tests/scale.rs,.github/workflows/scale.yml; see CONTRIBUTING.md). The numbers
are in design.md under "Scale": hundreds of workloads work on Felix 0.6.2 and 0.7.0-preview.2, 1000 often do not
settle on 0.7.0-preview.2 (about 17,000 claims in the 20 minute window, against about 1,100 over the 7 to 8
minutes 0.6.2 needed to get all running) and first output at 500 took 282 s there against 6.9 s on 0.6.2.
Creating streams (about 1.5 to 7 a second) and a broker failure are slow
(felix#1760,
felix#1761,
felix#1009). -
Divergence records (#73). A workload fails as poison when a node replays its
inputs and gets an output other than the one the stream holds. The node now
first appends adivergencerecord (output record kind 9: spec record,
input offset and index, the input of the snapshot the replay started from,
node id, contract, wasmtime and felixd versions, the sha256 of the output
the stream holds and of the one the replay produced, detail), fenced by epoch
like the failure record that follows it, which keeps its class. It is best
effort, and a deposed node writes neither record.felixd psprints it under
a failed workload,felixd replay --diffnames it when it finds the same
output differing, andfelixd_divergences_totalcounts them. Every reader
passes over it like a claim, and a node that cannot read one that is last on
the stream does not take the workload. A build from before kind 9 reads it
as an unknown kind: an old node that claims a failed workload after its spec
is submitted again fails the new spec, an oldps,checkorreplay
reports the stream unreadable, and a downstream workload run by an old node
needs an operator. Finish a rolling upgrade before submitting a failed
workload again. There is no switch. This closes #73;
defining the records in WIT waits for the effect records, and felixd does
not push OTLP. -
felixd agentshuts down cleanly on SIGTERM as well as Ctrl-C (#203). It stops its workloads and deletes
their leases, sodocker stopandsystemctl stophand work to another node at once instead of after
the lease TTL. As PID 1 in a container it no longer ignores SIGTERM. -
Degraded workloads (#73). A workload that keeps failing on a transient
error, such as a stream its node cannot read, used to be retried for ever
on that node, out of sight. Nowfelixd psshows it asdegradedwith the
error, and another node takes it. The holder gives up when attempts have
failed for--stuck-after-ms(default 60000, or the lease TTL if longer;
the agent refuses a value between 0 and the TTL) with no progress in
between, meaning no new input stepped and no record appended. It appends a
degradedrecord (output record kind 8: spec record, error class, node id,
detail) fenced by epoch, deletes its lease so another
node can claim it, and leaves the workload alone for
--degraded-backoff-ms(default 120000). The spec staysrunning, so
nothing has to be done.felixd_degraded_total{class}counts releases.
A build from before kind 8 refuses the record: it fails a workload whose
last record it is, never claims one whose first record it is, and a
downstream workload reading the stream needs an operator. During a rolling
upgrade run the new build with--stuck-after-ms 0, which turns the bound
off, until no old node is left. Counting degraded records across nodes and
blocking claims is #99. -
Modules over 8 MiB (#21). A module over 8 MiB is stored in the
modules
cache as 4 MiB chunks under<sha256>/<n>with a JSON manifest under the
module's own key, chunks first and manifest last. A module of 8 MiB or less,
which is all a build before this could hold, is stored as before. Readers
(nodes,felixd check,felixd replay, admission's projector) read four
chunks at a time, check each chunk and then the whole against the hash; a
missing chunk is themodule missingneeds-operator reason and a wrong one
fails the workload as poison. Admission's defaultmax_module_bytesand
felixd run's limit are now 64 MiB, up from 8 MiB. New
felixd_agent::module_storeandfelixd_core::moduleitems;
felixd-testkitgainspadded. -
Upgrading for module chunks (#21): upgrade every node before submitting a
module over 8 MiB, since an older node reads the manifest as a module that
does not match its hash. An admission module already running keeps the
8 MiB limit it was built with; to raise it, build the new admission module,
check it withfelixd replay --id admission --diff --all --module <hash>and
runfelixd initwith it. -
Refused stream creates (#149).
felixd run,felixd initandfelixd drain
read a stream or cache back after Felix answers 409 (unless it says the
resource already exists), and now exit with an
error carrying Felix's reason where they used to succeed and leave the next
read waiting for a stream that was never created. Felix refuses a stream
under the name of a deleted one until the cluster has finalized stream
incarnations; a control plane no broker registers with, like the
single-broker test stack, never finalizes them. Admission's projector hits the
same refusal when it creates an accepted workload's streams: it logs Felix's
reason and retries every second until the name can be reused, instead of
recording a spec whose streams do not exist. -
Felix 0.7.0-preview.2. felixd builds on
felix-client0.7.0-preview.2 and the
compose files and test stacks default to the 0.7.0-preview.2 images. Nothing
changes for users. This release is the first with conditional head trims,
which later changes use to delete history. The client's cluster type is now
calledClient. -
Upgrade check from a snapshot (#29).
felixd replay --diff --moduleand
replay --to ... --modulestart the new module from the workload's nearest
snapshot (before--fromfor a diff, before the end of the range for--to)
when the new module'sfelixd.snapshotcustom section says it reads the
layout the workload's module wrote, and from the initial state when it does
not, saying which and why on stderr. The section is optional JSON,
{"format": "<id>", "reads": ["<id>", ...]}, additive to contract 0.2
(docs/contract.md, "Snapshot layouts"); a module without one is readable only
by itself. A new module that declares the layout and traps or runs out of
fuel restoring it is reported as "snapshot not readable by the new module",
not as a divergence. A section that does not parse is an error when it is
read, and the message points at--from-start. New--from-startalways
replays from the initial state.--moduletakes a component file as well as
the hash of one in themodulescache. The--toform now prints two lines
on stderr,started from ...andran N more of the inputs below E, where
it printed one. Infelixd_agent::replay,replayanddifftake
from_start,Replayedcarries aStartin place ofrestoredandDiffed
gainsstart(breaking for library users). Thecounterexample declares
its layout and has cargo features that build the variants the tests replay
against it;felixd_testkit::workload_withbuilds one. -
Broker host names and token rotation (#160, #159).
--brokerstakes
host:portas well asip:port; names are resolved on connect and again
when no known broker answers.--token-fileand
--controlplane-token-fileare read again each time the token is used, so a
token rotated in place keeps the agent's leases; a missing or half-written
file keeps the last token.felixd_agent::settings::Settingsholds
Seedbrokers andTokenSourcetokens (breaking for library users). -
Grants (#117).
felixd initwrites Felix RBAC policies for three roles
per namespace,role:felixd.<namespace>.node,.admissionand
.submitter, and writes nothing it finds already there; the operator
running it needsrbac.viewandrbac.policy.manage. Only admission's
credential may writesys.specsand create workload streams; a submitter
publishes requests and puts modules; a node holds no control plane rights.
Breaking: thesys.*streams move to the namespace<namespace>.sys, so a
node's grant on every workload stream does not reach them;felixd runand
felixd initcreate it.felixd init, which grants nodes publish on
<namespace>/*, refuses while an oldsys.*stream is in the workload
namespace, so that check is what protects an upgrade; agents do not check.
The migration steps are in docs/design.md ("Migrating the system streams").initalso
refuses, writing nothing, when a felixd role holds a rule beyond its
grants or one principal holds two roles, and needsrbac.viewover the
whole tenant. Namespaces ending in.sysare refused. Newfelixd agent --admission-token-file
(FELIXD_ADMISSION_TOKEN_FILE), the broker token admission's projector
writes with. A node reading a spec record past the tail ofsys.specsno
longer needs publish there. -
Admission (#114).
felixd init admission.wasmbrings up the admission
workload, reading the newsys.intakestream; it is the one spec still
written directly. From then onfelixd runuploads the module, publishes a
request onsys.intakeand prints admission's decision with how long it
took, exiting 1 on a rejection, and needs no control plane token. Admission
outputs one decision per request onwl.admission.out, accepting the spec
or rejecting it with a reason code (limits, module size, ids, inputs), and
decides from the request alone, sofelixd replay --id admission --diff --all --module <hash>lists the recorded requests a new policy decides
otherwise (--allis new). The node holding admission's lease records
each accepted decision onsys.specs, naming it, after checking the
module's hash and length and creating the streams; only nodes with a control
plane token hold admission. A node then runs a spec record written after
admission's own only if it names a decision that accepted exactly that
spec; any other needs an operator with the new reasonspec_unadmittedand
never claims. New faultcrash-after-projection-sent=Nfor tests. -
Resubmitting a workload a newer build left to an operator (#136).
felixd run
from an older build submits a new spec over a needs-operator record whose
reason it cannot read, which is how the workload is released. Nodes on the
older build leave the workload alone until a node that can read the record
takes it. Such a record also raises the highest spec claimed, as one it can
read does. -
System workloads (
admission,scheduler) are refused through intake;
felixd runwrites their specs directly with the control plane token as
before. A spec record runs only if no record of the workload anywhere on
sys.specsnames a later accepting decision, so an old acceptance cannot
run again, whether it is appended again orworkloadsis pointed back at
it. An accepted decision whose module does not match gets a skip record
(skipped), and a projector resumes after the newest decision any verified
record names, so an acceptance one passed over is never recorded later. -
Deleting snapshots and the merge log (part of #19). After the trim record
lands, the holder trims the workload's.snapand merge log with Felix's
conditional head trim, up to the record's floors, and a holder that resumes
deletes whatever a crash left. Inputs and.outfollow below (#150).
Needs the Felix fleet to have finalizedhead_trim, which is a one-way step
(back up first), andstream.trimon the node's token, which is off by
default:felixd init --grant-trimgrants it to the node role. Without
either, the node logs one warning and the workload runs on without deleting.
felixd checkandfelixd replayread.snapfrom the oldest offset Felix
holds, and stop with the stream and the oldest offset when they need a merge
log entry that was deleted. The compose files and the cluster test stack
finalizehead_trim. -
felixd init --grant-trim(part of #19). Grants the node rolestream.trim
on the namespace's streams, so nodes can delete history. Felix grants a
stream by its exact name or a whole namespace, so a node's credential can
then trim any stream in it, inputs and other workloads' outputs included,
and a trim erases history for every reader and cannot be undone. Without the
flaginitgrants no trim and nothing is deleted. Re-mint node tokens once
the fleet has finalizedhead_trim, since Felix puts the grant in tokens
only after that. Runninginitwithout the flag later keeps the rules;
remove them with the control plane's RBAC API. The flag also grants the node
rolestream.trimonsys.factsby exact name (#164), so the scheduler's
run can delete facts; without it the facts are kept. A namespace that ran
init --grant-trimbefore this needs it run again and the nodes' tokens
re-minted, and an older felixd'sinitrefuses a namespace that holds the
new rule. -
Trimming
sys.facts(#164). The first input that is deleted: a scheduler run
with--keep-snapshotsalso deletessys.factsbelow the floor its trim
record names for that input, the same step that deletes.snapand the merge
log. Only the scheduler readssys.facts, andfelixd runnow refuses a
system stream as the input of any other workload, as admission already did
(breaking for a workload submitted without admission: one already running
with a system stream is not rechecked, and parks as needing an operator once
the scheduler trims). Every node appends to the stream, so a
refused tail is asked again at the one Felix reports, up to three more
times, where.snapand the merge log give up. Aboutkeep-snapshots x snapshot-everyfacts stay, each about 150 bytes plus the ids of the
workloads its node holds (3 bytes and the id each): about 1 MB for
--keep-snapshots 3at the default when nodes hold 8 workloads.
felixd replayof the scheduler starts from a snapshot and is unaffected.felixd checkof a scheduler that has
deleted history says which stream it needed and the oldest offset Felix
holds, as it does for the merge log.wl.scheduler.outis never trimmed
(#150). -
Deleting inputs and
.out(the deletion half of #150). With
--keep-snapshotsandfelixd init --grant-trim, a holder now also deletes
its inputs and its own.out, once its trim record has landed. A stream
other workloads read goes only as far as the lowest floor of all of them:
every workload in theworkloadscache that lists it counts, in any state,
by the input floor in the last trim record on its.out. A reader with no
trim record, a spec that does not decode, or a failed read counts as 0 and
holds the stream where it is. A workload's.outgoes below the lower of its
oldest kept snapshot's output tail and its readers' lowest floor, and only
once its last trim record carries the first claim's spec;wl.admission.out
andwl.scheduler.outare never cut..outis trimmed at the tail the
holder expects, so a superseded holder is refused; inputs are asked again at
the tail Felix reports. A node whose token cannot trim these streams logs one
warning of its own and still deletes.snap, the merge log andsys.facts.
No record format changes. Breaking, and the risk of--grant-trim: a program
outside felixd that reads an input or a workload's.outis not known to
this rule and can find its history gone; a conditional publisher to an input
is refused once per trim; and a workload submitted after the history was
deleted starts at a record that is gone and is left to an operator
(reason 9).felixd replay --diffof a cut.outcompares from the input
after the oldest kept snapshot;--from-startstops. Afelixd replayto an
offset older than the oldest snapshot it can restore, once the input before it is gone,
says so and names the oldest offset it can reach, where it said it needed the
stream from offset 0. Test agents add the new
skip-shared-deletionfault by default. Newfelixd_core::readers. -
Upgrading for deleted outputs and inputs (#150): upgrade every node before
setting--keep-snapshotswith the grant on. A node from before trim records
carried the stream's lineage cannot read a.outwhose head is cut, and
leaves the workload to an operator. -
If
felixd initdies between appending admission's spec record and
writingworkloads, run it again: it writes the spec again and nodes pick
admission up. -
Upgrading for admission: run
initonly once every node runs this build,
since an older node ignores admission. Afterinit, every workload but the
system ones needs a decision, so resubmit existing workloads through
intake. Withoutinitnothing changes. The
requester, grants (#117) and quotas are not built; see
Submitting work. -
felixd keeps its Felix and control plane tokens out of what it writes
(part of #74). They are held in a type whoseDebugandDisplayprint a
placeholder, and a new integration test runs a workload through a failover
with every log at trace level, then scans every record, cache entry, metric,
log line and refused-credential error for the token bytes. Per-lease tokens
wait on scoped token minting in Felix (GetFelix/felix#1625). -
The requester (#182). Workload contract 0.3 adds
publishertorecord-in:
the principal the broker authenticated an input record's write from,none
when the broker stored none and for another workload's output. felixd runs
0.3 and 0.2 workloads side by side, chosen by the interfaces a component
exports; a 0.2 workload is unchanged and never sees the publisher, and
wit/workload.witmoved towit/0.2/workload.witbeside the new
wit/0.3/workload.wit. The agent asks the broker for publishers and passes
the stored value on every path (live, takeover, resume, merge,check,
replay). Admission is built on 0.3 and records the requester in each
decision (requester, left out when none). Breaking: a request whose
publisher the broker did not record is rejected asno_requester, so a
deployment that brings up the 0.3 admission needs a broker that stores
publishers,FELIX_RECORD_PUBLISHERS=trueon a lone broker or the
publisher_principalfleet feature finalized in a cluster (one-way: storage
format v6, which older brokers refuse), beforefelixd init. A deployment
that already runs admission keeps its 0.2 module, which records no requester,
until module upgrades exist: a workload's module cannot change once it has
written output, and admission's id is fixed. Upgrade every node and the CLI
before submitting a 0.3 module or running the 0.3 admission: a felixd from
this release on leaves a module of a newer contract to an operator, and
earlier builds fail the workload; an olderfelixd runcannot read the new
reasons. Decisionreasonnow tolerates a code a newer build wrote
(unknown). NewPolicy::requesterslimits who may ask (not_permitted; the
admission-requestersexample).felixd_runtime::Inputand
felixd_core::admission::decidetake the publisher. The test stacks turn
recording on. -
A scheduler that is itself a felixd workload (#60). Submit
scheduler.wasm(in the container image) asfelixd run --id scheduler --input sys.specs --input sys.facts. It reads spec records and node facts
and writes an assignment, workload to node, towl.scheduler.outeach time
one changes, a progress record for a spec record that changes nothing, and
its whole table every ten minutes of input time. Its clocks are its inputs'
append times, so a replay assigns the same way, and it fails over like any
workload. While it runs, each node appends a fact to the newsys.facts
stream every--facts-every-ms(three lease TTLs) and within half a second
of a change. A node follows assignments only while the scheduler's holder
renews its lease: it claims what is assigned to it at once, leaves a
workload to an assignee whose recentnodesentry says it can take it for
at most--assignment-bound-ms(15 s), and waits for a scheduler that has
not read a new spec yet for at most--assignment-grace-ms(2 s).
Anything else is first claim wins at once. Claims still go through the lease and the
fence. A drained node's workloads move to where the scheduler assigns them,
andfelixd psshows anASSIGNEDcolumn. Without the scheduler nothing
changes.nodesentries gainblocked, set while a node cannot read its
drain mark. -
Upgrading:
felixd runcreatessys.facts. To use the scheduler, grant
nodes publish onsys.factsand read onwl.scheduler.out. An older node
ignores assignments and claims by first claim wins, which is safe; upgrade
every node to get the placement.sys.factsis trimmed only with
--keep-snapshotson the scheduler (#164, below). -
felixd agent --metrics-bind <addr>(orFELIXD_METRICS_BIND) serves
Prometheus metrics at/metricson that address, and/live. Off unless
given. Per node: leases held, claims by result, handoffs, workloads that
need an operator by reason, failures, snapshots and their time. Per
workload: held, steps and step latency, input lag per input, output
records and fenced refusals, merge log appends, restarts and snapshot
size. Workloads past--metrics-max-workloads(100) are reported together
asworkload="_other". The endpoint is plain HTTP with no authentication
(#18, docs/metrics.md). -
felixd psshows each workload's state (running, waiting, needs operator
with the reason, failed, stopped), holder, epoch, output stream tail and how
far it is along each input, and each node, all read from records (#17). -
Trim records carry the stream's lineage, and
felixd checkstarts from a
snapshot (#150). A trim record on the output stream now also holds the spec
record of the stream's first claim, copied from trim to trim, and the highest
spec record claimed so far. Once a cut head has taken the first claim, the
agent,felixd runandfelixd checkread what the stream was written under
from the oldest record Felix keeps, and the stale-spec check stops at a trim
record. Trim records without the new fields still read.felixd check
still replays from the first input; once the head of the output stream is
deleted it restores the oldest snapshot the last trim record keeps and checks
what follows it, the merge log from the entry at its step.--from-snapshot
asks for that earlier and--from-startrefuses it. A snapshot's state is
taken as given and outputs below its tail are not compared. -
felixd drain <node>makes a node release its leases, so other nodes take
its workloads over without waiting for them to expire, and claim nothing
untilfelixd undrain <node>(#17).felixd runcreates the newdrains
cache.--waitwaits until the node holds nothing and says it claims
nothing, up to--timeout-secs. -
Upgrading: a node must be able to read the
drainscache, or it claims
nothing. Grant node credentials read ondrainsbefore upgrading the
nodes;felixd psthen shows a drain mark aspendingon any node that
has not said it read it. -
felixd run --keep-snapshots Kkeeps a workload's last K snapshots. After
each snapshot of its own, the holder appends a trim record to the output
stream naming the floors below which this workload's resume reads nothing:
on the snapshot stream, the merge log and each input. It is fenced by the
lease epoch like an output. A shared input is trimmed only below every
reader's floor (#150). -
Upgrading: a trim record is a new output record kind (7). A node from an
earlier build that takes over a workload with one fails it as poison, and
one whose workload reads that output stream leaves it to an operator. Upgrade
every node before setting--keep-snapshots. -
Reads that walk back from a stream's tail stop at the oldest offset Felix
holds instead of failing withCursorTooOld. With age or size retention on
an input, a resume that finds its last stepped record gone has no append
time for it, so the nextsnapshot_every_mssnapshot can fall on a
different input than before. Outputs are unaffected. -
A record a running workload needs that Felix no longer holds, after
retention passed it, now leaves the workload to an operator (reason 9,
history trimmed) instead of retrying.felixd replayandfelixd check
say which stream and offset were trimmed.
Added
- The scale workflow takes
cp_down=true(withelection=on): it pauses the control plane, kills the broker leading the most shards, restarts it, and resumes the control plane, reportingcp_down_failover,cp_down_after_restartandcp_back(GetFelix/felix#1914). felixd run --max-takeovers N --takeover-window-ms MSsets a restart intensity in the spec (#99).
Every holder counts the takeovers in a row with no output between them from the claim records on
wl.<id>.out, so every node agrees on the count. Past the limit the workload is recorded as
needing an operator, reason 11restart intensity exceeded, fenced by epoch, until a new spec
record. Unset specs encode and run as before.
Changed
- The scale workflow uploads each broker's and the control plane's whole log, gzipped, instead of the last 300 lines of each (felix #1914).
Fixed
- A node reads a workload's output and snapshot streams before it takes a claim permit, so a few streams that are slow to be served after a failover no longer hold up every other claim.