Skip to content

Releases: ztomer/multitop

v0.67.0

Choose a tag to compare

@ztomer ztomer released this 26 Oct 17:10

v0.67.0 — the agent's MCP server and the Ops view's client are rmcp (2026-10-26)

Changed — MCP is the official SDK on both ends (owner ruling OW.60, queue MCP.MON)

  • multitop-agent mcp is rmcp 3.7 instead of a hand-rolled JSON-RPC loop:
    2026-07-28 statelessly through server/discover, the older versions
    through initialize. The wire was pinned against the real binary first
    (crates/agent/tests/mcp_stdio.rs, green on 0.66.0 and on this release).
  • Every tool states readOnlyHint, destructiveHint, idempotentHint and
    openWorldHint, and declares an outputSchema; a destructive tool is never
    listed or called. goh's MCP wire gate now judges the server on every push.
  • The Ops view's client opens each session with rmcp's Discover lifecycle
    and sends every request through it; the kept tool list, the absent
    reasons and the catalogue digest are read exactly as before. Proven
    against mcp_host 1.99.25 on all three hosts and against the 0.66.0
    agent; an older multitop is answered by the new agent too.
  • Upstream's deltas: a line that is not JSON is dropped (was -32700); a
    stateless request without the 2026-07-28 _meta is -32602.
  • The musl agents grow (x86_64 774K to 1.6M, aarch64 700K to 1.4M): rmcp
    and its runtime, uploaded once per build per host.

Added — the phone reads the server (ROADMAP A2.5)

  • The paired screen of the Android companion shows the server's own count
    (/api/health) and one line per host (/api/hosts), read over the pinned
    TLS and nothing else; each line carries its state in a glyph and words.
    Proven end to end against a real multitop --serve the test starts.

Added — the Ops view draws the machine's verdict (servers ROADMAP O92)

  • machine.triage is the first row of every Ops card: what needs the
    owner's hand, one row per item naming its stack, with the owner's own fix
    quoted (or the plain fact that there is none) and the sources' words
    under it; then each stack's quiet line, saying what was read there.
    A cache kept by an earlier version does not parse and is asked again.

Fixed — a host nobody had read was counted healthy

  • /api/hosts gave a host with no reading "health": 100, and /api/health
    counted it in healthy: a server that had read nothing said every host was
    well. health is now absent until there is a reading, /api/health adds
    unread, and /api/hosts adds the host's name and breaching.

v0.66.0

Choose a tag to compare

@ztomer ztomer released this 22 Oct 01:20

v0.66.0 — the agent's tick is read from the cadence the estate publishes (2026-10-21)

Changed — the installed agent paces ITSELF, and the launchd job carries no interval (servers ROADMAP E53)

  • THE TICK WAS A NUMBER THIS REPO CARRIED, AND LAUNCHD FIXES IT WHEN
    THE JOB IS LOADED.
    The installed agent
    (com.ztomer.multitop.ops-poll) ran on a StartInterval 390 written by
    tools/install_ops_poller.sh — a number in this repo, and an interval
    launchd cannot change while a job is loaded. So the tick could not follow
    the estate: had the publication moved from every 15 minutes to every 5,
    the agent would still have read every 6.5 minutes — slower than the
    publication, so its verdict could be two publications behind — and the
    client's own warning would have fired on every tick, correctly, with
    nothing anyone could do about it but reinstall the agent.
  • THE AGENT NOW READS THE CADENCE AND WAITS THE RATE IT IMPLIES.
    multitop --ops-agent is the poll in a loop: one pass, then the tick.
    The tick is the shortest cadence any host published, halved (the
    rule, DECISIONS D15), less a five-second margin so the tick's own jitter
    can never land a second past the rule — and the cadence is read from what
    the estate published with its verdict (health://latest), through the
    answers this client already keeps. A publication that moves is followed a
    cycle later, with no reinstall and nothing for the operator to do.
  • THE INSTALLED JOB CARRIES NO INTERVAL AT ALL. The launchd job the
    template writes keeps the agent ALIVE (RunAtLoad + KeepAlive) and
    nothing else: there is no rate in this repo for the estate's cadence to
    move under. --interval is REFUSED by the template — a rate handed
    to it is the defect the tick was raised about — and the only number in the
    job is the restart throttle for a failed agent, which says nothing about
    how often the agent reads. Installing over a job that carried one says
    what is being dropped: "the job being replaced carried an interval of
    390s - dropped: the agent paces itself by the published cadence"
    .
  • THE OTHER WAY ROUND WAS NOT CHOSEN, AND WHY. The tick could have been
    re-installed when the cadence moved: the agent rewrites its own plist and
    re-bootstraps it. That was not chosen because bootout kills the job
    that is running it — the agent would have to be killed from inside its
    own pass — and because a failed swap leaves the estate with no agent at
    all
    , which is a worse failure than a tick that is one cycle behind. The
    other way round is now impossible to get wrong: there is no interval to
    install, and no way to ask for one.
  • A CADENCE THE AGENT COULD NOT READ IS SAID, AND THE RATE IS KEPT. A
    host that published no cadence keeps the rate this client was keeping for
    it, and the log says so in the client's own words ("the cadence could not
    be read: this client keeps its rate"
    ); a host this client has never read
    keeps the client's fallback rate, and the log says that too. Neither is
    called a published rate, because neither was published — and the tick's
    own line names the cadence it followed, the host it read it from, and how
    old that reading is.
  • THE AGENT PICKS UP A REPLACED BINARY. The agent is a long-lived
    process — that is what lets it pace itself — so an upgraded binary would
    otherwise leave it running the old one, and the old one keeps the old
    rule. It hands the process over to the binary that is installed (exec),
    so the estate is read by the binary on disk and the launchd job keeps its
    identity.

Measured (2026-10-21, on the live estate)

  • BEFORE — the tick was the launchd interval this repo carried.
    Measured on this machine from the at each kept answer carries (the
    epoch second the PASS STARTED — the moment the client itself measures
    its sampling rate against): 396 s, against a StartInterval of 390.
    The same tick read from the cache's write times: 392 s, 393 s, 395 s,
    395 s, 395 s
    — the write lands when the gather lands, and the gather
    takes a different number of seconds each cycle.
  • AFTER — the tick is the rate the published cadence implies: 445 s
    (the rule, 450 s, less the five-second margin), measured the same way
    from the at the kept answers carry: 446 s, 446 s (the tick is
    445 s, and the second is the sleep's own overshoot).
  • THE CADENCE IT FOLLOWS, AND WHERE IT READ IT FROM — the agent's own
    log line, every cycle:
    tick every 7.4 min (445 s) · the shortest cadence any host published: every 15 min (900 s), read from health://latest on 192.168.0.33, 5 s ago
    — the 900 s is the estate's own published cadence (cadence_s in
    health://latest), not a number this repo carries, and the client keeps
    it at rule_every_s: 450 — so its own warning does not fire.
  • THE FALLBACK, MEASURED — the local host publishes no cadence (it has
    no health suite), so the agent's log carries, every cycle:
    127.0.0.1 · the cadence could not be read: this client keeps its rate (every 7.5 min) — the rate it was keeping, said in words, and not passed
    off as a published rate.
  • THE HANDOVER, MEASURED — the agent noticed the installed binary had
    been replaced under it and said so: the installed binary was replaced: this agent is now the new one.

v0.65.0

Choose a tag to compare

@ztomer ztomer released this 21 Oct 20:44

v0.65.0 — a timed-out call reads as UNKNOWN, and the estate's other readings stand (2026-10-21)

Fixed — the poll's timed-out call is a GAP in the reading, not a fault of the estate (servers ROADMAP E74)

  • THE LOCAL HOST'S READING WAS THROWN AWAY WITH THE CALL THAT MISSED
    ITS DEADLINE.
    The launchd poll's local host failed intermittently with
    tools/call: no answer in 20s — 17 of 125 passes in the poll's own
    log (measured 2026-10-21: 9 at a tools/call deadline and 8 at a
    server/discover one, so 13.6% of the passes). A missed deadline
    ENDED the session, so the host's whole reading was discarded, the host
    was reported as failed, and the sections the estate had answered were
    never asked. The estate's health is read from these answers, so an
    intermittent local failure was an intermittent blind spot — and it
    was read as "a flaky tool", which is a failure that is not a fault.
  • A MISSED DEADLINE IS A GAP IN ONE ANSWER, AND THE SESSION SURVIVES
    IT.
    A late answer carries the id of the request it answers, and this
    client matches that id — so a late answer is read and discarded, never
    taken for the next one. The old reason for ending the session ("a late
    reply could be mistaken for the next one's") could not happen, and
    paying for it was the whole of E74: the estate's other readings are
    read again. A test drives a server that answers late and reads the
    NEXT answer back, to keep the id match honest.
  • THE SECTION THAT MISSED IT READS UNKNOWN, AND NAMES THE CALL. The
    frame draws ⚠ containers unknown: docker.containers: no answer in 20s — the TOOL is named, where the old wording named the JSON-RPC
    method (tools/call: no answer in 20s) and so named no tool at all: a
    log of timeouts said nothing about which call had missed its deadline.
    A section is now in one of three states and they are not the same:
    ABSENT (the host does not offer it, and says why), FAILED (the
    estate's own report — failed: ... — a finding), UNKNOWN (the
    answer did not arrive — unknown: ... — a gap, never an all-clear and
    never a fault of the estate).
  • THE HOST IS NOT FAILED, AND THE PASS IS NOT ALL-CLEAR. The pass
    gathers the host, keeps the reading, and names the fact that is
    missing: 127.0.0.1 · gathered · containers unknown: docker.containers: no answer in 20s. The pass's exit code is still a failure while a gap
    stands — a gap is not an all-clear, and a fresh answer that still
    carries one says so too.
  • THE 20-SECOND BUDGET IS THE ESTATE'S, AND IT WAS NOT RAISED. The
    deadline is the estate's own (OPS_TIMEOUT), the agent was not given a
    longer or a second one of its own, and a missed deadline is reported
    rather than waited out.

Measured (2026-10-21, on the live estate)

  • THE LOCAL CALLS ON AN AWAKE MACHINE NEVER MISS IT. 100 runs of the
    local agent's docker.containers: max 5.039 s, 1 over the
    agent's own 5-second IO timeout (the daemon did not answer, and the
    tool said so), 0 over 20 s. 100 runs of cron.status: max
    0.094 s
    , 0 over 20 s. The 20-second timeouts in the poll's log
    are therefore not the call being slow — they are the machine not
    answering within the estate's budget
    , which is also why 8 of the 17
    failures were a server/discover deadline: the agent did not even
    answer the first request.
  • THE SAME ESTATE, THE SAME TIMEOUT, BEFORE AND AFTER. With a local
    agent that answers everything except docker.containers, on the live
    estate:
    • before (v0.64.0): 127.0.0.1 · failed: tools/call: no answer in 20s, exit 1, and no answer file at all — the reading was
      discarded, and the card went stale;
    • after (this build): 127.0.0.1 · gathered · containers unknown: docker.containers: no answer in 20s, exit 1, and the answer is
      kept — cron a value, containers {"Unknown": "docker.containers: no answer in 20s"}, and the absences keep their own reasons.
    • the frame it draws: ✓ cron 1 job ok beside ⚠ containers unknown: docker.containers: no answer in 20s.

Verified

  • tools/gate.sh --full green (14/14); the Rust suite and the pytest
    suite as the gate of record ran them.
  • The red before the fix: the four tests that carry E74 failed on the
    shipped behaviour — "tools/call: no answer in 200ms" (the tool not
    named), "a missed deadline does not end the session", "the section
    that could not be read does not say so"
    , and "the host was thrown
    away with the call that timed out"
    .

v0.64.0

Choose a tag to compare

@ztomer ztomer released this 21 Oct 19:02

v0.64.0 — the kept catalogue is checked against the server's own DIGEST (2026-10-21)

Changed — the kept list is checked against the server's WORD, not a guess (servers ROADMAP E72)

  • THE KEPT LIST WAS CHECKED BY A GUESS. The client decided that its
    kept copy of a host's tool list was still the truth by INFERring it from
    two things it could see cheaply: the server's advertised version and
    the set of tools the server says it does not offer. Both are hints,
    not facts — and the guess was wrong often enough to measure: it sent the
    client back for the whole catalogue at every release, and four of the
    last ten releases touched no tool table at all
    (v1.99.13→14, 15→16,
    16→17, 17→18), so those downloads were needless — and, the other way
    round, a table that moved WITHOUT a release was invisible to it.
  • THE SERVER NOW PUBLISHES THE DIGEST OF THE EXACT LIST tools/list
    ANSWERS
    , in server/discover, beside the absent map this client
    already reads: dev.multitop/catalogueDigest, the SHA-256 of the
    list's own canonical bytes, in the estate's spelling (sha256: and 64
    lowercase hex). That is the server's own word about the list it serves,
    and it is the fact the client checks its copy against — the server's
    half is media_server 7c3b02b1, shipped there and not written here.
  • A DIGEST THAT MATCHES KEEPS THE KEPT LIST — ACROSS A RELEASE. The
    version moving no longer costs a download: the digest says the list is
    byte-for-byte the one the client holds, so a release that touched no
    tool table leaves the catalogue alone (149,997 B a gather measured
    on the live estate today). The client asks for the list only when the
    digest says the list moved.
  • A DIGEST THAT CHANGED PUTS THE KEPT LIST ASIDE, and the host is
    asked again — and this is the case the guess could not see at all: the
    same binary, the same absent set, a different digest. The frame says
    so: "tool list: re-listed, the server's catalogue digest changed".
  • THE DIGEST IS REMEMBERED, NOT COMPUTED. It is of the whole list
    — names and schemas — and this client keeps only the names, so it
    cannot compute it again: it keeps the digest it was handed, beside
    the list the word was given for. A digest over the names alone would be
    a smaller fact and a weaker one (a schema change would go unseen); the
    server publishes the stronger one, and the client takes it.
  • A DIGEST THE CLIENT CANNOT CHECK SAYS THE FACT IS ABSENT. A server
    that publishes none — one too old to know the field, or the local
    agent, which publishes none — has said nothing about its list: the kept
    copy is then checked the OLDER way (the version and the absent set),
    the frame says "the server published no digest", and that is not an
    all-clear
    : an absent fact is stated, never drawn as a copy-is-fine. A
    digest that is not in the estate's spelling is not a digest either, and
    is treated the same way.
  • AND THE CLIENT DOES NOT RE-LIST EVERY SESSION. The absent digest
    does not turn into a download on every pass — that is the defect this
    closes, not a new one. A kept list taken before the server published a
    digest is put aside once (it cannot be checked), and the digest is
    remembered with the list from then on.
  • THE KEPT LIST IS STILL THE WHOLE LIST — an optimisation, never a
    filter (E61's rule): every tool of it, every page of it, by name.

Measured (2026-10-21, on the live estate and on the shipped binary)

  • THE ESTATE'S HALF OF THE BYTES: the digest costs +105 B per
    session per host
    on the wire — measured on the live estate, the
    server/discover answer grew by exactly that on every host when the
    fleet shipped v1.99.21: 667 → 772 B on .33, 1,097 → 1,202 B on
    .90, 1,711 → 1,816 B on .158 — and it saves the 149,997 B a
    gather
    the version inference forced at every release: the three
    tools/list answers, 65,557 B on .33, 51,716 B on .90, 32,724 B on
    .158. A release that touched no tool table now costs 315 B (three
    digests) instead of 149,997 B (three catalogues): 149,682 B a
    gather saved.
  • THE CLIENT'S HALF, ON DISK: the kept list grows by 87 B per host
    (672 → 759 B on .33 — the 71-byte digest and the JSON around it) and
    the answer the poll keeps grows by 108 B per host per gather (its
    catalogue field, 39 → 147 B — it now carries the proof). Neither is on
    the wire.
  • THE PASS, WITH AND WITHOUT THE KEPT LISTS, measured with the
    counting shim in front of ssh (tools/measure_poll.py): 149,024 B
    a gather
    with the kept list (no tools/list line in the per-answer
    table at all) against 299,023 B with them dropped — the same
    149,997 B of catalogue, measured twice.
  • AND THE FRAME SAYS IT, ON THE LIVE ESTATE: "tool list: kept 24m
    old (window 1h), the server's digest matches"
    .
  • THE MIGRATION, ON THE LIVE ESTATE: the fleet shipped the digest
    while this was being written, and the lists this client had kept under
    the older servers carried no digest — so they were put aside ONCE
    (the one path where a kept copy cannot be checked at all), and the
    digest was remembered with the list from then on. Measured on .158:
    the answer said Relisted/Uncheckable and the frame said "the kept
    list carried no digest to check against"
    ; the next session said
    "tool list: kept 4s old (window 1h), the server's digest matches",
    and the host was not asked for its list again.

Verified

  • tools/gate.sh --full green; the Rust suite green.
  • The tests were red first: a changed digest did not put the kept list
    aside, a release that changed no tool re-listed the catalogue anyway,
    the digest was not kept with the list, and the frame said nothing about
    the digest — all four failed against the old code, and pass now.

v0.63.0

Choose a tag to compare

@ztomer ztomer released this 21 Oct 16:52

v0.63.0 — the Ops poll stops re-downloading the TOOL CATALOGUE (2026-10-21)

Added — the tool list is kept, keyed to what determines it (servers ROADMAP E61)

  • THE OPS POLL PULLED THE WHOLE CATALOGUE, EVERY GATHER. The list of
    tools a host offers is the biggest thing this client pulls off a host —
    65,467 B on .33, 51,716 B on .90, 32,724 B on .158, measured on
    the live estate — and it is the SAME list every time: it changes at a
    RELEASE, not every six and a half minutes. At the scheduler's own rate
    (a pass every 390 s) that is 149,907 B a gather, 33.1 MB a day, of
    one immutable list, over and over.
  • THE KEY IS (the server's advertised version, the set of tools it says
    it does NOT offer)
    , and BOTH are read from the LIVE server on every
    session, so the kept list cannot hide a change: a release moves the
    version, and a host that gains or loses a tool moves the absent set. A
    kept list taken under another binary, or under another tool set, is put
    aside and the host is asked again — and the reasons the server gives for
    what it does not list are read live every session, so they are never
    the kept copy's.
  • THE SERVER'S OWN ttlMs BOUNDS IT — an hour, for a list — so this
    client never keeps an answer longer than the answer's owner says it is
    good for. A window of zero is the protocol's "a live fact": there is
    nothing to keep, so there is nothing to use, and nothing is kept. A
    server that names no version is not keyed at all, so nothing is kept
    either: the version is the only thing that says which binary holds the
    tool table.
  • THE CACHE IS AN OPTIMISATION, NEVER A FILTER. What is kept is EVERY
    tool the server listed, every page of them: a tool the kept list omitted
    would lose its section from the panel, and a list that kept only the
    first page would lose every tool on the second.
  • WHERE THE LIST CAME FROM IS ON THE PANEL. The list is the panel's
    SHAPE — a tool the list does not name has no section — so the frame says
    it: "tool list: kept 12m old (window 1h)", "tool list: from the
    server (window 1h)"
    , and, when a kept list was put aside, WHY: "tool
    list: re-listed, the kept list was for 1.99.19, the server is 1.99.20"
    ,
    "… the kept list had expired (2h old, window 1h)". A list nobody can
    date is a shape nobody can trust, and a list quietly replaced would hide
    the very thing the operator needs to know.

Fixed — the kept lists cannot be taken for the answers

  • THE OTHER CLIENT READS THIS CACHE DIRECTORY TO FIND WHAT THE MONITOR
    KNOWS
    , and it takes the FIRST file whose name holds the host — so a
    tool list kept beside the answer, named for the same host, would have
    been taken for the answer itself, and the parity check would have read a
    tool list where it expects the estate's own answers. The lists live in
    the cache directory's own catalogue/ directory now: out of that
    reader's way, and the directory still holds one answer per host and
    nothing else.

Measured, on the live estate (2026-10-21)

  • THE LIST, A GATHER: tools/list costs 65,467 B on .33,
    51,716 B on .90 and 32,724 B on .158 — 149,907 B a gather
    across the estate.
  • THE PASS, BEFORE AND AFTER, measured on the shipped binary with a
    counting shim in front of ssh (tools/measure_poll.py): the whole
    poll cost 298,728 B a gather (66.0 MB a day) with the released
    client, and 148,817 B a gather (32.9 MB a day) with the kept list —
    and the kept list is pulled an hour, not a gather (3.6 MB a day). So
    the poll's inbound bytes fall from 66.0 MB a day to 36.5 MB a day,
    and the catalogue's own share from 33.1 MB a day to 3.6 MB a day.
  • THE FALL IS THE CATALOGUE, TO THE BYTE: 298,728 − 148,817 =
    149,911 B a gather, against the 149,907 B the three tools/list
    answers cost on their own — and the warm pass's per-answer table has no
    tools/list line in it at all. The discover answer (667–1,711 B) is
    still read every session, because that is what says whether the kept
    list is still the truth.
  • A RELEASE STILL RE-LISTS: every kept list was given another binary's
    version, and the next pass asked every host for its list again — the
    whole poll's bytes back to the cold figure (298,726 B) — and the
    client said why, in its own answer ("Relisted — the kept list was for
    1.99.19-before-a-release, the server is 1.99.19"
    ) and on the panel
    ("tool list: re-listed, the kept list was for 1.99.19-before-a-release,
    the server is 1.99.19"
    ).

Reported, not written — the better place may be the server

  • THE CATALOGUE IS A PER-BINARY CONSTANT, AND THE SERVER KNOWS IT. The
    estate's plan says the better place for this may be the server
    (media_server's lane), and it is: the tool table is fixed in the
    binary, so the server could publish the list's DIGEST beside the absent
    map it already publishes in server/discover — a few dozen bytes — and
    then any client could check its copy against the server's own word
    instead of inferring it from the version and the absent set. That is
    reported to the estate's plan; nothing was written in that repo.

Verified

  • tools/gate.sh --full green; the Rust suite green.
  • The kept lists are the client's own copy and nothing else: the cache
    directory still holds one answer per host, and the tool lists live in
    catalogue/, where the reader that looks for the answers does not look.

v0.62.0

Choose a tag to compare

@ztomer ztomer released this 21 Oct 10:29

v0.62.0 — the rate is READ from the estate, and the Ops frame can be PRINTED (2026-10-21)

Changed — the client's sampling rate (servers ROADMAP E53)

  • THE RATE WAS A NUMBER COPIED INTO THIS CLIENT. MAX_SAMPLING_INTERVAL_S
    (450 s) was half of a fifteen-minute publication, written down here, and
    the due rule and the staleness threshold were written against it. The
    estate's cron can move — and if it moved to every five minutes, this
    client would have kept sampling every 7.5 minutes and neither of its
    own staleness thresholds would have noticed. The server now publishes its
    publication cadence with the verdict (health://latest, beside the age:
    cadence_s and cadence), and this client derives its rate from
    what it read: half of the cadence the server published.
  • THE WARNING IS DRIVEN BY THE PUBLISHED CADENCE, not by the local
    copy: a client that samples slower than half the cadence the server
    published says so where the verdict is shown — "sampled every 15 min,
    server publishes every 5 min: two publications behind"
    . The age on the
    line cannot say it: a verdict two publications behind looks exactly like
    a fresh one.
  • A CADENCE THE SERVER COULD NOT READ IS NOT A WARNING. It is a gap in
    the reading, not a fault: the client says the rate could not be read
    ("the cadence could not be read: this client keeps its rate (every 15
    min)"
    ) and keeps the rate it had.
  • THE STALENESS THRESHOLD FOLLOWS THE CADENCE TOO — the published
    cadence plus a grace for a suite that is still running — so a verdict is
    stale against the estate's own publication, not against a number typed
    here.
  • THE CACHE CARRIES THE RULE AND THE TRUTH. A kept answer holds the
    rate the client KEEPS (the rule) and the rate the answer was TAKEN AT
    (the rule, raised to the gap when the client missed its schedule). A
    missed cycle is not a new rate: a client that missed one must not keep
    sampling at half the rate for ever.

Added — the Ops frame can be printed (servers ROADMAP O90)

  • multitop --ops-frame <HOST> prints the Ops frame for one host,
    uncoloured: the same lines the Ops view draws, from this client's own
    answers (the cache the poll writes), and it asks the estate for nothing.
    The host is named by the name in its [[servers]] entry or by its
    address. The other client's parity check (routines parity) reads it
    through --monitor-cmd, so the two renderings can be compared without
    reading a terminal — the monitor draws its Ops view in its TUI and keeps
    no copy of it, so the check had to read the monitor's screen and say out
    loud that the rendered half was a stand-in.

Fixed — a machine that declares nothing is not a fault (servers ROADMAP E26)

  • THE vpn LINE READ A HEALTHY MACHINE AS A FAULT. The server's own
    reason for not offering net.egress reads "the machine's instance
    declaration is missing or unreadable (…): its instances are unknown"
    ,
    and the monitor repeated it. E26's rule is that a machine which declares
    nothing has one instance — itself — and says so; the routines card says
    "1 stack: adguard_server · no declaration". The monitor now says what
    it knows, in the card's words, and a declaration that EXISTS and could
    not be read still keeps the server's own reason — that one IS a fault.

Measured, on the live estate (2026-10-21)

  • THE CADENCE IS PUBLISHED, on all three hosts: cadence_s: 900,
    cadence: "every 15 min" (.33, .90 and .158, read over ssh with
    resources/read health://latest). The client's rate is therefore
    450 s, and the answers say so: rule_every_s: 450.
  • THE GATHERS A DAY: 221 per host — the launchd agent's tick is 390 s
    and every tick finds a gather due (measured over six ticks:
    390–391 s between them), against the rule's own floor of 192
    (450 s).
  • THE BYTES A DAY: 60.3 MB across the estate, from 153,111 B
    (.33), 77,273 B (.90) and 42,341 B (.158) per gather —
    measured with python3 tools/measure_gather.py <label> <ssh-target> <mcp-command>, which asks the host's own mcp_host over ssh for exactly
    the requests the gather makes and counts the bytes each reply crossed the
    wire for.
  • THE FRAME FOR A HOST WITH NO DECLARATION (.158), from
    multitop --ops-frame pihole:
    · vpn no tunnel on this machine: adguard_server, no declaration
    — where it read "no Docker on this machine: no daemon socket; the
    machine's instance declaration is missing or unreadable
    (/home/ztomer/prj/adguard_server/config/instances.tsv): its instances are
    unknown"
    . The machine's own facts are on the line above it
    (· stacks adguard_server: 1 instance, no declaration), and the
    containers section still says whether the machine has Docker at all.

Verified

  • tools/gate.sh --full green (14/14); the Rust suite (1,794 tests across
    the workspace) and the pytest suite as the gate of record ran them.
  • The other client's parity check now reads the monitor's RENDERED frame
    (routines parity --monitor-cmd "multitop --ops-frame"): 23
    differences across 3 hosts
    , every one named, and the note that the
    rendered half was a stand-in is gone.

v0.61.1

Choose a tag to compare

@ztomer ztomer released this 21 Oct 06:11

v0.61.1 — the poll's rate lands INSIDE the rule, whatever the scheduler does (2026-10-21)

Fixed — the poll's schedule

  • launchd's timer wheel has a granularity of its own. Measured on this
    Mac on 2026-10-21: a 420 s StartInterval fired 425 to 470 s after
    the last run, so the client's read rate was sometimes 7 minutes 50
    seconds — outside the rule's 7.5, and the card would have said so on the
    days it drifted. The agent's tick is now 390 s, which puts every
    gather between 6.5 and 7.5 minutes: inside the rule every time.
  • THE DUE RULE FOLLOWS THE SCHEDULER. The age a host must reach to be
    gathered is now 90 s under the poll's interval, so a tick that lands early
    still finds a gather DUE: the client's read rate cannot halve because a
    second was lost.
  • AN ANSWER SAYS WHEN THE CLIENT MISSED ITS SCHEDULE. The rate an
    answer carries is the poll's own — unless the gap since the last answer
    is LONGER, in which case the gap is the truth. A client that slept
    through a cycle says so, rather than carrying a rate it did not keep.

Measured, on the live estate (2026-10-21, after the fix)

  • The gathers: every 390–450 s on .33 (measured: 406 s between
    two gathers), against the retired rate's 30 s.
  • The answer carries 450 s, so the aliasing warning stays silent
    because the client IS inside the rule — and it fires the moment the
    client is not.

Verified

  • tools/gate.sh --full green (14/14); the Rust suite (1,773 tests across
    the workspace) and the pytest suite as the gate of record ran them.

v0.61.0

Choose a tag to compare

@ztomer ztomer released this 21 Oct 05:35

v0.61.0 — the Ops poll is a JOB: it runs whether or not anyone is watching (2026-10-21)

Changed — the Ops poll (servers ROADMAP E36)

  • THE POLL NEVER RAN. It was spawned by the Ops VIEW's keypress and by
    nothing else, so the estate's fingerprint tools fired 0.3 a day while
    the shared tools fired 87: the Ops view was shipped, installed, and nothing
    asked the estate for it. The code said so out loud — "Polled every 30 s
    while the view is up, and never while it is not"
    — so this was a design,
    not a bug, and the design was the defect.
  • THE POLL IS A JOB NOW. multitop --ops-poll is one pass: it gathers
    the hosts whose answer is due and keeps what it found beside the
    config. tools/install_ops_poller.sh installs the launchd agent that runs
    that pass every 7.5 minutes. A pass that exits cannot be left half-dead;
    the next pass opens a fresh session.
  • THE RATE IS THE RULING'S. DECISIONS D15 (owner, 2026-10-20): the
    estate publishes every 15 minutes, so a client reads at least every 7.5
    minutes. The 30 s rate is retired — it gathered 2,880 times a day, ten
    times what the rule needs, and every gather moved the whole tool
    catalogue off the host. The interval is now DERIVED from the rule
    (OPS_POLL = MAX_SAMPLING_INTERVAL_S), so the two cannot drift apart.
  • ONE CARD, ONE READ OF THE ESTATE. The answer a pass found is kept, and
    a host whose answer is younger than the interval is not asked again — by
    the pass, and not by the view either. The two pollers share one due rule,
    so the client's second reader costs the estate one read and not two.
  • THE ANSWER CARRIES THE CLIENT'S OWN RATE. A client that samples
    slower than half the publication interval says so where the verdict is
    shown, on a line of its own (appended to the summary it was cut off by the
    panel's width, which left the meaning off the end of the sentence).
  • THE CACHE KEEPS THE SERVER'S OWN WORDS. The raw answers are kept, and
    the client parses them when it draws: an answer read back from the cache
    is parsed by exactly the code that parses a live answer.

Measured, on the live estate (2026-10-21)

  • One gather, in bytes the host sends: .33 128,455 B (of which the
    tool catalogue is 63,749 B), .90 75,574 B, .158 40,988 B.
  • A day at the rule's rate (192 gathers a host): .33 24.7 MB,
    .90 14.5 MB, .158 7.9 MB — 47.1 MB a day for the estate,
    against 706 MB a day at the retired 30 s rate: a fifteenth.
  • The client: one pass over all four hosts is 0.55 s of wall time, so
    1.8 min a day — against 26 min a day at the 30 s rate.
  • The estate's own time for one gather on .33 is 15 ms of the
    server's own measurement (the audit log's ms column), so 2.9 s a day at
    the rule's rate.

Verified

  • tools/gate.sh --full green (14/14); the Rust suite (1,772 tests
    across the workspace), the pytest suite and the coverage gate as the gate
    of record ran them.
  • Each new behaviour was proven RED first: the interval, the warning on
    the line, the due rule, the pass that gathers what is due, and the view
    that shows the answer it already has each fail with an assertion when the
    behaviour is taken away.

v0.60.0

Choose a tag to compare

@ztomer ztomer released this 21 Oct 00:42

v0.60.0 — the Ops view samples at TWICE the estate's publication rate (2026-10-20)

Changed — the Ops view's health section (DECISIONS D15)

  • The owner's ruling, 2026-10-20: "don't we need to double the data
    polls at twice of routine's polls to keep with the sampling lemma?"

    Yes. The estate publishes its health verdict every 15 minutes, and a
    client that samples at the PUBLICATION rate can ALIAS: its read can
    land just before a publication and then wait a whole cycle, so its
    verdict is two publications behind the truth.
  • THE OPS POLL IS NOT CHANGED. The owner has not ruled on this
    client's interval, and E36 (the poll is not running) is not this
    release's to reopen. The Ops poll already samples every 30 s — far
    inside the rule — and the health section now carries the rule itself
    (ops::MAX_SAMPLING_INTERVAL_S), so a client whose interval was ever
    set too slow says so where the verdict is shown.
  • A client that samples TOO SLOWLY says so, in its answer — not in
    silence. The age on the line cannot say it: a verdict that is two
    publications behind looks exactly like one that is fresh.

Verified

  • tools/gate.sh --full 14/14 green; the Rust suite, the pytest suite and
    the coverage gate as the gate of record ran them.
  • The Ops view still never runs the suite — the same test as before.
  • Each new behaviour was proven RED first: the sampling rule and the
    warning it carries each fail with an assertion when the behaviour is
    taken away.

v0.59.0

Choose a tag to compare

@ztomer ztomer released this 20 Oct 23:14

v0.59.0 — the Ops view's health verdict is the PUBLISHED one, and it is DATED (2026-10-20)

Changed — the Ops view's health section (servers ROADMAP E12, the client half)

  • The verdict is READ, and it is the estate's own. The Ops view already
    read health://latest and never ran the suite; what it did with the
    answer was to count the checks. It now carries the whole verdict: the
    machine's own freshness (oldest_age, in the SERVER's words — 13 min ago), each check's own age, and the suite the verdict belongs to.
  • THE AGE IS ON THE LINE, and that is the point. The clients read the
    PUBLISHED verdict instead of re-running the server's 50-check suite, so
    what the section shows is as old as the estate's last cron run. A reader
    that cannot date a verdict cannot tell a 3-minute-old answer from a
    29-minute-old one — and those are different answers about the same
    machine. So the age is on the line, in the server's own words (one
    formatter dates a verdict everywhere it is shown), and a verdict past the
    estate's interval says so and is NOT green.
  • THE VERDICT IS THE MACHINE'S PERIODIC SUITE. The published table
    carries the DAILY checks' verdicts too, and those are a day old by
    design, so a count that included them would be paired with an age that
    does not describe them. The suite is the server's own answer
    (health.list), and the client asks: it never guesses which checks count.
  • A machine that published NO verdict says so — ⚠ health no verdict published: nothing has run on this machine — and it is never an
    all-clear: "could not ask is never fine", and an empty table is not a
    green tick.

Verified

  • tools/gate.sh --full 14/14 green; the Rust suite, the pytest suite and
    the coverage gate as the gate of record ran them.
  • The Ops view still never runs the suite. A test drives ops::gather
    against a fake mcp_host and asserts that no request named
    health.run_all or health.run was made: the verdict is read, never
    run.
  • Each new behaviour was proven RED first — the age on the line, the
    stale marking, the absent verdict and the narrowing to the suite each
    fail with an assertion when the behaviour is taken away.