Make a stranded agent fleet say so, and give 2-BM's three roles a slot each - #766
Merged
Conversation
`_seeded_fleet` made the fleet a value so a gesture could be made over it. Nothing yet asked the question that value exists for. A `Defined` Agent is registered and inert: the subscribers' lifecycle gate fires on `Versioned` only and refuses anything less WITHOUT SAYING SO. At 2-BM that stranded seventeen agents for three months behind a log that looked clean the whole time, and it is still true right now: sixteen of twenty are `Defined`, while `/actors` reports all twenty `active`, because that is a different axis. The remedy has existed since `promote_seeded_fleet` shipped. What did not exist was anything telling an operator the remedy was needed. So this is not a metric. It is a `warning` on every boot of a deployment whose fleet cannot act, naming the members and naming the remedy. An absence has to be its own loud verdict; the alternative is exactly what happened, a correct system with a clean log and nothing running. FOUR NOT-READY REASONS, KEPT APART, because collapsing them restores the silence at one remove. `Defined` is the only fault, and the only one nobody chose. `Suspended` is a live operator decision, reported and never warned about: nagging someone for a pause they made on purpose is how a line gets scrolled past, which costs the signal this exists to add. `Deprecated` is terminal. `absent` is worth seeing on a partly-seeded deployment rather than counting as a silent zero, the same reasoning `promote_seeded_fleet` already applies to it. The sort ranges over the FLEET, never over the status map it is handed. A member the caller failed to look up has to land in `absent`; ranging over the map would drop it from the denominator instead, and report "2 of 2 ready" for a fleet of three. That is the same silent-incompleteness shape the fleet value was introduced to close, and it is pinned by its own test plus one that counts against the real `SEEDED_FLEET` rather than a fixture. Fourteen tests, each proved falsifiable: ranging over the map instead of the fleet fails three, a `stranded` stuck false fails two, and treating `Suspended` as stranded fails two more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The boot verdict says it once, to a log. This says it on the surface someone is actually looking at. Two places, because they answer to different readers. The subtitle carries "16 of 20 agents registered but inert" and is the one line always on screen no matter where the chart is panned, so it reaches someone who never thought to ask the question. The Actors lane's card carries the detail: the fraction, the remedy, and every inert member BY NAME, because `RunWitness` tells an operator what to do where a count does not. Silent when the fleet is healthy, in the subtitle only. A warning that is always there is one nobody reads. The CARD is not silent, and that asymmetry is deliberate. "Ready to act: 20 of 20" draws on a good day precisely so the row earns trust for the day it says 4; a row that appears only when something is wrong teaches its reader that an absent row means nothing to check, which is the lesson that let a stranded fleet sit unnoticed for three months. `actors` joins ALWAYS_ON for a different reason from the other four. They are always drawn so an empty one reads as "nothing happened here" rather than as a row nobody drew. This one is always drawn because its card carries whether the fleet can act at all, and that answer matters most on exactly the days no agent has emitted anything -- which is when a presence-gated lane would not be there. It draws today with zero events on it and the readiness row populated, which is the whole point. PAUSED AND ABSENT STAY SEPARATE FROM INERT. Suspended is a live operator decision and Deprecated is terminal; reporting either as a fault would claim a problem where a person made a choice. Their rows appear only when non-empty, so a fleet with nothing paused shows no paused row rather than a zero. Producer side, the readiness rides the snapshot on its own 150-tick (5 minute) cadence. Reading it costs one event-store load per shipped agent and the answer only moves when an operator promotes, suspends or deprecates one, so re-reading twenty streams at 2Hz would be the most expensive thing in the tick loop. Not cached for the process's lifetime either, which is the other obvious choice and the wrong one: someone who has just run `promote_seeded_fleet` is looking at this page to see whether it worked, and "restart the API to find out" is not an answer. `build_snapshot`'s `fields` widens from a dict of row lists to `dict[str, Any]`. Every other field answers "which instances are open" and spells none as `[]`; this one answers "can the fleet act", which is a single standing fact and would be a lie as a list, since a healthy fleet would then have to be an empty one. The fixture is STRANDED, matching the real deployment on the day this shipped. A healthy-fleet fixture would render one row and exercise none of the reporting that exists for the case worth catching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The slots were `2-bm-operator-a` and `2-bm-operator-b`. A seat letter tells a future reader nothing about why that principal may do anything, and a Policy grant is exactly the place someone reads it. They are now `2-bm-admin`, `2-bm-group-manager` and `2-bm-staff`. Roles are safe in a public repo where names are not, which is the whole point: the display names still come from the deploy host and still land only in the `actor_profile` PII vault, and the `ActorRegistered` payload still carries no name. What the repo gains is the reason, not the person. THE IDS DID NOT MOVE. They are literals, not derived from the label, so the rename minted nothing: the id is what the record means by a person and what every grant already made hangs off. Re-pinning one would leave a second Actor for the same human, which has happened at this deployment once already, and the two records are still there with only one of them carrying any grants. A test now says so in as many words. The third slot (`...0c0010`) is new and its id was checked against every UUID literal in the repo before it was picked. A ROLE CARRIES NO SCOPE, and the docs now say so rather than implying otherwise. `Policy` holds `(principal, command)` pairs gated by a Conduit and a Surface; there is no beamline dimension in a grant. So "manages the imaging group across several beamlines" and "staffs this one" are indistinguishable to authorization, and those two roles hold identical commands here. That costs nothing at a single-beamline deployment and becomes real at the second. It is a gap in the Policy model, not something a label can close, and it must not be papered over by handing the roles different COMMANDS: recording a difference of scope as a difference of capability is both false and hard to unpick later. The slot list was hand-written in THREE places: the tuple, an `add_argument` call, and a dict literal in `main`. Adding a person meant remembering all three, and forgetting the third would have parsed a name and then never passed it to the ceremony. The parser and the slot-to-name map are now built from the tuple, and the flag rides on the slot, so a slot cannot exist without one and cannot carry a flag that reaches only one of the two readers. Adding the third slot broke three tests that had hardcoded two, which is the protection working. Two of them now count against `len(BEAMLINE_STAFF_SLOTS)` instead of a literal, and the all-names-configured case builds its input from the tuple: a hand-written dict of names would keep passing after a fourth slot arrived and the ceremony would refuse at deploy time against a green suite. Env vars rename with the slots (`BEAMLINE_STAFF_ADMIN_NAME`, `..._GROUP_MANAGER_NAME`, `..._STAFF_NAME`). A deploy host still carrying the old names gets the existing fail-loud refusal, which names the slot and the variable and never a value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes two silent failures loud, and gives 2-BM's three human roles a slot each.
Why
A
DefinedAgent is registered and inert: the subscribers' lifecycle gatefires on
Versionedonly and refuses anything less without saying so. At2-BM that stranded seventeen agents for three months behind a log that looked
clean the whole time, and it is still true today: sixteen of twenty are
Defined, while/actorsreports all twentyactive, because that is adifferent axis.
promote_seeded_fleethas existed since. What did not existwas anything telling an operator the remedy was needed.
Three commits
Say at boot whether the fleet can actually act. A
warningon every bootof a deployment whose fleet cannot act, naming the members and the remedy.
Four not-ready reasons stay apart, because collapsing them restores the
silence at one remove:
Definedis the only fault and the only one nobodychose;
Suspendedis a live operator decision, reported and never warnedabout;
Deprecatedis terminal;absentis a partly-seeded deployment ratherthan a silent zero. The sort ranges over the FLEET, never the status map it is
handed, so a member the caller failed to look up lands in
absentinstead ofvanishing from the denominator.
Put the fleet's readiness on the page an operator already has open. The
subtitle carries
16 of 20 agents registered but inertand is the one linealways on screen; the Actors lane's card carries the fraction, the remedy, and
every inert member by name. Silent in the subtitle when healthy. NOT silent on
the card, deliberately: a row that appears only on a bad day teaches its reader
that an absent row means nothing to check, which is the lesson that hid this
for three months.
actorsjoins ALWAYS_ON so the lane exists on exactly thedays no agent has emitted anything. Producer reads it on a 150-tick cadence,
not per tick (one stream load per agent) and not once per process (an operator
who just ran the promotion is looking at this page to see whether it worked).
Name the staff slots by role, and add 2-BM's third human.
2-bm-operator-a/bbecome2-bm-admin,2-bm-group-manager,2-bm-staff. Aseat letter tells a future reader nothing about why a principal may act, and a
Policy grant is where that gets read. The ids did not move: they are
literals, the id is what every existing grant hangs off, and re-pinning would
leave a second Actor for the same person, which has already happened here once.
Names still come from the deploy host and still land only in the PII vault.
A gap this does not close
A role carries no scope.
Policyholds(principal, command)pairs gated by aConduit and a Surface, with no beamline dimension, so "manages the imaging
group across several beamlines" and "staffs this one" are the same grant.
Harmless at a single-beamline deployment, real at the second. Documented in
governance.mdrather than papered over by giving the roles differentCOMMANDS, which would record a difference of scope as a difference of
capability.
Verification
instead of the fleet fails three; a stuck
strandedfails two; treatingSuspendedas a fault fails two more)"two", which is the protection working
mutant caught
Also removes a hand-copied-list trap: the staff slots were written out in three
places (the tuple, an
add_argumentcall, a dict literal inmain), so addinga person meant remembering all three and forgetting the last would parse a name
and never pass it on. The parser and the slot-to-name map now build from the
tuple.
Deploy note
The env var names change with the slots (
BEAMLINE_STAFF_ADMIN_NAME,..._GROUP_MANAGER_NAME,..._STAFF_NAME). A host still carrying the oldnames gets the existing fail-loud refusal, which names the slot and the
variable and never a value.
🤖 Generated with Claude Code