Repository navigation
v0.4.0 — the monitor-of-monitors release
The monitor-of-monitors release. Observogram assures that the artefacts which monitor a system keep doing their job: the live fetcher stops asserting what it never observed (steps 0-1), the stack samples its own self-metrics through the MCP (step 2), journeys keep stack-health evidence per run (step 3), and the requirement graph shows what would go blind when an artefact dies, a per-node ladder and candidate causes (step 4). The pack generators (boards including a Unified Observability board for every pack, burn-rate rules with the live-measured PromQL, gen-site for a fleet of environments from one pack and an inventory) became one vendorable library and the compiler emits the same PromQL. Also: stage 2 tenancy, first-boot admin and account menu, Diagnose views on the behavioural engine, the Tomograph to Observogram rebrand, legacy pack format support.
Blind-spot blast radius on the graph (step 4)
Step 3 kept the stack's own self-metrics per journey run; step 4 puts the monitor-of-monitors question on the requirement graph itself: what would go blind if an artefact died (structural exposure), whether a present artefact is doing its job or the vantage could not look at all (the per-node ladder), what changed since the previous run, and which candidate causes the evidence can offer. Three things are true of everything in this block, by contract and pinned by the suites: nothing scored changed — branch.integrity / verdict / counts, every node status, rollup.integrityMean and the grade's Drift-free criterion are byte-identical with and without the new annotations, everything new is an additive field beside them, and the switch to the ladder is a proposal (docs/SCORING_PROPOSAL_LADDER_INTEGRITY.md), not applied; the honesty vocabulary is kept — a blast radius says what WOULD go blind, never that something IS blind, unobserved means the vantage could not look (a probe family failed or is not exposed by the tier), never "absent", and present_unhealthy means on the wire but not doing its job; and causes are candidate causes ranked by evidence — not a root-cause verdict, with a change of the vantage itself reported separately and never as a cause. No new MCP tool name, no new dependency, no build step; the two new reasoning modules are zero-import and vendorable.
- Blast radius (reverse index). New zero-import
tools/lib/blast-radius.mjs:CONSUMER_SIDE(per edge type, which side consumes the other's assurance —fromforsli_of,sources,exported_by,produced_by,protects,forecasts,visualises,contains,routes,remediates,validates;toformaterialises, where the SLI consumes the rule's series),PROTECTION_SIDE(thetoside loses when thefromside dies: an SLO loses protection with its alert, an alert loses delivery with its route and remediation with its remediation — transitive, a dead route means an undelivered alert means an unprotected SLO),normalizeGraphShape(nodes as an array, a Map or a keyed object; edges withoutfrom/to/typedropped; edge types outside the table carry no known direction and are ignored),blastRadiusOf(shape, key, { weights })→{ key, kind, label, blinded: { total, weight, byKind, nodes: [{ key, kind, label, hop }] }, unprotected: { slos, alerts }, summary: { slos, alerts, panels, dashboards, routes, remediations, total } }(nullfor an unknown key; breadth-first over the reverse adjacency, cycle-safe;blinded.nodessorted hop →KIND_ORDER→ label and capped atBLINDED_NODES_CAP= 64 whiletotal/byKindstay uncapped; protection losers already blinded are not listed twice;weightsumsDEFAULT_WEIGHTS, a copy of the graph's limb weights, unknown kind 0.5) andblastRadiusIndex(shape)→Map key → summaryin sorted key order. Scaffold placeholders (ascaffoldflag orartefact.source === 'Scaffold') never blind anything, are never traversed and are never listed.traceability-graph.mjsexportsgraphShape(graph)— the artefact-free projection{ nodes: [{ key, identityKey, kind, layer, label, virtual, scaffold }], edges: [{ key, from, to, type, provenance }] }— andcompareBranchescomputes the A-graph and B-graph indexes once per comparison, attachingblastRadius: summary | nullto every node verdict (A's index for aligned / drifted / declared-only / unverifiable nodes, B's for live-only ones). - Per-node ladder (additive, unscored) + the scoring proposal. Every node verdict gains
ladder: { rung, status, detail }— rungsunobserved < absent < exists < alive < healthy, statuspresent_unhealthy|present_stale|unobserved|null— read from what the live fetcher wrote into Pack B:mcp.observed.scrape_targets/recording_rules/alert_rules, themcp.discovered.scrape_jobs_down/recording_rules_unhealthy/alert_rules_unhealthy/slis_unhealthylists,mcp.probesFailed/mcp.probesUnsupportedwithmcp.probeErrors.<family>,mcp.versions.<product>for backends andmcp.refreshedAtas "now" (a timestamp older than 2× the artefact's interval readspresent_stale; no interval or no timestamp means staleness is not judged). Burn-rate alerts link to live alert rules by the compiler's<slo>_burn_<factor>x_<short>_<long>convention, narrowed to the declared windows. Adeclared_onlyartefact the fetcher observed but withheld from Pack B (a scrape job whose every target is down) readspresent_unhealthy— on the wire beats absent; a kind whose probe family failed or is not exposed readsunobserved, never absent; anunverifiablekind reads rungunobservedwith statusnull; without anymcp.annotation every present node isexists. Per branch:ladderVerdict(broken>degraded>unobserved>healthy;undeclaredmirrors the scored fields) andladderIntegrity/ladderIntegrityPcton the same weights — unhealthy / stale credit 0.25, absent 0, unobserved nodes leave the denominator;rollup.ladder = { healthy, degraded, broken, unobserved, integrityMean, integrityPct }over the declared branches.docs/TRACEABILITY_GRAPH_COMPARISON_SPEC.md§5b carries the table and evidence rules;docs/SCORING_PROPOSAL_LADDER_INTEGRITY.mdproposes that Drift-free readrollup.ladder.integrityMeanundergradeSchema3, with the four worked rows on the test fixture — a proposal through the PHASE_1 acceptance path, not applied. - Per-run persistence: chains, versions, transition, live pack. New zero-import
tools/lib/chain-history.mjs:branchRecordsFromGraph(traceabilityGraph)keeps per branch{ rootKey, title, rootKind, verdict, ladderVerdict, integrityPct, ladderIntegrityPct, confidence, missingRoles, degraded, truncated? }with the degraded nodes (declared_only/drifted/live_only, or ladder statuspresent_unhealthy/present_stale/unobserved; aligned-and-healthy andunverifiablenodes are not recorded) worst first — absent, unhealthy, stale, drifted, unobserved, live-only, then widest blast radius — as{ key, kind, label, status, ladder, blastRadius: summary, deltaFields }(caps 64 branches × 16 nodes × 6 delta fields, a cut list markedtruncated);chainSummary(record)→{ declaredTotal, intact, partial, broken, undeclared, ladder: { healthy, degraded, broken, unobserved }, integrityPct, ladderIntegrityPct, degradedNodes, topExposure: { label, kind, slos, alerts } | null }(means arenullwith no declared chain — an empty set is not 100 % healthy;nullwhen the record carries nobranches);diffRunBranches(previous, current)→{ since, changed: [{ rootKey, title, from: { verdict, ladderVerdict }, to, direction, nodes: { newlyDegraded, recovered } }], appeared, disappeared, any }withdirectionfrom the rank tablesintact < partial < brokenandhealthy < degraded < unobserved < broken(worsewhen nothing improved and something got worse,betterthe mirror,changedotherwise or when either side isundeclared/ unknown;nullwhen either record carries no chains). The run record gainsbranches,chains,versions(mcp.versions.<product>→ value, bare product keys only,nullwhen none — an absence, never "unchanged"),transition(against the newest record read before this run is written;nullon the first run or after a record without chains, e.g. a vantage loss) andlivePack: { kept, path, bytes?, reason }. The definition keykeepLivePack: transitions | always | never(validated byloadJourneyDef; defaulttransitions) decides whether Pack B's canonical JSON is written toruns/<journey>/live/<record stem>.jsonbeside the record:transitionskeeps the first run, a run whose chains moved, a gate failure, the run after a vantage loss and the run after a record that cannot be compared; a file-sourced Pack B is never snapshotted (the file is the snapshot), whatever the policy. The purelivePackDecisionis exported; a snapshot write failure lands on the record ashistoryError, never thrown. Retention is unchanged (JOURNEY_RUN_FILE_REuntouched,readJourneyRunskeeps ignoringlive/) and now also removes orphaned snapshots — after the record prune,pruneLiveSnapshots(recordFiles, liveFiles)(pure) names the run-shapedlive/files whose record is gone, never a survivor's, never a non-run file;readLivePack(name, record)reads a snapshot back (nullwhen none was kept, the path is not thelive/<stem>.jsonshape, or the file is gone — a pruned snapshot reads as absent, never as an error).renderJourneyMarkdownadds aRequirement chainstable (chain | verdict | ladder | integrity | ladder integrity | worst node, cap 24, only when the record carries chains — no table means none were declared or recorded, never that every chain is intact; the worst-node cell carries the ladder detail andblinds N SLOs) and aTransitions since previous runsection (title — intact/healthy → broken/broken (worse); newly degraded: …; recovered: …,appeared/disappeared, then thelive pack: kept (…) — reason/not kept — reasonline). - Candidate-cause ranker.
chain-history.mjsgainsrankCauses({ previous, current, deploys })→{ transitions, causes: [{ rank, kind, score, evidence, chains, nodes }], vantage: { changed, from, to, detail } | null, note }withnote=candidate causes ranked by evidence — not a root-cause verdict. Only chains whose transition readsworse(plus chains that appeared already partial / broken / degraded) are considered, and only their nodes that moved (new on the current side, or the same identity with a different status / ladder status; every degraded node when the record cannot say); nodes whose ladder status isunobservedare dropped last, so a chain whose only movement is the vantage looking away yields no cause. Four kinds, scored by fixedCAUSE_SCORESso a rank is explainable by reading the table:observogram-deploy— a deploy inside the window whose itemartifactnames a moved node 0.9 (an artefact id on the node —aId/bId— matches exactly, otherwise the artefact must be contained in the node's label or key case-insensitively with adash:prefix dropped; only that direction, so a rule deploy cannot blame every SLI whose short name it embeds; stubs under three characters and the wildcardallname nothing; the persisted degraded-node record carries no ids today, so on a real run the match is by label / key), else a deploy that wrote the journey's pack (bypack.id/pack.nameagainst the record'spackA) 0.6;config-drift— a moved node that drifted on a decision-bearing field 0.8, on cosmetic or unrecorded fields 0.4;backend-version— a product both records reported with a different version, 0.6 when a moved node is a backend / metric / recording rule / burn-rate alert, 0.3 otherwise;stack-self-metric— a current-record stack row that answered data and reads as a signal (nonzerohint,lowerabove 0, or ahigherratio below 1) in the family feeding a moved node's kind perFAMILY_FOR_KIND(scrape →scrape_job; ruler →recording_rule/burn_rate/sli/slo; notify →alert_route; tsdb →backend/storage_metrics; collector →pipeline_*/otel; dashboards →panel/dashboard; synthetic →synthetic) 0.5. One cause per distinct evidence, aggregating every chain and node it explains; ordering score desc → kind order → evidence; a verify-type record or a dry run is skipped; malformed input never throws.deploysInWindow(deploys, since, until)windows Observogram's own audit lines to(previous.startedAt, current.startedAt]onatand merges the latest verify onto its deploy so the evidence can name the outcome (deploy dep_x by carlos at … (upsert) touched <artifact>; verify: pending);journey.mjsreadsdeploys.jsonlbesideruns/under the same workspace root without importing server code (missing file → nothing, a torn line skipped).vantagereports what changed about the vantage itself — the vantage word, a probe family that newly failed (with its error), is no longer exposed, no longer probed, answers again or is now exposed, and the exposed tool count — beside the causes, never among them. The record carriescauses(nullon the first run: nothing to explain yet);topCause(record)reads the rank-1 cause. Markdown:### Candidate causes — ranked by evidence, not a root-cause verdict(_no previous run_/_no chain got worse since …_,N. [kind] score — evidence (chains: titles), thenvantage changed: …/vantage: unchanged). - Surfaces.
packc journey listappendschainStatusLine(record)after the stack segment —chains 8/10 intact · ladder 7 healthy · 2 degraded(zero ladder buckets other than healthy omitted;chains nonefor a record without declared chains) — and, only when a chain got worse or a cause was ranked (transitionGotWorse),causeLine(record)(top cause: [kind] evidence/no candidate causes); a vantage-lost line carries neither.GET /api/journeyslastRungainschains(chainSummary(lastRun),nullwithout chains),transition: { any, changed, worse } | null(counts, not the full diff),topCause(the rank-1 cause object ornull) andvantageChanged(true/false/nullwithout a causes block);GET /api/journeys/:name/runshands the record through unchanged. The Journeys view prints one plain-text chains line per card —requirement chains: N/M intact · ladder: h healthy · d degraded · b broken · u unobserved · top exposure: <label> (<kind>) blinds N SLOswith a mutedchanged since previous run (W worse)marker — and one cause line —candidate cause: [kind] evidence — not a verdictwith a mutedvantage changedmarker; counts and markers, no colours, no new CSS. In the Diagnose chain cards (compare-view.mjs)evidenceFornamespresent but unhealthy,present but staleandunobserved — <detail>nodes (the ladder reading replacesmissing live/alignedfor that node) and appends· blinds N SLO(s)todeclared_only/driftednodes withblastRadius.slos > 0; the card meta showsladder: <ladderVerdict>and the section rollup appends· ladder NN%. - Suites. New
tools/test-blast-radius.mjs(npm run test:blast-radius, 58 assertions — the vendoring guard, the tables, hop distances along the full chain, the protection relation, scaffold exclusion, cycle safety, determinism, the listing cap, the accepted input shapes, no input mutation) andtools/test-chain-history.mjs(npm run test:chain-history, 122 assertions — the guard, the tables and caps, which nodes are recorded and in what order, the truncated markers, the summary and its top exposure, transition directions, appeared / disappeared and node lists, the deploy window and verify merge, every cause kind and score, the one-way artefact match, the moved-node rule, the vantage block including a vantage-lost previous record and file-sourced sides, determinism, malformed input); both in the aggregate (34 suites).tools/test-traceability-graph.mjsadds thegraphShapeprojection, theblastRadiuspins per node kind, the ladder block (healthy / unhealthy / stale rules, an unhealthy burn rule linked by name, a down target, unsupported and failed probe families, absent for real, withheld from Pack B) and the byte-identity pin onintegrity/verdict/counts/ nodestatus/rollup.integrityMeanwith and without the annotations.tools/test-journey.mjsdrives a fake MCP through the snapshot policy (every branch oflivePackDecision, the files underlive/, pruning,readLivePack), the record shape and its round-trip through the history file, the markdown sections, the CLI lines and the ranker against a seededdeploys.jsonl(a deploy inside the window, one outside, a torn line, the vantage block).server/test-smoke.mjsseeds a record withbranchesandcausesand pins thelastRunshape. - Review corrections (step 4). Two fix passes after the step-4 reviews; nothing scored changed on either side (
integrity,verdict,counts, nodestatus,rollup.integrityMean, the grade — byte-identical, still pinned). Graph side. The fetcher stamps its own clock:mcp.fetchStartedAt(the instant the fetch began) andmcp.observedAt.<family>(the instant each probe family answered, data or empty;dashboardsre-stamped after detail enrichment), and the ladder's "now" is the family'sobservedAt, elsefetchStartedAt, else the caller'smcp.refreshedAt(older packs — named in the detail only on that fallback), so arefreshedAttrailing a fresh observation by a minute no longer reads stale. A scrape job ispresent_unhealthyonly when EVERY target is down (the fetcher'sscrape_jobs_downrule); some targets down readsalivewithn/m targets down: <instances>, and a down target'slastErroris not the job's. Fractional burn factors (_burn_14_4x_↔ 14.4) parse in the graph and the fetcher; alert observation entries carryinterval(the group evaluation interval — the alert adapter keeps it now, and burn-rate freshness uses the longest linked one) andlabels: { slo, burn_rate, window_short, window_long }(present keys only, omitted when none), and the ladder links a live rule to its declared window by labels first, name second; thealert_rules_unhealthylist is narrowed to the declared windows like the rule set and the detail names the matched / failing rule(s). An unknown interval readshealthy/aliveonly within a 1 h ceiling ("interval unknown; 1 h ceiling applied"), older readspresent_stale; a timestamp more than 1 s past "now" readsalivewith "timestamp s in the future (clock skew); freshness not judged", neverhealthy; a reported health ofunknownstayspresent_unhealthy;ladderVerdictignoreslive_onlynodes as the scored verdict does; "no fetch timestamp on the wire (mcp.refreshedAt missing); staleness not judged" when the observation has a timestamp and the pack has no clock.compare-view'sevidenceFororders the evidence line before the 5-cap — declared_only load-bearing, present_unhealthy, present_stale, drifted, declared_only (other), unverifiable, unobserved, live_only — and adeclared_onlynode whose live twin sits under another chain reads "on the wire and in Pack B under another chain: …" (or "in Pack B under another chain; not reachable from this branch"); "withheld from Pack B" only when no twin exists. The goldenalert_rules.jsongains the twointervallines; the largest example pair's payload grows +88 % (spec §5c). Journey side. Deploy attribution is exact.runJourneyresolves every windowed deploy item's artifact selector against Pack A (resolveDeployArtifact, exported:declared:<i>→ the i-th declared recording rule's name,slo:<id>→ the SLO id, its SLI base and the SLI the pack binds it to,dash:<id>→ the id, a bare name → itself,all→ nothing; the item carriesresolvedbesideartifact) and the ranker matches a resolved name exactly — against the node'saId/bId(now persisted on every degraded node), its label, or an identity handle of itskind::{json}key (id,name,record,slo,job,uid) — never by substring, and only when the item's group can write the node's kind (DEPLOY_GROUP_KINDS: dashboards / restore / delete → dashboard, panel; rules → recording_rule, burn_rate, sli, slo; alerts → burn_rate, alert_route; alertmanager → alert_route; pipelines → pipeline_*, otel; an unknown group constrains nothing); the< 3stub rule is gone; the evidence readstouched declared:0 → payment:api_availability:ratio_5mand, for a wildcard-only deploy of the journey's pack,wrote pack X (all rules) — no item names a moved artefact(0.6 — needs the audit line'spack.id, the registry id = the pack file's stem, orpack.nameto equal the record'spackA.name). Without a caller's resolution the ranker knows the pack-free part (deployArtifactNames). Versions. A flip to or from the literallive(the fetcher's word for "answered without a number") is a vantage matter — reported invantage.detailasmcp.versions.<product> changed to/from live, never a cause — andbackend-versionscores 0.6 (versionFeedsMovedKind) only when the changed product feeds a moved node's family (PRODUCT_FAMILIES: prometheus / victoriametrics / thanos / mimir → scrape, ruler, tsdb; grafana → dashboards; alertmanager → notify; otel, otelcol → collector; loki, promtail → logs; jaeger, tempo → traces; unknown → none), else 0.3 — a Grafana-only change is no longer 0.6 on a Prometheus rule. Baseline. The chain diff, the versions and the deploy window are compared against the newest record that CARRIES chains (BASELINE_SCAN_LIMIT25 back), the vantage against the immediately previous record (rankCausesgainsbaseline);transitionis never null on a new record:{ reason, since: null, changed: [], appeared: [], disappeared: [], any: false, skipped }withreasonfirst run|previous run lost its vantage|previous runs carry no chain recordwhen nothing could be compared, else the diff withreason: nullandskipped: [{ startedAt, outcome }]naming the records passed over (cap 8); the report says "previous run lost its vantage — comparing against " in both sections and never "no previous run" when one exists. No broadening. A node whose status and ladder status are identical on both sides is never blamed; when nothing recorded moved, the changed transition entry carriesnote—declared side: missingRoles none → action,degraded list truncated (cap 16) — a node that moved may be unrecorded— and the chain yields no cause. Escaping. Every value the transitions, causes and breaches sections interpolate goes throughmdCell(line breaks collapsed,|escaped, a line-leading#/-/*/+/>/1.neutralised) and the ranker's evidence collapses whitespace; a label carrying\n\n### Gate breachescannot forge a section. Also:readDeployLog(exported) reads only the trailing 8 MB ofdeploys.jsonl(DEPLOY_LOG_TAIL_BYTES, the cut first line dropped);readJourneyRunsreads only run-shaped files (a straynotes.jsonis never the previous run); an unparseable baseline start gives an EMPTY deploy window, never all of history; a snapshot write failure readslivePack: { kept: false, reason: 'snapshot write failed: …' }besidehistoryError;packc journey listappendsvantage changed: <detail>(vantageLine) whenever the vantage block says so, with or without a top cause;pruneLiveSnapshotsnames only run-shapedlive/files OLDER than the oldest surviving record (retention victims), never a newer stem whose record a concurrent writer has not landed yet;stackSignalRowsreads direction + value only, never the storednonzerohint; thecausesblock no longer carries a second copy of the transition ({ causes, vantage, note });chainStatusLinedistinguisheschains none (pre-step-4 record)/chains none (vantage lost)/chains 0 declared;keepLivePack: neverwrites none from now on (earlier snapshots age out with their records);rankCausesaccepts onlytype: 'deploy'(asdeploysInWindowdid).cause.chainscarries the branch titles androotKeysthe identity keys;chainSummary.degradedNodes/topExposurecount and rank the declared chains only, andundeclaredNodescounts the live-only nodes of undeclared chains — the Journeys view appendsN live-only in undeclared chainsonly when > 0. Pinned: the server's real selector strings (declared:0,slo:checkout,all, a rollback ref underrestore),dash:paymentblaming neitherscrape_job paymentnor the rule embedding it, a three-letter dashboard id, wrong groups,branchRecordsFromGraphoutput satisfying the id match, theliveflips, a Grafana-only change on a Prometheus rule, the stale hint, missing-role / truncation notes, baseline vs previous, the wording after a vantage-lost run (and with no earlier chains, and after a pre-step-4 record),notes.json, the 8 MB tail with a cut first line, the empty window, markdown forgery through labels / actor / evidence / breaches / vantage / live-pack reason, the CLI vantage segment, the smoke seed made consistent and thevantageChanged: nullAPI case; the fake-MCP e2e deploy now carriesdeclared:0end to end.
Stack-health evidence per journey run (step 3)
Step 2 sampled the observability stack's own self-metrics on every live fetch as point-in-time signals; step 3 turns those samples into evidence over time by keeping them per journey run, so the run history becomes the time series the journeys surface reads. Nothing in this block creates a Verified stamp, touches the grade, or turns a sample into an SLO verdict.
-
Run-history retention (prerequisite). A journey on a cron cadence used to grow
runs/<journey>/without bound.writeRunRecordnow prunes the directory to the newestOBSERVOGRAM_JOURNEY_RUN_RETENTIONfiles after every write (default1000;0= unlimited; a value that is not a non-negative integer falls back to the default; legacyTOMOGRAPH_*spelling honoured viabrandEnv). The policy is the pure, exportedpruneRunFiles(files, keep)(names to delete, oldest first — filenames are ISO start times so lexical order is chronological; non-.jsonentries are never candidates) plusparseRunRetention/journeyRunRetention()(read at write time). A deletion failure lands on the record ashistoryError, never thrown;readJourneyRunsis unchanged. Scheduling stays external by decision (VALUE_BACKLOG item 11). -
stackEvidenceon the run record.liveEvidenceFacts(canonicalB)parsesmcp.observed.stack_metrics,mcp.observed.alertmanagerandmcp.observed.grafana.*intostackEvidence = { status, reason, rows: [{ id, family, product, value, unit, direction, outcome, hint, at, referenceSli, reason? }], alertmanager: { version, clusterStatus, silencesActive, error } | null, grafana: { datasources, unhealthyDatasources, contactPoints, error } | null }—nullwhen Pack B carries nomcp.stack.status(a file-sourced B is honest absence), rows enriched with the contracts table'sreferenceSliand display-onlyhint(a row the table no longer declares keepsreferenceSli: null),exprdropped, cap 64, malformed JSON degrading to no rows /nullsurfaces rather than a fabricated panel. The step-2stackcounts block is unchanged for compatibility. Suite:tools/test-journey.mjs(retention policy + effect, env parsing, evidence from a fixture-built annotated Pack B, null for file-sourced B, retired row ids, malformed JSON, advertised-but-failing surfaces, round-trip through history). -
Gate key
stack— thresholds on the samples.gate.stack: { requireSampled?, rows?: { <rowId>: { min?, max? } } }turns the samples into an early warning for a business owner, never an SLO verdict:requireSampledbreachesstackwhenstackEvidenceisnull(Pack B is not a live draft), the panel wasnot-attempted(with the reason), or no row answereddata;rows.<id>breachesstack.<id>withno sample for <id> (<outcome>[: reason]) — threshold cannot be checkedwhen the row is absent or did not answer data (a file-sourced B with a declared threshold breaches honestly instead of passing by absence), and with<id> = <value> <unit> outside [min … max] — point-in-time sample, not an SLO verdictwhen the number falls outside the band (equality passes; aninforow may carry a threshold).loadJourneyDefvalidates the block through the exportedvalidateGateStack(stack, name)— unknown row ids (case-sensitive againstSTACK_SELF_METRIC_PROBES) throw with the known ids listed,min/maxmust be finite numbers, an entry with neither (ormin > max, or a non-booleanrequireSampled) is refused. PureformatStackValue(value, unit)renders83.3%/0.004/s/0.0/h/7.4s/1.renderJourneyMarkdownadds aStack self-metrics — point-in-time samplestable (id | family | value unit | outcome | hint | reference SLI, cap 24, only when rows exist) and lists stack breaches with the others;packc journey listappendsstackStatusLine(record)(stack sampled N/stack not attempted/stack none). The capture default gate and every existing key are unchanged; nothing touches the grade. Pinned intools/test-journey.mjs(every branch of the gate, the validation errors, the formatter, the table and cap, the CLI line). -
History helpers, listing summary and the journeys chips. New browser-safe
tools/lib/stack-evidence.mjs(imports only the contracts table; vendorable, listed indocs/VENDORING.md):stackSeries(runs, rowId)(oldest → newest per row, gaps skipped never interpolated, non-data outcomes kept withvalue: null),latestByFamily(record)(the data row per family, ties by table order, retired rows last),stackSummary(record)({ status, reason, sampled, families }ornullwithout evidence),nonzeroRuns(series),stackPostureBudget(series, { objective, cadenceMs, windowMs, isBad? })(the cadence heuristic — a sampled posture is onlymeasurablewhen the window allows ≥ 10 bad samples; every note reads "signal, not verdict") and the sharedformatStackValue/stackOutcomeLabel(journey.mjs now re-exports the formatter from here — one vocabulary for CLI, report and studio).GET /api/journeysaddslastRun.stack = stackSummary(lastRun)(nullfor a file-sourced B); the Journeys view loads the module from/lib/stack-evidence.mjsat call time and renders astack self-metrics — point-in-time samplesline per card — one chip per family, thenonzerohint as a muted marker, id + reference SLI in the title,nonzero in N of last M runsfor lower-is-comfortable rows over the fetched history, one muted chip with the reason when not attempted; no chip carries an ok/error colour. New suitetools/test-stack-evidence.mjs(npm run test:stack-evidence, in the aggregate);server/test-smoke.mjsseeds a record withstackEvidenceand checks the listing shape. -
Docs reconciled with the branch. README's journeys section carries the gate example with the
stack:block, the retention env var and the studio chips, and its API table / repository map now list the journey routes and modules (tools/lib/journey.mjs,tools/lib/stack-evidence.mjs,studio/journeys-view.mjs);docs/MCP_INTEGRATION.md's journey section documents the record fields, the gate keys with their breach table and honesty rules, retention, and the history helpers with the cadence heuristic;docs/VENDORING.mdlistsstack-evidence.mjs+ the contracts table as a vendorable pair;docs/VALUE_BACKLOG.mditem 11 gains a status note (drift over time now includes the stack samples; scheduling stays external). -
Review corrections (step 3). Reproduced against a seeded workspace and the fake MCP:
latestByFamilyranked data rows by contracts-table order alone, so whenever a family's first table row ishigher/info(scrape_success_ratio,tsdb_active_series) and answered data, the family's lower-is-better rows — and with them thenonzeromarker and thenonzero in N of last M runshistory the chips exist for — never reached the surface: the card readscrape 95.0%while the run breached onscrape_targets_down. Data rows now rank bynonzerohint first, then declared-in-table before retired, thenlowerbeforehigher/info, then table order (tools/test-stack-evidence.mjspins each rung; the old "ties by table order" assertion pinned the hiding). Also from the review: arows.<id>threshold on a row the fetcher never wrote names why —(not-attempted: <tier reason>)on a not-attempted panel,(not attempted by the sampler — call budget exhausted or row not observed)on a sampled one — instead of a bare "absent from this run" that read like a fetch hole;evaluateStackGateno longer trusts an unvalidated gate object (a bound that is not a finite number, an entry with neither bound,min > maxor a non-mapping entry breachesstack.<id>asthreshold invalid (…) — cannot be checked, never passes by silence); a breach whose display rounding equals its bound appends the raw number (0.000/s (raw 0.0004)); an outcome the contracts do not declare is kept verbatim (missing →unknown) rather than relabelledfailed;POST /api/journeys/capturerunsvalidateGateStackbefore saving (400) andGET /api/journeys/packc journey list/ the studio card carryloadErrorfor a definition that cannot load;pruneRunFilesonly treats the run-record filename shape (JOURNEY_RUN_FILE_RE) as a run, so a hand-droppednotes.jsonneither displaces a record nor is deleted, and the deletion-failure branch (historyError, never thrown) is now under test. README anddocs/MCP_INTEGRATION.mdreworded accordingly (requireSampled's third branch, "the only gate reader", the raw-number fallback of the chips without the helper module).
Stack self-metrics (step 2)
Observogram monitors the artefacts that monitor the system; step 2 of the roadmap starts acquiring the observability stack's own health signals through the MCP so later steps can build stack-health evidence and early warning. Three things are true of everything in this block, by contract and pinned by the suites: every sampled number is a point-in-time signal, not a verdict — no Verified stamp, no Scaffold marker, no SLO verdict and no grade criterion or grade input changed anywhere; a restricted MCP tier reads not-attempted with the reason, never "absent"; and the alias table reads only names a real product exposed — it was documentation-grounded when written (every metric name following the product's upstream docs, named on the row) and now carries three live evidence sources: the pinned Docker stack (npm run test:stack:live, every alias stamped with the image it was verified on) and the public (2026-09-07) and authenticated (2026-09-08) Krystaline tiers (npm run record-fixtures); a product a given stack does not run reads an honest not-in-inventory.
- Alias table (contracts). New data-only
tools/lib/contracts/stack-self-metrics.mjs:STACK_SELF_METRIC_PROBES— 24 rows over nine families (scrape, ruler, notify, tsdb, collector, dashboards, synthetic, logs, traces), each with its signal, unit, display-only direction, the reference-pack SLI whose vocabulary it follows, ordered product aliases ({ product, expr, requires }) and the upstream docs the names come from — plus lookup resolvers (probeRows,rowsForFamily,eligibleAliases,productPreferenceOrder,displayHint,bestOutcome) and the allowed sets (STACK_FAMILIES,STACK_UNITS,STACK_DIRECTIONS,STACK_OUTCOMES). One known discrepancy is documented on the row rather than fixed: the reference pack'sscrape_duration_p99namesscrape_duration_seconds_bucket, which Prometheus never exposes (the metric is a per-target gauge), soscrape_duration_maxsamplesmax(scrape_duration_seconds). - Registry rows + shapes.
mcp-capabilities.mjsgainsstack_self_metrics(carrier for the table, ridesmetrics_query),alertmanager_status,alertmanager_silences,grafana_datasources,grafana_datasource_health(enrich) andgrafana_contact_points;build_info_versionsnow declares itsinstant-vectorshape.response-shapes.mjsaddsinstant-vector({ result }and{ data: { result } }),silences,datasources,contact-points, and a new object shape kind (object: true,objectAt,anyOfKeys) forstatus-objectandhealth-object—validateResponseShapelocates the first candidate object carrying every key group, tolerant of extras and of a{ data: {…} }envelope. Five tool names join the pinned wire surface. - Sampler (fetcher).
tools/fetch-live-pack.mjsexportssampleStackSelfMetrics({ callTool, quiet, metricsQueryTool, inventory, seenProducts, discoveredToolNames, hasToolsList, refreshedAt, maxCalls })→{ status, reason, rows, callsMade }, run byfetchMcpafter the version probes. Policy: attempt only whenmetrics_queryis advertised (a server with notools/listis attempted), otherwisenot-attemptedwith the reasonmetrics_query not exposed by this MCP (restricted tier)and zero calls; with ametric_namesinventory an alias is eligible only when everyrequiresname is present (rows with none arenot-in-inventory, no call); eligible aliases ordered generic-first then products already seen; at most 2 calls per row, 48 per panel (call budget exhaustedbeyond it); every call throughquiet()under the familystack_self_metrics; instant vectors parsed from{ result }or{ data: { result } }bysampleFromInstantVector(exported — the recorder reads answers the same way),NaN/±Inf/missing samples areempty. Outcomes are exactlydata | empty | failed | not-in-inventory | not-attempted— never "ok". - Alertmanager / Grafana status rows.
observeAlertmanager(alertmanager_status→ version / uptime / clusterStatus;alertmanager_silences→ active + total) andobserveGrafana(grafana_datasources→[{uid, name, type}],grafana_datasource_healthper uid ≤ 10 →ok | error | unknown+ a message trimmed to 200 chars,grafana_contact_points→ count + ≤ 32 names), each tool guarded bytools/listwhen present, tolerant parsing through the response shapes.fetchMcpreturnsstackSamples,alertmanagerObserved,grafanaObserved; answered tools leaveunmatchedTools. - Annotations.
buildCanonicalPack(new paramsstackSamples,alertmanagerObserved,grafanaObserved, allnull-safe — a caller that predates step 2 writes nothing) emitsmcp.stack.status,mcp.stack.reason(only when not attempted),mcp.stack.sampled/empty/failed/notInInventory/notAttempted,mcp.stack.families(<family>:<best outcome>),mcp.observed.stack_metrics(attempted + not-in-inventory rows, cap 64),mcp.observed.alertmanager,mcp.observed.grafana.datasources,mcp.observed.grafana.contact_points, and lists the answering tools inmcp.toolsCalled. Nomcp.verified.*ormcp.scaffold.*key is touched by any of it (pinned by the suite). - Surfaces.
POST /api/draft-from-mcpanswerssummary.stack(status,reason, the five counts as numbers,families: { family: best outcome },rows: [{ id, family, product, value, unit, direction, outcome, hint, reason? }]—hintis the contracts' display-onlydisplayHint, computed on read and never stored),summary.alertmanagerandsummary.grafana(eachnullwhen not exposed), and warnsStack self-metrics not attempted — metrics_query not exposed by this MCP tier.on a restricted tier;GET /api/live-statusaddsstackStatusandstackSampled. The studio draft summary gains a "stack self-metrics — point-in-time sample, signal not verdict" block: one row per family with the best row's formatted value (or— empty/— probe failed/— not in inventory), a single— not attemptedrow on a restricted tier, then the Alertmanager version + active silences, the Grafana datasource count with the unhealthy names, and the contact-point count.liveEvidenceFactsaddsstack { status, reason, sampled, empty, failed, notAttempted }, carried on the journey run record and printed as oneStack self-metricsline in the markdown report — no gate key reads it. - Fixtures: synthetic now, recorded later.
tools/fixtures/mcp/synthetic/holds hand-written samples for each new shape (an otel-mcp-server and a Prometheus-API instant vector, Alertmanager status and silences, Grafana datasources, datasource health, contact points), each carrying a_syntheticmarker the shape check tolerates andtools/test-contract-shapes.mjsasserts; they pin shape only. The fixtures README now carries a provenance table (recorded 2026-06-12 from the Krystaline otel-mcp-server vs synthetic 2026-09-07; the recorded metric-name list is trimmed to 25 names and proves none of the alias table'srequires), the recorder procedure, and what to re-run afterwards. A recorded<tool>.jsonat the fixtures top level takes precedence over its synthetic copy in the shapes suite (same assertions, plus "no_syntheticmarker"), and every file inrecorded-stack/must be a valid instant vector for a declared row. - Recorder.
tools/record-mcp-fixtures.mjs(npm run record-fixtures,MCP_URL+ optionalMCP_AUTH) reuses the fetcher's JSON-RPC client and the contract registry for every tool name (the guard suite scans it for literals), is read-only against the MCP, and never prints or stores the token (every string that leaves the process is redacted). Default mode reports: thetools/listsurface with its drift againstallKnownToolNames(), the metric inventory, and for every alias of every row whether itsrequiresare all present plus the value read withsampleFromInstantVector(data/empty/FAILED/not-in-inventory (missing …)/not-attempted (restricted tier)), then the status tools and probe families with their shape verdicts.-- --writerecords: the full inventory to the git-ignored.tmp-mcp-metric-names.json, the trimmed inventory (every required name and the*_build_infometrics kept, plus the first 25 others, in the server's order, replaced where the adapter found the list), scrape targets / the winning rule tool(s) / dashboard search with every list capped at 6 and structure verbatim, onemetrics_queryinstant vector per family underrecorded-stack/<row id>.jsonwith_recordedprovenance, and the status tools when advertised;--out <dir>redirects everything. New scriptslint:recorder,test:recorder. - First live recording (2026-09-07, public Krystaline tier,
https://www.krystaline.io/mcp/public, no credentials). The recorder's first real run replaced the guesswork for part of the table: that tier advertises 43 tools includingmetrics_query,metrics_label_values(2,682 names),alertmanager_statusandalertmanager_silences, sometrics_label_values.jsonis now a 40-name recording that evidencesup,scrape_duration_seconds,vm_*,alertmanager_*,otelcol_exporter_queue_*,grafana_http_request_duration_seconds_countandpromtail_dropped_entries_total;alertmanager_status.jsonis recorded (itsconfigfield, the full Alertmanager configuration, omitted by hand); andrecorded-stack/holds six realmetrics_queryinstant vectors (otel-mcp-server answers a flat{ status, resultType, result }envelope). Ten aliases readdata, oneempty, twenty-sixnot-in-inventory— the stack runs VictoriaMetrics, not Prometheus, and does not scrape vmalert (its ruler's own health is invisible there: exactly the blind spot the sampler exists to name). Every Grafana-backed tool exceptgrafana_healthanswersHTTP 401on that tier (no Grafana credentials), so the Grafana status fixtures stay synthetic until the authenticated tier is recorded; the silences recording was empty and pins nothing, so its synthetic copy stays too. The 2026-06-12metrics_targets.json/vmalert_rules.json/grafana_dashboards_search.jsonrecordings were kept on purpose (today's targets are allup; thedownAlertmanager target is the liveness case the adapters are pinned against). Contract fix found by the run:grafana_dashboard_getwas listed as a search candidate of thedashboardsprobe, so a failed search fell through to a uid-less call whose-32602 Invalid argumentserror masked the real search error; it is now only thedashboard_detailenrichment tool, and the family reports the 401 it actually got. Suites replay a recorded status fixture in preference to its synthetic copy and read expected values from the fixture they replayed, and the recorder suite's fake inventory is frozen to the 2026-06-12 names so re-recording never moves its expectations. - Authenticated-tier recording (2026-09-08,
https://www.krystaline.io/mcp,MCP_AUTHbearer). The recorder's first run against the authenticated Krystaline tier re-recordedvmalert_rules.json(the 2026-06-12 recording predated most of today's rule groups; adapted goldens re-pinned; its annotations carry rule descriptions and runbook URLs into the private krystalinex-core repo — references, not credentials, and the identical payload the credential-less public tier serves),alertmanager_status.json(_recorded.scrubbednames the hand-removedconfig— receivers, SMTP settings, internal hostnames — same trim as the 2026-09-07 file it replaces) and the sixrecorded-stack/instant vectors (same rows and outcomes as the public tier: 14 aliasesdata· 0failed; the trimmedmetrics_label_values.jsonrecording came back byte-identical to the committed public-tier file, and both tiers report a 2,682-name inventory — the full lists were not compared). The intended goal — replacing the synthetic Grafana fixtures — was not reachable: the tier advertises no Grafana-backed tools with any issued key (its otel-mcp-server deployment carries no Grafana integration; the public tier advertises them but answers HTTP 401), sosynthetic/grafana_*.jsonand the 2026-06-12grafana_dashboards_search.jsonstay,metrics_targets.jsonkeeps its 2026-06-12down-target recording (the 2026-09-08 payload is all-up), and the empty silences recording again pins nothing. The tier also advertises logs and k8s tools the registry has no rows for (noted in the fixtures README; no registry rows added). - Local-stack recording of the Grafana-backed tools (2026-09-08,
docker/stack.compose.yamlfronted by otel-mcp-server 1.8.0). The fourth evidence source, and today the only placegrafana_datasources,grafana_datasource_healthandgrafana_contact_pointsanswer: the pinned stack (Grafana 12.4.4, admin-seeded with aVictoriaMetrics (stack)datasource, aLoki (absent)datasource pointing at a service the stack does not run, thewebhook-oncallcontact point and one dashboard) behind a local otel-mcp-server started withGRAFANA_URL+GRAFANA_AUTH_BASICand no MCP auth keys, recorded throughMCP_URL=http://127.0.0.1:3011/mcp npm run record-fixtures -- --write --out .tmp-recorded-local. The threesynthetic/grafana_*.jsonfiles are gone, replaced by recordings whose_recordedcarriesscrubbed: []with what was reviewed (jsonData/secureJsonFieldsempty; compose hostnames and the loopback Grafana URL inside a failed check's error kept verbatim) and the server line, added by hand. Two shapes the hand-written files had wrong:grafana_datasource_healthanswers{ datasource, health }—healthis{ supported: true, status: 'OK' | 'ERROR', message, details }when Grafana's endpoint answered 2xx and{ supported: false, error: 'HTTP 400: Bad Request — <url>' }when it did not (Grafana 12.4.4 answers a check that ran and failed, backend unreachable, withHTTP 400; an unknown uid with500) — so thehealth-objectcontract now locates['health', 'data', '']and accepts a verdict key or thesupported/errorpair (one key group: the checker requires every group and the two forms share no key), andobserveGrafanareadsOK→ok,ERROR→error,supported: false+HTTP 400→errorwith the error text as message, any othersupported: false→unknown(not checked, never "not unhealthy"),supported: truewithout a status →unknown; andgrafana_contact_pointsanswers the receivers API ({ count, contactPoints: [{ name, active, integrations: [{ name, sendResolved, lastNotifyAttempt, lastNotifyAttemptDuration }] }] }— nouid,typeorsettings, so no webhook URL or secret ever rides in it), not the provisioning shape. The recorder keeps both datasource-health verdicts when several uids answer (the first uid's answer asgrafana_datasource_health.json; the first answer of the other case as.ok.json/.error.jsonwith its uid and case, skipped when identical to the primary file). Suites replay the recording with expected values read from the files:test-contract-shapespins both health cases and the envelope location,test-fetch-livedrivesobserveGrafanawith the recorded uids and asserts the unhealthy datasources name exactlyLoki (absent)(plus the non-400 and no-statusunknowncases), the smoke and end-to-end fakes answer health per uid, and the recorder suite's host guard is the fake's own host:port.grafana_dashboards_search.jsonkeeps its richer 2026-06-12 recording (the stack answers the same{ count, results }shape). - Live validation tier (2026-09-07, the real products in Docker). The public MCP tier could only evidence what that stack happens to scrape;
docker/stack.compose.yaml(name: observogram-stack, loopback-only ports 19090 / 19093 / 18428 / 18880 / 18888+14318 / 13030 / 19115 / 19080 / 14269, disjoint from the validate stack) now runs every product the table names at an exactly pinned version — Prometheus v2.55.1 scraping all of them plus a blackbox probe job, Alertmanager v0.27.0 with a dead webhook receiver, VictoriaMetrics v1.113.0 scraping itself and a dead target, vmalert v1.113.0, otel-collector-contrib 0.115.1 with adebugexporter beside an OTLP exporter to a dead endpoint (queue + bounded retries), Grafana 12.4.4 with a provisioned Prometheus datasource and one provisioned always-firing alert rule, blackbox-exporter v0.25.0, promtail 3.3.2 tailing a sample file into a dead Loki, and Jaeger all-in-one 1.62.0 — every ruler carrying one recording rule and one always-firing alert so the failure and notification counters move on a fresh stack (up -d --waitreturns in ~10s).tools/test-stack-live.mjs(npm run test:stack:live[:strict], NOT innpm test— it needs Docker and skips loudly without it) brings the stack up, recreates the collector and Grafana so neither carries a previous run's stimulus, waits for every scrape job to be up and for the rate windows to hold two samples, then for every alias of every row: (a) asserts eachrequiresname is a metric family on the product's own exposition (/metrics; the Prometheus TSDB name inventory for the scrape-synthesisedup/scrape_duration_seconds, which never appear on Prometheus' own/metrics; the blackbox/probeoutput forprobe_success; the vmalert service forvmalert_*), recording separately the presence of any name theexprreads thatrequiresdoes not demand (the lazily-registered-counter policy's shape); (b) evaluates theexpron the real Prometheus and reads the answer through the fetcher's ownsampleFromInstantVector(data/empty; a PromQL error is a FAIL naming the error); (c) records the collector's name set at startup, sends one OTLP/HTTP request per signal kind (and one query through Grafana's datasource proxy), and records the name set again. The ledger (alias | product@version from the image tag | exposition | query) is printed and written to the git-ignored.tmp-stack-live-ledger.json; any exposition ✗ or query ERROR exits non-zero. First ledger, table untouched by design (the next slice corrects it): 28 of 37 aliases expose every required name and every one of the 37 expressions evaluates without a PromQL error (25data, 12empty); the 9 ✗ are all otelcol — on 0.115.1 the internal telemetry carries no_totalsuffix (the bare-name aliases pass),otelcol_processor_dropped_*never exist (removed by the processorhelper rework), andotelcol_exporter_send_failed_*(likeotelcol_exporter_sent_*and every receiver/processor counter) are absent at startup and present only after the first export attempt — 10 names at startup, 39 the momentsend_failed_*register, 48 by the time that first ledger's collector had been retrying its dead exporter for some minutes (the counts are timing snapshots of one growing set; the 10 is stable) — whileotelcol_exporter_queue_size/queue_capacity/otelcol_process_uptimeare the always-present siblings. Also observed:otelcol_exporter_queue_sizecarries adata_typelabel thatqueue_capacitylacks, socollector_queue_saturation's division matches nothing (emptyon a healthy collector); Grafana'sgrafana_datasource_request_totalandgrafana_alerting_rule_evaluation_*register lazily (after the first proxied request / the first rule evaluation of an org); the two Grafana 5xx rows reademptyrather than 0 on a healthy Grafana because their label filter matches nothing. - Live validation tier — the alias table corrected from live exposition (2026-09-07). With the stack ledger and the public tier as evidence,
tools/lib/contracts/stack-self-metrics.mjsnow reads only names a real product exposed: every one of its 32 aliases carries averifiedstamp (<image:tag> <exposition|TSDB inventory|probe output> + PromQL, 2026-09-07) thatnpm run test:stack:livechecks against the compose image, and the live suite is a green gate — 32/32 aliases expose every required name (lazily-registered counters asserted present after the stimulus), 32/32 expressions readdataon the real Prometheus (the first run: 28/37 exposition, 12empty). What changed and why: (1) the six otelcol_totalaliases are gone — neither 0.115.1 (stack) nor 0.154.0 (public tier) suffixes its internal telemetry; (2) the threecollector_dropped_*rows are retired forcollector_refused_{metrics,spans,logs}overotelcol_receiver_refused_<kind>—otelcol_processor_dropped_*exist on no current collector in either spelling (removed by the processorhelper rework), while the receiver counters are the current-generation "collector is losing telemetry" signal (24 rows, 9 families unchanged); (3) the lazily-registered counter policy: the collector registerssend_failed_<kind>withsent_<kind>at the first export (0.115.1, observed before/after a forced failure: 10 names at startup, ≥39 after one OTLP request per kind — 39 the momentsend_failed_*register, more as the dead exporter retries; the never-failingdebugexporter then carriessend_failed_* 0) or, on 0.154.0, plausibly only on the first failure (send_failed_spansabsent whilesent_spansis present on the read-only public tier — consistent with registration on the first failure, not forced there), so those six aliasesrequirethe sibling that registers with or before the counter (otelcol_exporter_sent_<kind>,otelcol_receiver_accepted_<kind>) and end withor (count(<sibling>) * 0)— a scraped collector that has exported reads 0 unless the counter exists, one that never exported that signal readsnot-in-inventory, and a renamed counter renames the sibling so the alias never reads a false 0 (a bareor vector(0)was not used: the sampler's bounded cascade queries aliases without a trusted inventory, wherevector(0)would fabricate a healthy collector on a backend that has none — the reason the count rows already reject it); (4)collector_queue_saturationismax(max by (exporter, instance, job) (queue_size) / max by (exporter, instance, job) (queue_capacity))— on 0.115.1queue_sizecarriesdata_typeandqueue_capacitydoes not (the plain division matched nothing →empty), on 0.154.0 both do (ignoring(data_type) group_leftwould be many-to-many on a multi-pipeline collector); the per-exporter max is the fullest queue's fill on either; (5)datasource_errorsreads the reference pack'sgrafana_datasource_request_total{code=~"5.."}first — registered on Grafana's first datasource request, which is any rule evaluation,/api/ds/queryor proxied query, so every Grafana with a dashboard or a rule has it — and keeps the pre-registeredgrafana_proxy_response_status_total(code="500"at 0 from startup, but counting only the legacy/api/datasources/proxy/...path) as the fallback, documented as discrepancy 3 (this slice first shipped the proxy counter first; the review below swapped the order); (6) the two Grafana 5xx rate rows carry the presence-guarded zero (or (count(<base>) * 0)) so a healthy Grafana reads0instead ofempty; (7) confirmed unchanged against the expositions: vmalert's pluralerrors_total,vm_cache_entries{type="storage/hour_metric_ids"}, the summary labelsslice/quantile,alertmanager_*,promtail_dropped_entries_total,probe_successon the/probeoutput, andjaeger_collector_spans_dropped_totalon Jaeger 1.62.0's admin port with no flag — every one pre-registered at 0, so strictrequiresstay (the public tier's Jaeger is v2.18.0, an otelcol distribution: it answers through the collector rows and the jaeger alias reads an honestnot-in-inventorythere). Public tier re-run after the correction (read-only): 14 aliasesdata· 0empty· 0failed· 18not-in-inventoryof 32 (was 10 · 1 · 0 · 26 of 37), 13 of 24 rows with data; the trimmed inventory fixture grew to 43 names (grafana_proxy_response_status_total,otelcol_exporter_sent_spans,otelcol_receiver_accepted_spans) andrecorded-stack/holds six instant vectors again,collector_export_failures_spansanddatasource_errorsreplacing the queue and HTTP-errors files as their families' data rows. Suites:test-stack-self-metricspins theverifiedshape on every alias, the sibling guard on exactly six lazy aliases, the absence ofvector(0)/ otelcol_total/processor_dropped_*, and the collector row ids;test-stack-liveasserts lazy counters after the stimulus and the stamp-vs-image match, and strips grouping clauses before reading metric names out of an expr.docs/MCP_INTEGRATION.mdgains a "Live validation tier" section with the verification ledger per product@version. - Live validation tier — review corrections (2026-09-07). Reproduced on the running stack (Grafana 12.4.4): an
/api/ds/query— the path dashboards and Grafana-managed rules use — incrementsgrafana_datasource_request_totaland leaves everygrafana_proxy_response_status_totalseries untouched (the proxy counter counts only the legacy/api/datasources/proxy/...path; a proxied call increments both), and the sampler stops at the first alias that answersdata. With the proxy counter first,datasource_errorsread a pre-registered 0 forever on any Grafana whose datasource queries were failing 5xx through/api/ds/query— a systematically blind 0 on the surfaces. The row now reads the referencegrafana_datasource_request_totalfirst (strictrequires) and the proxy counter as the fallback for a Grafana whose inventory lacks the reference name (the public tier: its first alias readsnot-in-inventoryand the row falls through to the proxy alias, sorecorded-stack/datasource_errors.jsonkeeps the proxy expression as its provenance query);test-stack-self-metricspins the new order and the reason. Also from the review: the live suite snapshots Grafana's/metricsthe moment--force-recreate --wait grafanareturns (the provisioned 10s rule is Grafana's own first datasource request, so its before column is "before the suite's stimulus", not "at startup" — the collector's is exact), every suite request carries a 10sAbortSignal.timeout, the compose header says one run at a time (two concurrent runs recreate the collector and Grafana under each other) and the Grafana provisioning mount is:rolike every other mount; the collector name counts are stated as the timing snapshots they are (10 at startup is stable; 39 the momentsend_failed_*register, 48 minutes later), the 0.154.0 observation is worded as the read-only inference it is, the pre-registered-at-0 citation names the right libraries (client_golang forprometheus_*/alertmanager_*/promtail_*/jaeger_collector_*; VictoriaMetrics' ownmetricslibrary forvm_*, and vmalert's*_rules_errors_totalper loaded rule — per-rule label sets, none on a rule-less vmalert),SRC.prometheusno longer reads as ifprometheus_*were confirmed on the public tier (no Prometheus server there; onlyup/scrape_duration_secondswere), andotel/opentelemetry-collector-contrib:0.115.1self-reportingservice_version=0.115.0is noted beside the stamp. The shipped lazy-counter policy deliberately deviates from the brief's wording (an always-present sibling +or vector(0)): it requires the per-kind sibling that registers with the counter and ends withor (count(<sibling>) * 0), because the sampler's bounded cascade queries aliases without a trusted inventory, wherevector(0)would fabricate a healthy collector on a backend that has none — the canonical statement is the "Lazily-registered counters" paragraph ofdocs/MCP_INTEGRATION.md, the contracts header points at it. - Review follow-ups (honesty re-audit of the slice). Six places where the panel could read as assurance for something never sampled, plus the names the review checked against upstream source:
- Alias table. vmalert's ruler counters are
vmalert_recording_rules_errors_total/vmalert_alerting_rules_errors_total(plural — the singular names could never answer on the recorded VictoriaMetrics + vmalert reference stack).query_latency_p99now readsmax(prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.99"}): Prometheus registers that metric as a summary, so the_bucketthe reference pack (and the old row) quantiled over never exists — documented as known discrepancy 2 beside the scrape-duration one. The count rows carry a presence-guarded zero,count(up == 0) or (count(up) * 0)(andsum(up) / count(up)for the ratio), so a healthy stack reads0instead ofemptywhile a backend without the metric still readsempty—or vector(0)was rejected because it would fabricate "0 down" where nothing is scraped.SRC.otelcolnow says theotelcol_processor_dropped_*counters exist only on collectors before the processorhelper metric rework. - Sampler. A series-only instant vector (metric identities, no sample) is
emptyfor every unit — acountrow used to turn it into "1 target down". The metric-name inventory is evidence of presence, never of absence: only an inventory that carriesupgates a row tonot-in-inventory(with the inventory size in the reason); an inventory withoutup— the recorded 25-name fixture, an empty list, a capped server — gates nothing and the rows are queried on the bounded cascade (queried anywayin the reason of an empty answer). With a trusted inventory every eligible alias may be tried (the 2-per-row cap applies to the fallback cascade only).hasToolsListis whether thetools/listRPC succeeded, so an empty advertised list readsnot-attempted(tier), not 24 ×failed. New exportstackInventoryTrust(inventory); the sampler result carriesinventory: { size, trusted, reason }. - Shapes + observers. Object shapes locate envelope-first (
objectAt: ['data', ''], newlocateObjectPayload(shapeId, response)the fetcher reuses): a Prometheus-API-style{ status: 'success', data: {...} }wrapper used to satisfystatus-object/health-objectat its root, reading versionnull/ clustersuccessand a wrapped{ status: 'ERROR' }health verdict asok.observeAlertmanager/observeGrafanareturnnullonly when none of their tools is advertised; an advertised tool that fails (HTTP error, timeout, bad shape) yields a non-null result witherror, annotated asmcp.observed.alertmanager.error/mcp.observed.grafana.error, carried onsummary.alertmanager.error/summary.grafana.errorwith a… status probe failed — <error>warning, and rendered— probe failed: <error>—— not exposedis reserved for tools absent fromtools/list. - Studio. The datasources row never prints
0 unhealthyfor a surface nothing checked:N · health not checked (grafana_datasource_health not exposed or did not answer)when every health isunknown, otherwiseN · M error: <names> · K unchecked: <names>(summary.grafana.healthCheckedcounts the verdicts). The not-attempted row renderssummary.stack.reason; a family with no observed row reads— not attempted: call budget exhaustedinstead ofnot attempted: not attempted. - Recorder. Committed fixtures'
_recordedprovenance carries tool + timestamp (+ query) only — the maintainer's MCP hostname stays in the git-ignored review copy; every query-string value ofMCP_URLis redacted alongsideMCP_AUTH(a URL-borne token echoed by an upstream error body comes out<redacted>); the report prints the fetcher's inventory-trust verdict and the inventory size beside the not-in-inventory count. - Docs + suites.
MCP_INTEGRATION.mdsampling policy rewritten for the trust rule (the old sentence claimed a trimmed fixture made every alias eligible while the code gated everything); the annotation reference listsmcp.observed.alertmanager/grafana.*/grafana.error; new cases intest-fetch-live.mjs(series-only count rows, trusted vs untrusted vs empty inventory, no per-row cap with an inventory, advertised-but-failing and envelope observers, error annotations),test-contract-shapes.mjs(envelope location),test-stack-self-metrics.mjs(pinned upstream names, guarded zeros, discrepancy 2),test-record-fixtures.mjs(URL-only token, no host in provenance, inventory-trust line).
- Alias table. vmalert's ruler counters are
- Suites:
tools/test-stack-self-metrics.mjs(npm run test:stack) — unique ids, allowed sets, every alias'srequires⊆ the metric names itsexprreads, everyreferenceSliresolves in its reference pack, resolver purity;tools/test-fetch-live.mjs— sampler cases (inventory eligibility, product preference, data/empty/failed, NaN → empty, both envelopes, bounded cascade, global cap, restricted tier, no tools/list,quiet()family), observer cases, the annotation contract (schema-valid, no Verified/Scaffold change, adapted layers unchanged) and an end-to-end run against the fake JSON-RPC server;server/test-smoke.mjs(draft summary with and withoutmetrics_query, live-status fields);tools/test-journey.mjs(stack facts, record, markdown); newtools/test-record-fixtures.mjs— the recorder against a fake MCP replaying the fixtures: report content and drift, no writes in report mode, every--writefile's trimming and shape, the token absent from every output and file, the restricted tier, missingMCP_URL;tools/test-contract-shapes.mjs— recorded precedence and therecorded-stack/scan;tools/test-contract-guard.mjsnow scans the recorder too. Docs:MCP_INTEGRATION.md"Stack self-metrics (registry / sampling / surfaces)" with everymcp.stack.*/mcp.observed.*key listed once in the annotation reference; README "Fetch Live From MCP".
Live evidence honesty: the fetcher stops asserting what it never observed
The top product risk is false assurance — a live pack reporting Verified / aligned / fresh for an artefact that is not doing its job. This entry removes the known false-assurance paths in tools/fetch-live-pack.mjs, makes the on-wire liveness signals visible, lets journeys gate on the vantage, and stops two authoring-only differences from reading as drift. Existing users should expect alignment percentages and grades to drop against the same MCP — placeholders that used to count as Verified now park as Scaffold, burn-rate alerts the platform never had no longer align with the ones the repo declares, and scrape jobs whose targets are all down no longer attest scraping. The lower number is the honest one.
False-assurance fixes
- Burn-rate alerts are mapped from discovered rules, never synthesised. The fetcher used to emit a two-window burn-rate alert for every inferred SLO and stamp
mcp.verified.policy.burn_rate_alertswhenever any alert name came back.spec.policy.burn_rate_alertsis now built from the alerting rules the MCP exposes — compiler-labelled rules (slo/burn_rate/window_short/window_long/severity) and<slo>_burn_<N>x_<short>_<long>-named rules, grouped per SLO, deduped, short window first — and each entry gets an indexedmcp.verified.policy.burn_rate_alerts[<i>]stamp (the unindexed key is gone). A group whose SLO shares the SLI base of an inferred placeholder re-identifies that SLO to the discovered id and replaces its placeholder objective/window from theslo_objective/slo_windowannotations; groups naming an SLO nobody inferred, or carrying a single window, land inmcp.discovered.alert_rules_unmapped; forecast rules are skipped; severities outside theSEVnenum are mapped from the Prometheus vocabulary or the burn factor and listed inmcp.discovered.alert_rules_severity_inferred. When nothing maps, the schema-forced placeholder is stampedmcp.scaffold.policy.burn_rate_alerts[0]. - Every placeholder the fetcher invents is
Scaffold, neverVerified.mcp.scaffold.<symbol>(the live counterpart ofcrawler.scaffold.<symbol>;tools/lib/adapter.mjssourceOfnow honours both prefixes) marks the hard-codedspec.otelblock (system_healthanswering used to verify it), collector receivers/processors, the logs/traces exporters, the fallbacklogs-elasticbackend, theplatform-overviewdashboard stub, the SEV1 → Teams route, the baselines, and the per-service /platform_availabilitySLI+SLO guesses.pipelines.exporters.metricsisVerifiedonly when scrape targets or a metric inventory evidenced it;metrics-promonly on a*_build_infocapture;traces-jaegeronly when the topology names jaeger ortraces_servicesanswered. SLIs inferred from real recorded rules stayVerified; an SLO re-identified by a discovered burn-rate group becomesVerified. - MTTD is no longer derived from anomaly thresholds.
spec.baselines.mttd_target_p50was the smallestanomalies_baselinesthresholdMs(e.g.90ms) stampedVerified— a latency-anomaly threshold is not a time-to-detect target. Baselines are now the platform default for the declared criticality (measurement_source: platform-default, stampedScaffold) until a §5.10 measurement exists;mcp.baselinesComputedstill counts the anomaly baselines returned, which is evidence the tool answered, not a measurement.
Liveness evidence (on-wire health)
- A scrape job NAME used to be enough to stamp
mcp.verified.telemetry.scrapeandmcp.verified.pipelines.exporters.metrics, and a rule's presence was enough to verify it, even when the same payload said every target wasdownor the rule was failing to evaluate. Thescrape_configsprobe now adapts to per-job target lists ([{job, targets:[{instance, health, lastScrape, lastError}]}], tolerant of otel-mcp-server'smetrics_targetsand Prometheus/api/v1/targets): only jobs with at least one targetup(or of unknown health) land inmcp.discovered.scrape_jobsand attest scraping; jobs whose every target is down go tomcp.discovered.scrape_jobs_down; every target's health is kept inmcp.observed.scrape_targets(JSON, 200 entries,lastErrortrimmed to 200 chars). When no job is up the metrics exporter falls back to its scaffold marker. - The
recording_rules/alert_rulesadapters carryhealth,lastError,lastEvaluation,evaluationTime(andstate,activeAtfor alerting rules); the fetcher strips them beforespec.queriesand writesmcp.observed.recording_rules/mcp.observed.alert_rulesplusmcp.discovered.recording_rules_unhealthy/alert_rules_unhealthy. Recording rules are stamped per index (mcp.verified.queries.recording_rules[<i>]— the symbol the adapter actually reads; they used to projectDeclareddespite the group stamp) and only when healthy or health-less; a burn-rate group fed by an unhealthy alerting rule still maps but earns no indexed stamp. Absent health is never treated as unhealthy; legacy job-namestring[]probe results still count as health-less scrape evidence. - Draft summary (
POST /api/draft-from-mcp) gainsscrapeJobsDown,recordingRulesUnhealthy,alertRulesUnhealthy; the studio listsN scrape jobs down: …andN rules unhealthy: …when non-empty.
Probe honesty (what the MCP could and could not answer)
- A probe family whose every candidate is absent from
tools/listwas recorded asunsupportedby the probe loop but never emitted; the studio read it as "probed, none found" and warned that the family "returned empty". The pack now stampsmcp.probesUnsupported(a restricted MCP tier) besidemcp.probesFailed(a hole of unknown size), andmcp.probeErrors.<family>carries the last erroring candidate's message (trimmed to 200 chars).fetchMcpreturns itsprobeFailuresmap;buildCanonicalPackaccepts it. partialLiveEvidence(vendored, zero-import) gains additive keys only:unsupported,errors, andvantage(full/partial/restricted/lost/none);partialkeeps meaning outright failures.GET /api/live-statusreturnsprobesFailedandprobesUnsupported; the MCP badge goes to its error state for failed probe families and names a restricted tier in its title; the draft summary lists unsupported families as "— not exposed by this MCP" (with the probe error inline for failed ones) and warns once about the restricted tier; the partial-evidence banners append the not-exposed families.
Journeys: studio grade parity, vantage-aware gate, vantage-lost records
- One score for one comparison.
tools/lib/journey.mjsattaches the requirement-chain comparison (comparePackBranches) to the diff astraceabilityGraphbefore grading — exactly what the studio's/api/diffdoes — so the CLI and the studio score Drift-free on the same construct. The run record says which construct scored it (grade.driftConstruct:requirement-chain/diff-buckets) and carries the chain rollup (traceability). - The vantage is part of the verdict. Run records gain
probes {attempted, succeeded, empty, failed, unsupported},probeErrors,vantage,toolsExposedCount,scrapeJobsDownandunhealthyRules, read from Pack B's fetcher annotations (a file-sourced B reports empty lists,0,null,none). Two new gate keys:failOnPartialEvidence: truebreaches when any probe family failed or the vantage is entirely lost;maxUnhealthy: Nbreaches when scrape jobs down + unhealthy rules exceed N. Empty and unsupported families never breach on their own. The markdown report printsLive probesandOn-wire healthlines; the README gate example shows both keys. - Vantage loss is a point in history, not a hole. When Pack B is an
mcp:source and the fetch itself fails, the journey writes anoutcome: vantage-lostrecord (error, both sources, timing, no fabricated grade) before rethrowing — exit code stays2, the studio route still answers 502. Configuration errors (missing file, unsetauthEnv) leave no record.packc journey listand the studio Journeys view render the outcome without a fakealignment undefined%.
Canon: two authoring-only differences stop reading as drift
tools/lib/promql-canon.mjsremoves the spaces around the symbolic binary operators+ * / % ^ == != <= >= < > =~ !~(outside strings, parser-proven expressions only), soa / b≡a/bandrate(x[5m]) > 0.5≡rate(x[5m])>0.5.-is left alone (unary/binary ambiguity), keyword operators keep the space that bounds them, label-value strings stay byte-identical, and a space inside an operator (a < = b) is never glued into a different one. Decision-bearing differences (>vs>=, thresholds,=~vs!~) still read as drift.tools/lib/artefact-model.mjsbehaviorForstrips a leadingref:fromslo,sli,trigger,error_budget_policy, and fromexprwhen the whole value is a reference (ref:slis.x), via a purestripRefFieldscopy of the spec — identity already stripped it, so a burn alert declaredslo: ref:xpaired with the liveslo: xand then reported itself drifted on its own binding. Both modules are vendored downstream (docs/VENDORING.md): additive behaviour only, no new imports, signatures unchanged — re-copy the pair together.
Review follow-ups (false-assurance re-audit)
- Placeholders never pair. The studio construct (
diffPacks+comparePackBranches, which the journey now grades on) still paired a liveScaffoldplaceholder with a declared artefact: a declared burn-rate alert on the live pack's first SLO readalignedwhen it carried the compiler's default windows (exactly the placeholder's) ordriftedotherwise — neverdeclared, not live— and BAK-01 / PIP-PRC-01/02 / the SEV1 route aligned or drifted against scaffold backends, processors and the fallback route. Only the diff-bucket grade path parked them.diffPacksnow removessource: Scaffoldartefacts from both sides before pairing into a newscaffoldbucket (summary.scaffold; a repo placeholder is not a declaration either, so a real live route the repo only had a placeholder for readslive, not declared);comparePackBranchesrefuses aScaffoldlive node as evidence (declared_only), never emits it aslive_only, drops a live-only branch rooted on a scaffold SLO, and does not count a scaffold route/dashboard as proof the live connector can see that kind. Journey facts (alignmentPct,aligned,declaredNotLive→maxDeclaredNotLive) follow; the run record carriesdrift.scaffold. The studio drift drill, verdict and prototype views read the new bucket beside the olderisScaffoldDiffEntryfilter. - Tiered SLOs on one SLI no longer produce a dangling ref.
mapDiscoveredBurnAlertsresolved groups in one pass and only tracked re-id candidates as claimed, so anx_99_9group could re-identify the SLO anx_99group had already bound exactly —spec.slosthen heldx_99_9whileburn_rate_alerts[0].slostill saidx_99(POL-01 Verified with a ref pointing at nothing; the schema's pattern-onlyRefnever caught it). Exact matches now bind and claim first; base matches re-identify only unclaimed placeholders; the leftover group is reported inmcp.discovered.alert_rules_unmapped. Exact matches apply theslo_objective/slo_windowevidence and drop the SLO's scaffold marker like a re-id does; a re-id to an id that is not a schema Slug (length included) is refused. - Health follows the evidence down the chain. An SLI inferred from recorded rules of which one is reported unhealthy is no longer stamped
mcp.verified.slis.<id>(newmcp.discovered.slis_unhealthy); an SLO bound by a burn group fed by an unhealthy rule is not stamped either; rule health is keyed by name with any-unhealthy-wins semantics (a duplicated name reportederrin one group andokin another stamped both spec entries Verified while being named unhealthy); the requirement traceability chain lists an unhealthy live alerting rule withverified: false/health: err, does not let it (or a scaffold burn-rate entry) closemissing_alert_evidence, and countswithAlertson evidence only — the studio chain rows and the requirement-trace drawer show(unhealthy). - Under-reporting fixed too. Discovered dashboards are stamped per id (
mcp.verified.dashboards.<id>, the symbol the adapter reads) beside the aggregate key, so they projectVerifiedinstead ofDeclared.mcp.probeErrors.<family>is written only for families whose outcome isfailed— a family whose later candidate answered no longer carries the earlier candidate's error while sitting inprobesSucceeded. - Fresh names the vantage. The grade's Fresh criterion keeps its staleness pass/fail (scoring is out of scope) but its detail now says
vantage lost: no probe family answered (failed: …; not exposed: …)orvantage partial: failed …, so a refresh that saw nothing does not read as a fresh look at production..journey-outcome.is-lostuses a new--warntext token instead of a hard-coded colour.
Fixtures re-pinned (npm run test:contracts:update): tools/fixtures/mcp/adapted/scrape_configs.json is now per-job target health instead of a job-name string[] (the alertmanager job is present with its down target and lastError); recording_rules.json / alert_rules.json carry the observation fields (health, lastError, lastEvaluation, evaluationTime, and state / activeAt). No crawler or compiler golden changed.
Suites: tools/test-fetch-live.mjs (placeholder + mapping paths, every scaffold key and its adapter projection, the excluded down job / all-down / legacy / adapter-shape cases, per-index rule stamps, the unsupported/failed split and per-family error mapping), tools/test-diagnostic-grade.mjs (every vantage), server/test-smoke.mjs (live-status fields on a planted live pack), tools/test-journey.mjs (both gate keys, record fields, studio parity, the vantage-lost record via a closed loopback port, the no-record rule), tools/test-promql-canon.mjs and tools/test-diff.mjs (operators, fences, ref: ≡ bare). Docs: MCP_INTEGRATION.md (annotation reference, what the fetcher invents, on-wire liveness, burn mapping, journeys), DIFF.md, ADAPTER.md, VENDORING.md, README.
Studio Diagnose views rejoin the behavioural engine
- Diagnose → Compare cards classify from the diff again. The side-by-side view derived its in-both / only-in chips by recomputing keys client-side as
defines || id— a lossy approximation of the old diff keyspace that stopped matching entirely once the engine moved to behavioural identity keys (kind::{identity}): the summary band said N matched while all cards below rendered unclassified and the "In both" / "A − B" slice filters returned nonsense. Cards and slice filters now classify by the artefact ids embedded in the diff entries themselves (unique per pack side) — no client-side key math to drift again. - Traceability (Advanced) is diff-driven. Its repo-vs-live buckets were re-derived locally by raw id match + whole-object JSON equality (only
annotations/_substripped), so behaviourally-identical artefacts with different ids double-counted as declared-not-verified and verified-not-declared, and endpoint/expression-formatting differences flagged Stale. The four buckets now re-bin the server diff — aligned = inBoth-aligned, stale = inBoth-drifted (with the engine's delta fields named on each row) — and reconcile with the drift drill's unlensed totals: out-of-scope artefacts and scaffold placeholders are parked the same way (counts + live-scope control shown), while the drill additionally applies the active product lens. Panels stay compared at dashboard granularity (the engine's rule), now stated in the view. The view fetches the diff itself when it isn't loaded yet. Suppress/resolve prefs now key on behavioural identity keys (collision entries append the artefact symbol so#NNordinal reshuffles don't migrate a suppression), so previously-saved entries no longer match and reset once. - Stale-diff hardening.
state.diffis stamped with the selection it was computed for (packs + envs + scope + service) and every consumer treats a mismatch as no diff, refetching instead of rendering — previously a Pack A/env switch could leave the views classifying the new pack against the old diff's positional ids. Late responses for a superseded selection are discarded, concurrent renders share one in-flight fetch, and the three loading branches only re-render on progress (an unregistered Pack A now gets an honest error instead of a render loop). prettyDiffKeylearned the behavioural keyspace (kind::{identity-json}with#NNoccurrence ordinals) — backend, pipeline-stage, storage, mesh/collection, product-keyed, scrape-job, imports, and remediation identities render as readable labels instead of raw JSON, and forecast alerts are no longer mislabelled as burn-rate alerts.
Account menu: profile, change password, sign out
- The identity chip in the studio header is now an account menu: who you're signed in as (and the auth mode), change password… (stand-alone mode — OIDC passwords belong to the IdP), and sign out. It renders in identity postures only; the open posture (
OBSERVOGRAM_AUTH=offor no users file) has no accounts to manage. - Signed-in users can finally change their password:
/auth/change-passwordnow also accepts a session. The current password is required — that knowledge is the CSRF defence — and wrong guesses share the login damper (5 failures → 30s lockout). The mid-login forced-change flow (flow cookie, skip) is behaviourally preserved, with one deliberate alignment: a user deleted mid-flow now gets "unknown user" before password validation, matching the session path. - Login now clears a leftover pwchange flow cookie. An abandoned forced change (admin/admin typed on a shared browser, then closed) used to leave a 10-minute flow cookie that shadowed the session on
/auth/change-password— the account menu's link could silently target the other account. A normal login expires it. - Known limitation (stateless sessions, pre-existing model): rotating a password does not revoke other outstanding sessions — an already-stolen cookie rides out its TTL (default 8h). Real revocation needs a per-user epoch in
users.jsonchecked at session verification. - Suite: 15 new assertions in
server/test-auth-local.mjs— the session variant of the page (current-password field, no skip control; the flow variant is pinned to lack the field), wrong-current 401 in JSON and HTML, short-replacement 400, rotation in JSON and HTML (302 home), old password dead / new signs in, deleted-user GET/POST with a live session, the stale-flow-cookie clear on login, the 429 lockout, and the damper being shared with/auth/login.
"Skip for now" on the forced password change
- The first-boot change-password page now offers Skip for now — ask again next sign-in: a session is issued without replacing
admin/admin, the forced-change flags stay set, and every subsequent sign-in re-prompts. NewPOST /auth/change-password/skip, credentialed by the same signed flow cookie as the change itself. - Deliberately narrow: skipping works only while the record still holds the seeded default — an admin-set temporary password (
mustChangewithoutseededDefault) stays a forced change, and the loopback guard is unaffected (a skipped default still refuses to bind beyond loopback). - Suite: 12 new assertions in
server/test-auth-local.mjs— skip control rendering, no-cookie rejection, session issuance with flags intact (JSON and browser-form paths), network-bind refusal after a skip, re-prompt at next sign-in, and the temporary-password 403 (JSON and HTML, skip control absent).
Explicit collision reporting + collision-proof coverage
- New top-level
collisionson thediffPacks()result ({ layer, kind, key, aCount, bCount }) andgraph.collisionsonbuildDependencyGraph()({ key, kind, count }): every identity key held by more than one artefact, surfaced explicitly so callers can fail loud instead of inferring collisions from#NNoccurrence suffixes. Additive — no existing bucket, key, or summary count changes. - Alert-route identity deliberately stays severity-keyed. Widening it with channel kinds was prototyped and declined: the live connector and crawler fabricate a channel kind when routing cannot be introspected (fetch-live's SEV1 msteams placeholder, unmapped-receiver stubs), so channel-kind identity would split declared routes from their live placeholders and report them falsely missing in production. Channel changes stay decision-bearing drift on the paired route; a new regression test pins the placeholder pairing.
otel/baselinesempty identities and pipeline-stage{name}identity are now documented invariants (spec-singular objects; collectorbatch/2-style duplicates survive via occurrence ordinals), in code comments and docs/DIFF.md.- Vendoring contract hardened (docs/VENDORING.md):
tools/lib/artefact-model.mjs,promql-canon.mjs, andpromql.mjsjoin the vendorable table and the staying-current diff command — they were alwaysdiff.mjs's import closure, but a downstream copy that missed them could silently mix engine generations (the root cause behind a downstream studio's 2026-08 collision report, filed against code replaced on 2026-06-09). - Compiler: duplicate-severity routes now emit a valid, complete Alertmanager config. Bringing the crawled pack into strict CI validation exposed the same severity-collision bug in
compileAlertmanager(): one receiver per route named by severity, so N same-severity routes emitted N identically-named receivers (amtool check-configrejects the file) — and their identical-matcher sibling routes meant only the first could ever fire. Duplicates now get ordinal receiver names (svc-sev2-2, …) and identical-matcher groups chain withcontinue: true, so every declared channel notifies. Regression-tested intools/test-compile.mjs. - Suite: the collision fixture grew same-named pipeline stages and a fourth same-severity route;
tools/test-packs.mjsnow asserts the self-diff invariant (every artefact preserved, alignment/Jaccard 1.0) for every bundled pack;examples/krystaline.repo.pack.Carlos.yamlrenamed tokrystaline-repo-carlos.pack.yamlso the one crawled real-world pack (324 artefacts) finally enters the suite, with its genuine collisions pinned inPACK_EXPECTATIONS;tools/test-traceability-graph.mjsgained duplicate-identity-key coverage.
Chronicled retroactively: the identity-collision safety net (2026-06-09)
- Never changelogged when it landed (commit
4c5bae4, between 0.3.0 and this line):diffPacksgroups identity keys instead of last-write-winsMap.set, pairs exact behavioural matches first, and preserves duplicates with stable#NNoccurrence suffixes;canonicalize()began treating empty arrays like absent fields in the same commit.buildDependencyGraph(created the same day in6fcb7ef) has had the grouped behaviour from birth. If a vendoreddiff.mjsstill containsaByKey.set(keyOf(a), a), it predates 2026-06-09 — re-vendor the full engine closure.
Ships like Grafana: default admin on first boot
- First boot with nothing configured now seeds a default
adminuser (passwordadmin) and the login page is live — the studio is a signed-in app out of the box, no CLI setup. The password change is forced at first sign-in (new/auth/change-password, signed short-lived flow cookie, no session until the change lands). Softened later in this release: skippable per session for the seeded default — see the "Skip for now" entry above. - The default credential never crosses loopback: a network boot never seeds
admin/admin(it refuses to start and names the options), andstart()refuses to bind beyond loopback while an already-seeded default is unchanged. Container/network boots setOBSERVOGRAM_ADMIN_PASSWORD=<secret>to seed a real credential (no forced change); the same env var also replaces a still-default admin record, so a workspace seeded on loopback can be moved behind the network without a deadlock. - The seed backs off from any expressed intent: OIDC configured, an existing users file,
OBSERVOGRAM_API_TOKEN(the token-only 10B contract is unchanged), or armed tenancy. - New
OBSERVOGRAM_AUTH=off— the one switch that disables identity entirely (no login, no seeding,/auth/*answers 404): the pre-0.5 open posture, kept for dev shells, scripts and CI. This is a default-behaviour change:npm starton a fresh workspace now lands on a login page. - Stand-alone auth arming is now per-request (routes register once, posture follows the users file), so the seed at
start()andnpm run usersboth take effect without a re-import;npm run users -- passwdclears a pending forced-change flag. - Suite:
server/test-auth-local.mjsgrew 18 assertions — seed shape, 401-by-default, loopback guard, the full forced-change flow,OBSERVOGRAM_ADMIN_PASSWORD(fresh seed + rescue), never-seed-on-network-boot, and the off switch;server/test-smoke.mjsruns underOBSERVOGRAM_AUTH=offand is the open-posture regression.
Tomograph → Observogram (rebrand, backwards-compatible)
- Every user-facing name is now Observogram: package name and npm bin (
observogram;packcunchanged), studio shell, login page, CLI help, Docker image (observogram:<version>), k8s manifests, docs, and diagrams. - Nothing breaks on upgrade — the legacy spellings keep working via
tools/lib/brand-env.mjs(drop planned for 0.6):- Env vars: every knob reads
OBSERVOGRAM_*first, then the legacyTOMOGRAPH_*(WORKSPACE,API_TOKEN[_LABEL],OIDC_*,SESSION_*,USERS_FILE,INSECURE_NO_AUTH,ALLOW_LOCAL_MCP,STRICT_SNAPSHOT,BUILD,DEBUG,DIFF_SCOPE,MCP_TIMEOUT_MS,GRAFANA_*). - Workspace: the default directory is
.observogram/, but an existing.tomograph/keeps being used as-is (no migration needed, no data orphaned). - HTTP: the studio sends
X-Observogram-Org/X-Observogram-CSRF; the server accepts theX-Tomograph-*spellings too and echoesX-Observogram-Org. - Sessions: the cookie is
observogram_session; sessions issued pre-rebrand undertomo_sessionstay valid (same HMAC secret), and logout clears both. - Pack annotations: writers emit
observogram.*(diff.scopeMode,retrofeed.*,services); readers accept thetomograph.*namespace from pre-rebrand packs, new key wins when both exist.
- Env vars: every knob reads
- Legacy-compat is CI-asserted (workspace env fallback, org/CSRF headers, session cookie, annotation namespace).
Vendorable verdict engines (downstream-studio decoupling)
studio/diagnostic-grade.mjsis now zero-dependency (L4_SUBGROUPSinlined) — vendorable verbatim, liketools/lib/diff.mjsandtools/lib/protocols.mjs.studio/verdict-ui.mjsno longer reads global state:buildVerdictModel()/projectGrade()take{ pack, packB, diff, compareBId, catalogEntry }explicitly andloadRunHistory()accepts an injectablefetchFn.- New
studio/compare-catalog.mjs—catalogEntryFor()+LAYERS_FOR_DIFFextracted from the 3000-line compare-view (which re-exports them), so verdict modules import the small catalog module instead of the whole view. - See the new docs/VENDORING.md for the downstream contract.
The studio host seam (no view imports app.mjs for orchestration)
- New
studio/host.mjs— a zero-import module holding the app-level callbacks (loadPackB,openDeployModal,renderMainView,renderTabs); app.mjs fills it once at boot viainitHost(). Every view module (compare, compile, layers, drawer, journeys, references, atlas, all protos) now calls these throughhost(imported asappHost) instead of importing app.mjs — app-specific helpers remain direct imports until each view adopts the model-passing convention. - New
tools/test-studio-graph.mjsinnpm test— links the entire studio ES-module graph headlessly, so a dangling import/export anywhere in the studio fails CI (node --check is parse-only and cannot catch it). - New
studio/verdict-ui.css— the.mc-*widget styles verdict-ui.mjs emits, split out of app.css along the vendoring seam (the file header documents the theme variables it expects). - New docs/UI_CONVENTIONS.md — the adopt-on-touch conventions: host seam, loader/renderer split with injectable
fetchFn,render(container, model, host)signature, and per-zone CSS prefixes. - Rebrand follow-through: the last "tomogram" vocabulary is now "observogram" (Discover's scan title/tagline, hero-asset path
assets/observogram-hero.png, docs).
Fixed: comparing legacy imports (and any cross-service packs)
- The PACK B picker filtered out every pack whose service didn't match the active one — two legacy imports (each deriving its own service from the old pack id) could never be compared. Cross-service packs are now selectable under an "other services" group; same-service packs and live aggregates still lead the list, and the diff's service scope keeps cross-service comparisons honest.
Stage 2 tenancy — workspace-per-org
- New
server/tenancy.mjs: org registry in<workspace>/orgs.json(the file existing arms tenancy, mirroringusers.json), per-request org context via AsyncLocalStorage, idempotent flat →orgs/default/boot migration. Requires identity; refuses to boot otherwise (fail-closed). workspaceRoot()is context-aware: registry, deploys, snapshots, journeys, runs all answer from<workspace>/orgs/<orgId>/inside a request. The in-memory upload registry and workspace index cache are keyed per org.- Org selection via
X-Observogram-Org(default: first membership); membership enforced in middleware; bearer token = deployment-level service account. NewGET /api/orgs;/auth/mecarries memberships; roles recorded for Stage 3 (not yet enforced). - New CLI
npm run orgs -- create|remove|add-member|remove-member|list. - Studio: active org resolved before the first catalog fetch; ORG chip (switcher when multi-org) in the OBSERVA bar.
- New suite
server/test-tenancy.mjs— the isolation gate: org B reads/writes nothing of org A (API + filesystem-path assertions), migration, per-org reset, fail-closed posture.
Previous pack format (layered JSON) supported again
- New
tools/lib/legacy.mjs— detects the pre-v1.2 layered "studio-shape" JSON and upconverts it to a canonical v1.2 manifest. Lossless (every legacy artefact kept verbatim inlegacy.artefact.*annotations), honest (every schema-forced placeholder markedcrawler.scaffold.*→ projects as Scaffold, never Declared), deterministic. POST /api/validateupconverts legacy uploads transparently; the studio toast reports the conversion (N artefacts mapped, M scaffolds).- New CLI
npm run upconvert-legacy <file> [-o out.pack.json]. - The four original layered JSON packs restored from git history as working examples:
examples/legacy/(+ README documenting the format). - New suite
tools/test-legacy-pack.mjsgates all four examples on everynpm test; the validator's gatekeeper error now points legacy packs at the converter.