Skip to content

v0.4.0 — the monitor-of-monitors release

Choose a tag to compare

@MoebiusX MoebiusX released this 20 Sep 18:00
· 763 commits to develop since this release
428c1c5

The monitor-of-monitors release. Observogram assures that the artefacts which monitor a system keep doing their job: the live fetcher stops asserting what it never observed (steps 0-1), the stack samples its own self-metrics through the MCP (step 2), journeys keep stack-health evidence per run (step 3), and the requirement graph shows what would go blind when an artefact dies, a per-node ladder and candidate causes (step 4). The pack generators (boards including a Unified Observability board for every pack, burn-rate rules with the live-measured PromQL, gen-site for a fleet of environments from one pack and an inventory) became one vendorable library and the compiler emits the same PromQL. Also: stage 2 tenancy, first-boot admin and account menu, Diagnose views on the behavioural engine, the Tomograph to Observogram rebrand, legacy pack format support.

Blind-spot blast radius on the graph (step 4)

Step 3 kept the stack's own self-metrics per journey run; step 4 puts the monitor-of-monitors question on the requirement graph itself: what would go blind if an artefact died (structural exposure), whether a present artefact is doing its job or the vantage could not look at all (the per-node ladder), what changed since the previous run, and which candidate causes the evidence can offer. Three things are true of everything in this block, by contract and pinned by the suites: nothing scored changed — branch.integrity / verdict / counts, every node status, rollup.integrityMean and the grade's Drift-free criterion are byte-identical with and without the new annotations, everything new is an additive field beside them, and the switch to the ladder is a proposal (docs/SCORING_PROPOSAL_LADDER_INTEGRITY.md), not applied; the honesty vocabulary is kept — a blast radius says what WOULD go blind, never that something IS blind, unobserved means the vantage could not look (a probe family failed or is not exposed by the tier), never "absent", and present_unhealthy means on the wire but not doing its job; and causes are candidate causes ranked by evidence — not a root-cause verdict, with a change of the vantage itself reported separately and never as a cause. No new MCP tool name, no new dependency, no build step; the two new reasoning modules are zero-import and vendorable.

  • Blast radius (reverse index). New zero-import tools/lib/blast-radius.mjs: CONSUMER_SIDE (per edge type, which side consumes the other's assurance — from for sli_of, sources, exported_by, produced_by, protects, forecasts, visualises, contains, routes, remediates, validates; to for materialises, where the SLI consumes the rule's series), PROTECTION_SIDE (the to side loses when the from side dies: an SLO loses protection with its alert, an alert loses delivery with its route and remediation with its remediation — transitive, a dead route means an undelivered alert means an unprotected SLO), normalizeGraphShape (nodes as an array, a Map or a keyed object; edges without from / to / type dropped; edge types outside the table carry no known direction and are ignored), blastRadiusOf(shape, key, { weights }) → { key, kind, label, blinded: { total, weight, byKind, nodes: [{ key, kind, label, hop }] }, unprotected: { slos, alerts }, summary: { slos, alerts, panels, dashboards, routes, remediations, total } } (null for an unknown key; breadth-first over the reverse adjacency, cycle-safe; blinded.nodes sorted hop → KIND_ORDER → label and capped at BLINDED_NODES_CAP = 64 while total / byKind stay uncapped; protection losers already blinded are not listed twice; weight sums DEFAULT_WEIGHTS, a copy of the graph's limb weights, unknown kind 0.5) and blastRadiusIndex(shape) → Map key → summary in sorted key order. Scaffold placeholders (a scaffold flag or artefact.source === 'Scaffold') never blind anything, are never traversed and are never listed. traceability-graph.mjs exports graphShape(graph) — the artefact-free projection { nodes: [{ key, identityKey, kind, layer, label, virtual, scaffold }], edges: [{ key, from, to, type, provenance }] } — and compareBranches computes the A-graph and B-graph indexes once per comparison, attaching blastRadius: summary | null to every node verdict (A's index for aligned / drifted / declared-only / unverifiable nodes, B's for live-only ones).
  • Per-node ladder (additive, unscored) + the scoring proposal. Every node verdict gains ladder: { rung, status, detail } — rungs unobserved < absent < exists < alive < healthy, status present_unhealthy | present_stale | unobserved | null — read from what the live fetcher wrote into Pack B: mcp.observed.scrape_targets / recording_rules / alert_rules, the mcp.discovered.scrape_jobs_down / recording_rules_unhealthy / alert_rules_unhealthy / slis_unhealthy lists, mcp.probesFailed / mcp.probesUnsupported with mcp.probeErrors.<family>, mcp.versions.<product> for backends and mcp.refreshedAt as "now" (a timestamp older than 2× the artefact's interval reads present_stale; no interval or no timestamp means staleness is not judged). Burn-rate alerts link to live alert rules by the compiler's <slo>_burn_<factor>x_<short>_<long> convention, narrowed to the declared windows. A declared_only artefact the fetcher observed but withheld from Pack B (a scrape job whose every target is down) reads present_unhealthy — on the wire beats absent; a kind whose probe family failed or is not exposed reads unobserved, never absent; an unverifiable kind reads rung unobserved with status null; without any mcp. annotation every present node is exists. Per branch: ladderVerdict (broken > degraded > unobserved > healthy; undeclared mirrors the scored fields) and ladderIntegrity / ladderIntegrityPct on the same weights — unhealthy / stale credit 0.25, absent 0, unobserved nodes leave the denominator; rollup.ladder = { healthy, degraded, broken, unobserved, integrityMean, integrityPct } over the declared branches. docs/TRACEABILITY_GRAPH_COMPARISON_SPEC.md §5b carries the table and evidence rules; docs/SCORING_PROPOSAL_LADDER_INTEGRITY.md proposes that Drift-free read rollup.ladder.integrityMean under gradeSchema 3, with the four worked rows on the test fixture — a proposal through the PHASE_1 acceptance path, not applied.
  • Per-run persistence: chains, versions, transition, live pack. New zero-import tools/lib/chain-history.mjs: branchRecordsFromGraph(traceabilityGraph) keeps per branch { rootKey, title, rootKind, verdict, ladderVerdict, integrityPct, ladderIntegrityPct, confidence, missingRoles, degraded, truncated? } with the degraded nodes (declared_only / drifted / live_only, or ladder status present_unhealthy / present_stale / unobserved; aligned-and-healthy and unverifiable nodes are not recorded) worst first — absent, unhealthy, stale, drifted, unobserved, live-only, then widest blast radius — as { key, kind, label, status, ladder, blastRadius: summary, deltaFields } (caps 64 branches × 16 nodes × 6 delta fields, a cut list marked truncated); chainSummary(record) → { declaredTotal, intact, partial, broken, undeclared, ladder: { healthy, degraded, broken, unobserved }, integrityPct, ladderIntegrityPct, degradedNodes, topExposure: { label, kind, slos, alerts } | null } (means are null with no declared chain — an empty set is not 100 % healthy; null when the record carries no branches); diffRunBranches(previous, current) → { since, changed: [{ rootKey, title, from: { verdict, ladderVerdict }, to, direction, nodes: { newlyDegraded, recovered } }], appeared, disappeared, any } with direction from the rank tables intact < partial < broken and healthy < degraded < unobserved < broken (worse when nothing improved and something got worse, better the mirror, changed otherwise or when either side is undeclared / unknown; null when either record carries no chains). The run record gains branches, chains, versions (mcp.versions.<product> → value, bare product keys only, null when none — an absence, never "unchanged"), transition (against the newest record read before this run is written; null on the first run or after a record without chains, e.g. a vantage loss) and livePack: { kept, path, bytes?, reason }. The definition key keepLivePack: transitions | always | never (validated by loadJourneyDef; default transitions) decides whether Pack B's canonical JSON is written to runs/<journey>/live/<record stem>.json beside the record: transitions keeps the first run, a run whose chains moved, a gate failure, the run after a vantage loss and the run after a record that cannot be compared; a file-sourced Pack B is never snapshotted (the file is the snapshot), whatever the policy. The pure livePackDecision is exported; a snapshot write failure lands on the record as historyError, never thrown. Retention is unchanged (JOURNEY_RUN_FILE_RE untouched, readJourneyRuns keeps ignoring live/) and now also removes orphaned snapshots — after the record prune, pruneLiveSnapshots(recordFiles, liveFiles) (pure) names the run-shaped live/ files whose record is gone, never a survivor's, never a non-run file; readLivePack(name, record) reads a snapshot back (null when none was kept, the path is not the live/<stem>.json shape, or the file is gone — a pruned snapshot reads as absent, never as an error). renderJourneyMarkdown adds a Requirement chains table (chain | verdict | ladder | integrity | ladder integrity | worst node, cap 24, only when the record carries chains — no table means none were declared or recorded, never that every chain is intact; the worst-node cell carries the ladder detail and blinds N SLOs) and a Transitions since previous run section (title — intact/healthy → broken/broken (worse); newly degraded: …; recovered: …, appeared / disappeared, then the live pack: kept (…) — reason / not kept — reason line).
  • Candidate-cause ranker. chain-history.mjs gains rankCauses({ previous, current, deploys }) → { transitions, causes: [{ rank, kind, score, evidence, chains, nodes }], vantage: { changed, from, to, detail } | null, note } with note = candidate causes ranked by evidence — not a root-cause verdict. Only chains whose transition reads worse (plus chains that appeared already partial / broken / degraded) are considered, and only their nodes that moved (new on the current side, or the same identity with a different status / ladder status; every degraded node when the record cannot say); nodes whose ladder status is unobserved are dropped last, so a chain whose only movement is the vantage looking away yields no cause. Four kinds, scored by fixed CAUSE_SCORES so a rank is explainable by reading the table: observogram-deploy — a deploy inside the window whose item artifact names a moved node 0.9 (an artefact id on the node — aId / bId — matches exactly, otherwise the artefact must be contained in the node's label or key case-insensitively with a dash: prefix dropped; only that direction, so a rule deploy cannot blame every SLI whose short name it embeds; stubs under three characters and the wildcard all name nothing; the persisted degraded-node record carries no ids today, so on a real run the match is by label / key), else a deploy that wrote the journey's pack (by pack.id / pack.name against the record's packA) 0.6; config-drift — a moved node that drifted on a decision-bearing field 0.8, on cosmetic or unrecorded fields 0.4; backend-version — a product both records reported with a different version, 0.6 when a moved node is a backend / metric / recording rule / burn-rate alert, 0.3 otherwise; stack-self-metric — a current-record stack row that answered data and reads as a signal (nonzero hint, lower above 0, or a higher ratio below 1) in the family feeding a moved node's kind per FAMILY_FOR_KIND (scrape → scrape_job; ruler → recording_rule / burn_rate / sli / slo; notify → alert_route; tsdb → backend / storage_metrics; collector → pipeline_* / otel; dashboards → panel / dashboard; synthetic → synthetic) 0.5. One cause per distinct evidence, aggregating every chain and node it explains; ordering score desc → kind order → evidence; a verify-type record or a dry run is skipped; malformed input never throws. deploysInWindow(deploys, since, until) windows Observogram's own audit lines to (previous.startedAt, current.startedAt] on at and merges the latest verify onto its deploy so the evidence can name the outcome (deploy dep_x by carlos at … (upsert) touched <artifact>; verify: pending); journey.mjs reads deploys.jsonl beside runs/ under the same workspace root without importing server code (missing file → nothing, a torn line skipped). vantage reports what changed about the vantage itself — the vantage word, a probe family that newly failed (with its error), is no longer exposed, no longer probed, answers again or is now exposed, and the exposed tool count — beside the causes, never among them. The record carries causes (null on the first run: nothing to explain yet); topCause(record) reads the rank-1 cause. Markdown: ### Candidate causes — ranked by evidence, not a root-cause verdict (_no previous run_ / _no chain got worse since …_, N. [kind] score — evidence (chains: titles), then vantage changed: … / vantage: unchanged).
  • Surfaces. packc journey list appends chainStatusLine(record) after the stack segment — chains 8/10 intact · ladder 7 healthy · 2 degraded (zero ladder buckets other than healthy omitted; chains none for a record without declared chains) — and, only when a chain got worse or a cause was ranked (transitionGotWorse), causeLine(record) (top cause: [kind] evidence / no candidate causes); a vantage-lost line carries neither. GET /api/journeys lastRun gains chains (chainSummary(lastRun), null without chains), transition: { any, changed, worse } | null (counts, not the full diff), topCause (the rank-1 cause object or null) and vantageChanged (true / false / null without a causes block); GET /api/journeys/:name/runs hands the record through unchanged. The Journeys view prints one plain-text chains line per card — requirement chains: N/M intact · ladder: h healthy · d degraded · b broken · u unobserved · top exposure: <label> (<kind>) blinds N SLOs with a muted changed since previous run (W worse) marker — and one cause line — candidate cause: [kind] evidence — not a verdict with a muted vantage changed marker; counts and markers, no colours, no new CSS. In the Diagnose chain cards (compare-view.mjs) evidenceFor names present but unhealthy, present but stale and unobserved — <detail> nodes (the ladder reading replaces missing live / aligned for that node) and appends · blinds N SLO(s) to declared_only / drifted nodes with blastRadius.slos > 0; the card meta shows ladder: <ladderVerdict> and the section rollup appends · ladder NN%.
  • Suites. New tools/test-blast-radius.mjs (npm run test:blast-radius, 58 assertions — the vendoring guard, the tables, hop distances along the full chain, the protection relation, scaffold exclusion, cycle safety, determinism, the listing cap, the accepted input shapes, no input mutation) and tools/test-chain-history.mjs (npm run test:chain-history, 122 assertions — the guard, the tables and caps, which nodes are recorded and in what order, the truncated markers, the summary and its top exposure, transition directions, appeared / disappeared and node lists, the deploy window and verify merge, every cause kind and score, the one-way artefact match, the moved-node rule, the vantage block including a vantage-lost previous record and file-sourced sides, determinism, malformed input); both in the aggregate (34 suites). tools/test-traceability-graph.mjs adds the graphShape projection, the blastRadius pins per node kind, the ladder block (healthy / unhealthy / stale rules, an unhealthy burn rule linked by name, a down target, unsupported and failed probe families, absent for real, withheld from Pack B) and the byte-identity pin on integrity / verdict / counts / node status / rollup.integrityMean with and without the annotations. tools/test-journey.mjs drives a fake MCP through the snapshot policy (every branch of livePackDecision, the files under live/, pruning, readLivePack), the record shape and its round-trip through the history file, the markdown sections, the CLI lines and the ranker against a seeded deploys.jsonl (a deploy inside the window, one outside, a torn line, the vantage block). server/test-smoke.mjs seeds a record with branches and causes and pins the lastRun shape.
  • Review corrections (step 4). Two fix passes after the step-4 reviews; nothing scored changed on either side (integrity, verdict, counts, node status, rollup.integrityMean, the grade — byte-identical, still pinned). Graph side. The fetcher stamps its own clock: mcp.fetchStartedAt (the instant the fetch began) and mcp.observedAt.<family> (the instant each probe family answered, data or empty; dashboards re-stamped after detail enrichment), and the ladder's "now" is the family's observedAt, else fetchStartedAt, else the caller's mcp.refreshedAt (older packs — named in the detail only on that fallback), so a refreshedAt trailing a fresh observation by a minute no longer reads stale. A scrape job is present_unhealthy only when EVERY target is down (the fetcher's scrape_jobs_down rule); some targets down reads alive with n/m targets down: <instances>, and a down target's lastError is not the job's. Fractional burn factors (_burn_14_4x_ ↔ 14.4) parse in the graph and the fetcher; alert observation entries carry interval (the group evaluation interval — the alert adapter keeps it now, and burn-rate freshness uses the longest linked one) and labels: { slo, burn_rate, window_short, window_long } (present keys only, omitted when none), and the ladder links a live rule to its declared window by labels first, name second; the alert_rules_unhealthy list is narrowed to the declared windows like the rule set and the detail names the matched / failing rule(s). An unknown interval reads healthy / alive only within a 1 h ceiling ("interval unknown; 1 h ceiling applied"), older reads present_stale; a timestamp more than 1 s past "now" reads alive with "timestamp s in the future (clock skew); freshness not judged", never healthy; a reported health of unknown stays present_unhealthy; ladderVerdict ignores live_only nodes as the scored verdict does; "no fetch timestamp on the wire (mcp.refreshedAt missing); staleness not judged" when the observation has a timestamp and the pack has no clock. compare-view's evidenceFor orders the evidence line before the 5-cap — declared_only load-bearing, present_unhealthy, present_stale, drifted, declared_only (other), unverifiable, unobserved, live_only — and a declared_only node whose live twin sits under another chain reads "on the wire and in Pack B under another chain: …" (or "in Pack B under another chain; not reachable from this branch"); "withheld from Pack B" only when no twin exists. The golden alert_rules.json gains the two interval lines; the largest example pair's payload grows +88 % (spec §5c). Journey side. Deploy attribution is exact. runJourney resolves every windowed deploy item's artifact selector against Pack A (resolveDeployArtifact, exported: declared:<i> → the i-th declared recording rule's name, slo:<id> → the SLO id, its SLI base and the SLI the pack binds it to, dash:<id> → the id, a bare name → itself, all → nothing; the item carries resolved beside artifact) and the ranker matches a resolved name exactly — against the node's aId / bId (now persisted on every degraded node), its label, or an identity handle of its kind::{json} key (id, name, record, slo, job, uid) — never by substring, and only when the item's group can write the node's kind (DEPLOY_GROUP_KINDS: dashboards / restore / delete → dashboard, panel; rules → recording_rule, burn_rate, sli, slo; alerts → burn_rate, alert_route; alertmanager → alert_route; pipelines → pipeline_*, otel; an unknown group constrains nothing); the < 3 stub rule is gone; the evidence reads touched declared:0 → payment:api_availability:ratio_5m and, for a wildcard-only deploy of the journey's pack, wrote pack X (all rules) — no item names a moved artefact (0.6 — needs the audit line's pack.id, the registry id = the pack file's stem, or pack.name to equal the record's packA.name). Without a caller's resolution the ranker knows the pack-free part (deployArtifactNames). Versions. A flip to or from the literal live (the fetcher's word for "answered without a number") is a vantage matter — reported in vantage.detail as mcp.versions.<product> changed to/from live, never a cause — and backend-version scores 0.6 (versionFeedsMovedKind) only when the changed product feeds a moved node's family (PRODUCT_FAMILIES: prometheus / victoriametrics / thanos / mimir → scrape, ruler, tsdb; grafana → dashboards; alertmanager → notify; otel, otelcol → collector; loki, promtail → logs; jaeger, tempo → traces; unknown → none), else 0.3 — a Grafana-only change is no longer 0.6 on a Prometheus rule. Baseline. The chain diff, the versions and the deploy window are compared against the newest record that CARRIES chains (BASELINE_SCAN_LIMIT 25 back), the vantage against the immediately previous record (rankCauses gains baseline); transition is never null on a new record: { reason, since: null, changed: [], appeared: [], disappeared: [], any: false, skipped } with reason first run | previous run lost its vantage | previous runs carry no chain record when nothing could be compared, else the diff with reason: null and skipped: [{ startedAt, outcome }] naming the records passed over (cap 8); the report says "previous run lost its vantage — comparing against " in both sections and never "no previous run" when one exists. No broadening. A node whose status and ladder status are identical on both sides is never blamed; when nothing recorded moved, the changed transition entry carries note — declared side: missingRoles none → action, degraded list truncated (cap 16) — a node that moved may be unrecorded — and the chain yields no cause. Escaping. Every value the transitions, causes and breaches sections interpolate goes through mdCell (line breaks collapsed, | escaped, a line-leading # / - / * / + / > / 1. neutralised) and the ranker's evidence collapses whitespace; a label carrying \n\n### Gate breaches cannot forge a section. Also: readDeployLog (exported) reads only the trailing 8 MB of deploys.jsonl (DEPLOY_LOG_TAIL_BYTES, the cut first line dropped); readJourneyRuns reads only run-shaped files (a stray notes.json is never the previous run); an unparseable baseline start gives an EMPTY deploy window, never all of history; a snapshot write failure reads livePack: { kept: false, reason: 'snapshot write failed: …' } beside historyError; packc journey list appends vantage changed: <detail> (vantageLine) whenever the vantage block says so, with or without a top cause; pruneLiveSnapshots names only run-shaped live/ files OLDER than the oldest surviving record (retention victims), never a newer stem whose record a concurrent writer has not landed yet; stackSignalRows reads direction + value only, never the stored nonzero hint; the causes block no longer carries a second copy of the transition ({ causes, vantage, note }); chainStatusLine distinguishes chains none (pre-step-4 record) / chains none (vantage lost) / chains 0 declared; keepLivePack: never writes none from now on (earlier snapshots age out with their records); rankCauses accepts only type: 'deploy' (as deploysInWindow did). cause.chains carries the branch titles and rootKeys the identity keys; chainSummary.degradedNodes / topExposure count and rank the declared chains only, and undeclaredNodes counts the live-only nodes of undeclared chains — the Journeys view appends N live-only in undeclared chains only when > 0. Pinned: the server's real selector strings (declared:0, slo:checkout, all, a rollback ref under restore), dash:payment blaming neither scrape_job payment nor the rule embedding it, a three-letter dashboard id, wrong groups, branchRecordsFromGraph output satisfying the id match, the live flips, a Grafana-only change on a Prometheus rule, the stale hint, missing-role / truncation notes, baseline vs previous, the wording after a vantage-lost run (and with no earlier chains, and after a pre-step-4 record), notes.json, the 8 MB tail with a cut first line, the empty window, markdown forgery through labels / actor / evidence / breaches / vantage / live-pack reason, the CLI vantage segment, the smoke seed made consistent and the vantageChanged: null API case; the fake-MCP e2e deploy now carries declared:0 end to end.

Stack-health evidence per journey run (step 3)

Step 2 sampled the observability stack's own self-metrics on every live fetch as point-in-time signals; step 3 turns those samples into evidence over time by keeping them per journey run, so the run history becomes the time series the journeys surface reads. Nothing in this block creates a Verified stamp, touches the grade, or turns a sample into an SLO verdict.

  • Run-history retention (prerequisite). A journey on a cron cadence used to grow runs/<journey>/ without bound. writeRunRecord now prunes the directory to the newest OBSERVOGRAM_JOURNEY_RUN_RETENTION files after every write (default 1000; 0 = unlimited; a value that is not a non-negative integer falls back to the default; legacy TOMOGRAPH_* spelling honoured via brandEnv). The policy is the pure, exported pruneRunFiles(files, keep) (names to delete, oldest first — filenames are ISO start times so lexical order is chronological; non-.json entries are never candidates) plus parseRunRetention / journeyRunRetention() (read at write time). A deletion failure lands on the record as historyError, never thrown; readJourneyRuns is unchanged. Scheduling stays external by decision (VALUE_BACKLOG item 11).

  • stackEvidence on the run record. liveEvidenceFacts(canonicalB) parses mcp.observed.stack_metrics, mcp.observed.alertmanager and mcp.observed.grafana.* into stackEvidence = { status, reason, rows: [{ id, family, product, value, unit, direction, outcome, hint, at, referenceSli, reason? }], alertmanager: { version, clusterStatus, silencesActive, error } | null, grafana: { datasources, unhealthyDatasources, contactPoints, error } | null } — null when Pack B carries no mcp.stack.status (a file-sourced B is honest absence), rows enriched with the contracts table's referenceSli and display-only hint (a row the table no longer declares keeps referenceSli: null), expr dropped, cap 64, malformed JSON degrading to no rows / null surfaces rather than a fabricated panel. The step-2 stack counts block is unchanged for compatibility. Suite: tools/test-journey.mjs (retention policy + effect, env parsing, evidence from a fixture-built annotated Pack B, null for file-sourced B, retired row ids, malformed JSON, advertised-but-failing surfaces, round-trip through history).

  • Gate key stack — thresholds on the samples. gate.stack: { requireSampled?, rows?: { <rowId>: { min?, max? } } } turns the samples into an early warning for a business owner, never an SLO verdict: requireSampled breaches stack when stackEvidence is null (Pack B is not a live draft), the panel was not-attempted (with the reason), or no row answered data; rows.<id> breaches stack.<id> with no sample for <id> (<outcome>[: reason]) — threshold cannot be checked when the row is absent or did not answer data (a file-sourced B with a declared threshold breaches honestly instead of passing by absence), and with <id> = <value> <unit> outside [min … max] — point-in-time sample, not an SLO verdict when the number falls outside the band (equality passes; an info row may carry a threshold). loadJourneyDef validates the block through the exported validateGateStack(stack, name) — unknown row ids (case-sensitive against STACK_SELF_METRIC_PROBES) throw with the known ids listed, min / max must be finite numbers, an entry with neither (or min > max, or a non-boolean requireSampled) is refused. Pure formatStackValue(value, unit) renders 83.3% / 0.004/s / 0.0/h / 7.4s / 1. renderJourneyMarkdown adds a Stack self-metrics — point-in-time samples table (id | family | value unit | outcome | hint | reference SLI, cap 24, only when rows exist) and lists stack breaches with the others; packc journey list appends stackStatusLine(record) (stack sampled N / stack not attempted / stack none). The capture default gate and every existing key are unchanged; nothing touches the grade. Pinned in tools/test-journey.mjs (every branch of the gate, the validation errors, the formatter, the table and cap, the CLI line).

  • History helpers, listing summary and the journeys chips. New browser-safe tools/lib/stack-evidence.mjs (imports only the contracts table; vendorable, listed in docs/VENDORING.md): stackSeries(runs, rowId) (oldest → newest per row, gaps skipped never interpolated, non-data outcomes kept with value: null), latestByFamily(record) (the data row per family, ties by table order, retired rows last), stackSummary(record) ({ status, reason, sampled, families } or null without evidence), nonzeroRuns(series), stackPostureBudget(series, { objective, cadenceMs, windowMs, isBad? }) (the cadence heuristic — a sampled posture is only measurable when the window allows ≥ 10 bad samples; every note reads "signal, not verdict") and the shared formatStackValue / stackOutcomeLabel (journey.mjs now re-exports the formatter from here — one vocabulary for CLI, report and studio). GET /api/journeys adds lastRun.stack = stackSummary(lastRun) (null for a file-sourced B); the Journeys view loads the module from /lib/stack-evidence.mjs at call time and renders a stack self-metrics — point-in-time samples line per card — one chip per family, the nonzero hint as a muted marker, id + reference SLI in the title, nonzero in N of last M runs for lower-is-comfortable rows over the fetched history, one muted chip with the reason when not attempted; no chip carries an ok/error colour. New suite tools/test-stack-evidence.mjs (npm run test:stack-evidence, in the aggregate); server/test-smoke.mjs seeds a record with stackEvidence and checks the listing shape.

  • Docs reconciled with the branch. README's journeys section carries the gate example with the stack: block, the retention env var and the studio chips, and its API table / repository map now list the journey routes and modules (tools/lib/journey.mjs, tools/lib/stack-evidence.mjs, studio/journeys-view.mjs); docs/MCP_INTEGRATION.md's journey section documents the record fields, the gate keys with their breach table and honesty rules, retention, and the history helpers with the cadence heuristic; docs/VENDORING.md lists stack-evidence.mjs + the contracts table as a vendorable pair; docs/VALUE_BACKLOG.md item 11 gains a status note (drift over time now includes the stack samples; scheduling stays external).

  • Review corrections (step 3). Reproduced against a seeded workspace and the fake MCP: latestByFamily ranked data rows by contracts-table order alone, so whenever a family's first table row is higher / info (scrape_success_ratio, tsdb_active_series) and answered data, the family's lower-is-better rows — and with them the nonzero marker and the nonzero in N of last M runs history the chips exist for — never reached the surface: the card read scrape 95.0% while the run breached on scrape_targets_down. Data rows now rank by nonzero hint first, then declared-in-table before retired, then lower before higher / info, then table order (tools/test-stack-evidence.mjs pins each rung; the old "ties by table order" assertion pinned the hiding). Also from the review: a rows.<id> threshold on a row the fetcher never wrote names why — (not-attempted: <tier reason>) on a not-attempted panel, (not attempted by the sampler — call budget exhausted or row not observed) on a sampled one — instead of a bare "absent from this run" that read like a fetch hole; evaluateStackGate no longer trusts an unvalidated gate object (a bound that is not a finite number, an entry with neither bound, min > max or a non-mapping entry breaches stack.<id> as threshold invalid (…) — cannot be checked, never passes by silence); a breach whose display rounding equals its bound appends the raw number (0.000/s (raw 0.0004)); an outcome the contracts do not declare is kept verbatim (missing → unknown) rather than relabelled failed; POST /api/journeys/capture runs validateGateStack before saving (400) and GET /api/journeys / packc journey list / the studio card carry loadError for a definition that cannot load; pruneRunFiles only treats the run-record filename shape (JOURNEY_RUN_FILE_RE) as a run, so a hand-dropped notes.json neither displaces a record nor is deleted, and the deletion-failure branch (historyError, never thrown) is now under test. README and docs/MCP_INTEGRATION.md reworded accordingly (requireSampled's third branch, "the only gate reader", the raw-number fallback of the chips without the helper module).

Stack self-metrics (step 2)

Observogram monitors the artefacts that monitor the system; step 2 of the roadmap starts acquiring the observability stack's own health signals through the MCP so later steps can build stack-health evidence and early warning. Three things are true of everything in this block, by contract and pinned by the suites: every sampled number is a point-in-time signal, not a verdict — no Verified stamp, no Scaffold marker, no SLO verdict and no grade criterion or grade input changed anywhere; a restricted MCP tier reads not-attempted with the reason, never "absent"; and the alias table reads only names a real product exposed — it was documentation-grounded when written (every metric name following the product's upstream docs, named on the row) and now carries three live evidence sources: the pinned Docker stack (npm run test:stack:live, every alias stamped with the image it was verified on) and the public (2026-09-07) and authenticated (2026-09-08) Krystaline tiers (npm run record-fixtures); a product a given stack does not run reads an honest not-in-inventory.

  • Alias table (contracts). New data-only tools/lib/contracts/stack-self-metrics.mjs: STACK_SELF_METRIC_PROBES — 24 rows over nine families (scrape, ruler, notify, tsdb, collector, dashboards, synthetic, logs, traces), each with its signal, unit, display-only direction, the reference-pack SLI whose vocabulary it follows, ordered product aliases ({ product, expr, requires }) and the upstream docs the names come from — plus lookup resolvers (probeRows, rowsForFamily, eligibleAliases, productPreferenceOrder, displayHint, bestOutcome) and the allowed sets (STACK_FAMILIES, STACK_UNITS, STACK_DIRECTIONS, STACK_OUTCOMES). One known discrepancy is documented on the row rather than fixed: the reference pack's scrape_duration_p99 names scrape_duration_seconds_bucket, which Prometheus never exposes (the metric is a per-target gauge), so scrape_duration_max samples max(scrape_duration_seconds).
  • Registry rows + shapes. mcp-capabilities.mjs gains stack_self_metrics (carrier for the table, rides metrics_query), alertmanager_status, alertmanager_silences, grafana_datasources, grafana_datasource_health (enrich) and grafana_contact_points; build_info_versions now declares its instant-vector shape. response-shapes.mjs adds instant-vector ({ result } and { data: { result } }), silences, datasources, contact-points, and a new object shape kind (object: true, objectAt, anyOfKeys) for status-object and health-object — validateResponseShape locates the first candidate object carrying every key group, tolerant of extras and of a { data: {…} } envelope. Five tool names join the pinned wire surface.
  • Sampler (fetcher). tools/fetch-live-pack.mjs exports sampleStackSelfMetrics({ callTool, quiet, metricsQueryTool, inventory, seenProducts, discoveredToolNames, hasToolsList, refreshedAt, maxCalls }) → { status, reason, rows, callsMade }, run by fetchMcp after the version probes. Policy: attempt only when metrics_query is advertised (a server with no tools/list is attempted), otherwise not-attempted with the reason metrics_query not exposed by this MCP (restricted tier) and zero calls; with a metric_names inventory an alias is eligible only when every requires name is present (rows with none are not-in-inventory, no call); eligible aliases ordered generic-first then products already seen; at most 2 calls per row, 48 per panel (call budget exhausted beyond it); every call through quiet() under the family stack_self_metrics; instant vectors parsed from { result } or { data: { result } } by sampleFromInstantVector (exported — the recorder reads answers the same way), NaN/±Inf/missing samples are empty. Outcomes are exactly data | empty | failed | not-in-inventory | not-attempted — never "ok".
  • Alertmanager / Grafana status rows. observeAlertmanager (alertmanager_status → version / uptime / clusterStatus; alertmanager_silences → active + total) and observeGrafana (grafana_datasources → [{uid, name, type}], grafana_datasource_health per uid ≤ 10 → ok | error | unknown + a message trimmed to 200 chars, grafana_contact_points → count + ≤ 32 names), each tool guarded by tools/list when present, tolerant parsing through the response shapes. fetchMcp returns stackSamples, alertmanagerObserved, grafanaObserved; answered tools leave unmatchedTools.
  • Annotations. buildCanonicalPack (new params stackSamples, alertmanagerObserved, grafanaObserved, all null-safe — a caller that predates step 2 writes nothing) emits mcp.stack.status, mcp.stack.reason (only when not attempted), mcp.stack.sampled / empty / failed / notInInventory / notAttempted, mcp.stack.families (<family>:<best outcome>), mcp.observed.stack_metrics (attempted + not-in-inventory rows, cap 64), mcp.observed.alertmanager, mcp.observed.grafana.datasources, mcp.observed.grafana.contact_points, and lists the answering tools in mcp.toolsCalled. No mcp.verified.* or mcp.scaffold.* key is touched by any of it (pinned by the suite).
  • Surfaces. POST /api/draft-from-mcp answers summary.stack (status, reason, the five counts as numbers, families: { family: best outcome }, rows: [{ id, family, product, value, unit, direction, outcome, hint, reason? }] — hint is the contracts' display-only displayHint, computed on read and never stored), summary.alertmanager and summary.grafana (each null when not exposed), and warns Stack self-metrics not attempted — metrics_query not exposed by this MCP tier. on a restricted tier; GET /api/live-status adds stackStatus and stackSampled. The studio draft summary gains a "stack self-metrics — point-in-time sample, signal not verdict" block: one row per family with the best row's formatted value (or — empty / — probe failed / — not in inventory), a single — not attempted row on a restricted tier, then the Alertmanager version + active silences, the Grafana datasource count with the unhealthy names, and the contact-point count. liveEvidenceFacts adds stack { status, reason, sampled, empty, failed, notAttempted }, carried on the journey run record and printed as one Stack self-metrics line in the markdown report — no gate key reads it.
  • Fixtures: synthetic now, recorded later. tools/fixtures/mcp/synthetic/ holds hand-written samples for each new shape (an otel-mcp-server and a Prometheus-API instant vector, Alertmanager status and silences, Grafana datasources, datasource health, contact points), each carrying a _synthetic marker the shape check tolerates and tools/test-contract-shapes.mjs asserts; they pin shape only. The fixtures README now carries a provenance table (recorded 2026-06-12 from the Krystaline otel-mcp-server vs synthetic 2026-09-07; the recorded metric-name list is trimmed to 25 names and proves none of the alias table's requires), the recorder procedure, and what to re-run afterwards. A recorded <tool>.json at the fixtures top level takes precedence over its synthetic copy in the shapes suite (same assertions, plus "no _synthetic marker"), and every file in recorded-stack/ must be a valid instant vector for a declared row.
  • Recorder. tools/record-mcp-fixtures.mjs (npm run record-fixtures, MCP_URL + optional MCP_AUTH) reuses the fetcher's JSON-RPC client and the contract registry for every tool name (the guard suite scans it for literals), is read-only against the MCP, and never prints or stores the token (every string that leaves the process is redacted). Default mode reports: the tools/list surface with its drift against allKnownToolNames(), the metric inventory, and for every alias of every row whether its requires are all present plus the value read with sampleFromInstantVector (data / empty / FAILED / not-in-inventory (missing …) / not-attempted (restricted tier)), then the status tools and probe families with their shape verdicts. -- --write records: the full inventory to the git-ignored .tmp-mcp-metric-names.json, the trimmed inventory (every required name and the *_build_info metrics kept, plus the first 25 others, in the server's order, replaced where the adapter found the list), scrape targets / the winning rule tool(s) / dashboard search with every list capped at 6 and structure verbatim, one metrics_query instant vector per family under recorded-stack/<row id>.json with _recorded provenance, and the status tools when advertised; --out <dir> redirects everything. New scripts lint:recorder, test:recorder.
  • First live recording (2026-09-07, public Krystaline tier, https://www.krystaline.io/mcp/public, no credentials). The recorder's first real run replaced the guesswork for part of the table: that tier advertises 43 tools including metrics_query, metrics_label_values (2,682 names), alertmanager_status and alertmanager_silences, so metrics_label_values.json is now a 40-name recording that evidences up, scrape_duration_seconds, vm_*, alertmanager_*, otelcol_exporter_queue_*, grafana_http_request_duration_seconds_count and promtail_dropped_entries_total; alertmanager_status.json is recorded (its config field, the full Alertmanager configuration, omitted by hand); and recorded-stack/ holds six real metrics_query instant vectors (otel-mcp-server answers a flat { status, resultType, result } envelope). Ten aliases read data, one empty, twenty-six not-in-inventory — the stack runs VictoriaMetrics, not Prometheus, and does not scrape vmalert (its ruler's own health is invisible there: exactly the blind spot the sampler exists to name). Every Grafana-backed tool except grafana_health answers HTTP 401 on that tier (no Grafana credentials), so the Grafana status fixtures stay synthetic until the authenticated tier is recorded; the silences recording was empty and pins nothing, so its synthetic copy stays too. The 2026-06-12 metrics_targets.json / vmalert_rules.json / grafana_dashboards_search.json recordings were kept on purpose (today's targets are all up; the down Alertmanager target is the liveness case the adapters are pinned against). Contract fix found by the run: grafana_dashboard_get was listed as a search candidate of the dashboards probe, so a failed search fell through to a uid-less call whose -32602 Invalid arguments error masked the real search error; it is now only the dashboard_detail enrichment tool, and the family reports the 401 it actually got. Suites replay a recorded status fixture in preference to its synthetic copy and read expected values from the fixture they replayed, and the recorder suite's fake inventory is frozen to the 2026-06-12 names so re-recording never moves its expectations.
  • Authenticated-tier recording (2026-09-08, https://www.krystaline.io/mcp, MCP_AUTH bearer). The recorder's first run against the authenticated Krystaline tier re-recorded vmalert_rules.json (the 2026-06-12 recording predated most of today's rule groups; adapted goldens re-pinned; its annotations carry rule descriptions and runbook URLs into the private krystalinex-core repo — references, not credentials, and the identical payload the credential-less public tier serves), alertmanager_status.json (_recorded.scrubbed names the hand-removed config — receivers, SMTP settings, internal hostnames — same trim as the 2026-09-07 file it replaces) and the six recorded-stack/ instant vectors (same rows and outcomes as the public tier: 14 aliases data · 0 failed; the trimmed metrics_label_values.json recording came back byte-identical to the committed public-tier file, and both tiers report a 2,682-name inventory — the full lists were not compared). The intended goal — replacing the synthetic Grafana fixtures — was not reachable: the tier advertises no Grafana-backed tools with any issued key (its otel-mcp-server deployment carries no Grafana integration; the public tier advertises them but answers HTTP 401), so synthetic/grafana_*.json and the 2026-06-12 grafana_dashboards_search.json stay, metrics_targets.json keeps its 2026-06-12 down-target recording (the 2026-09-08 payload is all-up), and the empty silences recording again pins nothing. The tier also advertises logs and k8s tools the registry has no rows for (noted in the fixtures README; no registry rows added).
  • Local-stack recording of the Grafana-backed tools (2026-09-08, docker/stack.compose.yaml fronted by otel-mcp-server 1.8.0). The fourth evidence source, and today the only place grafana_datasources, grafana_datasource_health and grafana_contact_points answer: the pinned stack (Grafana 12.4.4, admin-seeded with a VictoriaMetrics (stack) datasource, a Loki (absent) datasource pointing at a service the stack does not run, the webhook-oncall contact point and one dashboard) behind a local otel-mcp-server started with GRAFANA_URL + GRAFANA_AUTH_BASIC and no MCP auth keys, recorded through MCP_URL=http://127.0.0.1:3011/mcp npm run record-fixtures -- --write --out .tmp-recorded-local. The three synthetic/grafana_*.json files are gone, replaced by recordings whose _recorded carries scrubbed: [] with what was reviewed (jsonData / secureJsonFields empty; compose hostnames and the loopback Grafana URL inside a failed check's error kept verbatim) and the server line, added by hand. Two shapes the hand-written files had wrong: grafana_datasource_health answers { datasource, health } — health is { supported: true, status: 'OK' | 'ERROR', message, details } when Grafana's endpoint answered 2xx and { supported: false, error: 'HTTP 400: Bad Request — <url>' } when it did not (Grafana 12.4.4 answers a check that ran and failed, backend unreachable, with HTTP 400; an unknown uid with 500) — so the health-object contract now locates ['health', 'data', ''] and accepts a verdict key or the supported / error pair (one key group: the checker requires every group and the two forms share no key), and observeGrafana reads OK → ok, ERROR → error, supported: false + HTTP 400 → error with the error text as message, any other supported: false → unknown (not checked, never "not unhealthy"), supported: true without a status → unknown; and grafana_contact_points answers the receivers API ({ count, contactPoints: [{ name, active, integrations: [{ name, sendResolved, lastNotifyAttempt, lastNotifyAttemptDuration }] }] } — no uid, type or settings, so no webhook URL or secret ever rides in it), not the provisioning shape. The recorder keeps both datasource-health verdicts when several uids answer (the first uid's answer as grafana_datasource_health.json; the first answer of the other case as .ok.json / .error.json with its uid and case, skipped when identical to the primary file). Suites replay the recording with expected values read from the files: test-contract-shapes pins both health cases and the envelope location, test-fetch-live drives observeGrafana with the recorded uids and asserts the unhealthy datasources name exactly Loki (absent) (plus the non-400 and no-status unknown cases), the smoke and end-to-end fakes answer health per uid, and the recorder suite's host guard is the fake's own host:port. grafana_dashboards_search.json keeps its richer 2026-06-12 recording (the stack answers the same { count, results } shape).
  • Live validation tier (2026-09-07, the real products in Docker). The public MCP tier could only evidence what that stack happens to scrape; docker/stack.compose.yaml (name: observogram-stack, loopback-only ports 19090 / 19093 / 18428 / 18880 / 18888+14318 / 13030 / 19115 / 19080 / 14269, disjoint from the validate stack) now runs every product the table names at an exactly pinned version — Prometheus v2.55.1 scraping all of them plus a blackbox probe job, Alertmanager v0.27.0 with a dead webhook receiver, VictoriaMetrics v1.113.0 scraping itself and a dead target, vmalert v1.113.0, otel-collector-contrib 0.115.1 with a debug exporter beside an OTLP exporter to a dead endpoint (queue + bounded retries), Grafana 12.4.4 with a provisioned Prometheus datasource and one provisioned always-firing alert rule, blackbox-exporter v0.25.0, promtail 3.3.2 tailing a sample file into a dead Loki, and Jaeger all-in-one 1.62.0 — every ruler carrying one recording rule and one always-firing alert so the failure and notification counters move on a fresh stack (up -d --wait returns in ~10s). tools/test-stack-live.mjs (npm run test:stack:live[:strict], NOT in npm test — it needs Docker and skips loudly without it) brings the stack up, recreates the collector and Grafana so neither carries a previous run's stimulus, waits for every scrape job to be up and for the rate windows to hold two samples, then for every alias of every row: (a) asserts each requires name is a metric family on the product's own exposition (/metrics; the Prometheus TSDB name inventory for the scrape-synthesised up / scrape_duration_seconds, which never appear on Prometheus' own /metrics; the blackbox /probe output for probe_success; the vmalert service for vmalert_*), recording separately the presence of any name the expr reads that requires does not demand (the lazily-registered-counter policy's shape); (b) evaluates the expr on the real Prometheus and reads the answer through the fetcher's own sampleFromInstantVector (data / empty; a PromQL error is a FAIL naming the error); (c) records the collector's name set at startup, sends one OTLP/HTTP request per signal kind (and one query through Grafana's datasource proxy), and records the name set again. The ledger (alias | product@version from the image tag | exposition | query) is printed and written to the git-ignored .tmp-stack-live-ledger.json; any exposition ✗ or query ERROR exits non-zero. First ledger, table untouched by design (the next slice corrects it): 28 of 37 aliases expose every required name and every one of the 37 expressions evaluates without a PromQL error (25 data, 12 empty); the 9 ✗ are all otelcol — on 0.115.1 the internal telemetry carries no _total suffix (the bare-name aliases pass), otelcol_processor_dropped_* never exist (removed by the processorhelper rework), and otelcol_exporter_send_failed_* (like otelcol_exporter_sent_* and every receiver/processor counter) are absent at startup and present only after the first export attempt — 10 names at startup, 39 the moment send_failed_* register, 48 by the time that first ledger's collector had been retrying its dead exporter for some minutes (the counts are timing snapshots of one growing set; the 10 is stable) — while otelcol_exporter_queue_size / queue_capacity / otelcol_process_uptime are the always-present siblings. Also observed: otelcol_exporter_queue_size carries a data_type label that queue_capacity lacks, so collector_queue_saturation's division matches nothing (empty on a healthy collector); Grafana's grafana_datasource_request_total and grafana_alerting_rule_evaluation_* register lazily (after the first proxied request / the first rule evaluation of an org); the two Grafana 5xx rows read empty rather than 0 on a healthy Grafana because their label filter matches nothing.
  • Live validation tier — the alias table corrected from live exposition (2026-09-07). With the stack ledger and the public tier as evidence, tools/lib/contracts/stack-self-metrics.mjs now reads only names a real product exposed: every one of its 32 aliases carries a verified stamp (<image:tag> <exposition|TSDB inventory|probe output> + PromQL, 2026-09-07) that npm run test:stack:live checks against the compose image, and the live suite is a green gate — 32/32 aliases expose every required name (lazily-registered counters asserted present after the stimulus), 32/32 expressions read data on the real Prometheus (the first run: 28/37 exposition, 12 empty). What changed and why: (1) the six otelcol _total aliases are gone — neither 0.115.1 (stack) nor 0.154.0 (public tier) suffixes its internal telemetry; (2) the three collector_dropped_* rows are retired for collector_refused_{metrics,spans,logs} over otelcol_receiver_refused_<kind> — otelcol_processor_dropped_* exist on no current collector in either spelling (removed by the processorhelper rework), while the receiver counters are the current-generation "collector is losing telemetry" signal (24 rows, 9 families unchanged); (3) the lazily-registered counter policy: the collector registers send_failed_<kind> with sent_<kind> at the first export (0.115.1, observed before/after a forced failure: 10 names at startup, ≥39 after one OTLP request per kind — 39 the moment send_failed_* register, more as the dead exporter retries; the never-failing debug exporter then carries send_failed_* 0) or, on 0.154.0, plausibly only on the first failure (send_failed_spans absent while sent_spans is present on the read-only public tier — consistent with registration on the first failure, not forced there), so those six aliases require the sibling that registers with or before the counter (otelcol_exporter_sent_<kind>, otelcol_receiver_accepted_<kind>) and end with or (count(<sibling>) * 0) — a scraped collector that has exported reads 0 unless the counter exists, one that never exported that signal reads not-in-inventory, and a renamed counter renames the sibling so the alias never reads a false 0 (a bare or vector(0) was not used: the sampler's bounded cascade queries aliases without a trusted inventory, where vector(0) would fabricate a healthy collector on a backend that has none — the reason the count rows already reject it); (4) collector_queue_saturation is max(max by (exporter, instance, job) (queue_size) / max by (exporter, instance, job) (queue_capacity)) — on 0.115.1 queue_size carries data_type and queue_capacity does not (the plain division matched nothing → empty), on 0.154.0 both do (ignoring(data_type) group_left would be many-to-many on a multi-pipeline collector); the per-exporter max is the fullest queue's fill on either; (5) datasource_errors reads the reference pack's grafana_datasource_request_total{code=~"5.."} first — registered on Grafana's first datasource request, which is any rule evaluation, /api/ds/query or proxied query, so every Grafana with a dashboard or a rule has it — and keeps the pre-registered grafana_proxy_response_status_total (code="500" at 0 from startup, but counting only the legacy /api/datasources/proxy/... path) as the fallback, documented as discrepancy 3 (this slice first shipped the proxy counter first; the review below swapped the order); (6) the two Grafana 5xx rate rows carry the presence-guarded zero (or (count(<base>) * 0)) so a healthy Grafana reads 0 instead of empty; (7) confirmed unchanged against the expositions: vmalert's plural errors_total, vm_cache_entries{type="storage/hour_metric_ids"}, the summary labels slice / quantile, alertmanager_*, promtail_dropped_entries_total, probe_success on the /probe output, and jaeger_collector_spans_dropped_total on Jaeger 1.62.0's admin port with no flag — every one pre-registered at 0, so strict requires stay (the public tier's Jaeger is v2.18.0, an otelcol distribution: it answers through the collector rows and the jaeger alias reads an honest not-in-inventory there). Public tier re-run after the correction (read-only): 14 aliases data · 0 empty · 0 failed · 18 not-in-inventory of 32 (was 10 · 1 · 0 · 26 of 37), 13 of 24 rows with data; the trimmed inventory fixture grew to 43 names (grafana_proxy_response_status_total, otelcol_exporter_sent_spans, otelcol_receiver_accepted_spans) and recorded-stack/ holds six instant vectors again, collector_export_failures_spans and datasource_errors replacing the queue and HTTP-errors files as their families' data rows. Suites: test-stack-self-metrics pins the verified shape on every alias, the sibling guard on exactly six lazy aliases, the absence of vector(0) / otelcol _total / processor_dropped_*, and the collector row ids; test-stack-live asserts lazy counters after the stimulus and the stamp-vs-image match, and strips grouping clauses before reading metric names out of an expr. docs/MCP_INTEGRATION.md gains a "Live validation tier" section with the verification ledger per product@version.
  • Live validation tier — review corrections (2026-09-07). Reproduced on the running stack (Grafana 12.4.4): an /api/ds/query — the path dashboards and Grafana-managed rules use — increments grafana_datasource_request_total and leaves every grafana_proxy_response_status_total series untouched (the proxy counter counts only the legacy /api/datasources/proxy/... path; a proxied call increments both), and the sampler stops at the first alias that answers data. With the proxy counter first, datasource_errors read a pre-registered 0 forever on any Grafana whose datasource queries were failing 5xx through /api/ds/query — a systematically blind 0 on the surfaces. The row now reads the reference grafana_datasource_request_total first (strict requires) and the proxy counter as the fallback for a Grafana whose inventory lacks the reference name (the public tier: its first alias reads not-in-inventory and the row falls through to the proxy alias, so recorded-stack/datasource_errors.json keeps the proxy expression as its provenance query); test-stack-self-metrics pins the new order and the reason. Also from the review: the live suite snapshots Grafana's /metrics the moment --force-recreate --wait grafana returns (the provisioned 10s rule is Grafana's own first datasource request, so its before column is "before the suite's stimulus", not "at startup" — the collector's is exact), every suite request carries a 10s AbortSignal.timeout, the compose header says one run at a time (two concurrent runs recreate the collector and Grafana under each other) and the Grafana provisioning mount is :ro like every other mount; the collector name counts are stated as the timing snapshots they are (10 at startup is stable; 39 the moment send_failed_* register, 48 minutes later), the 0.154.0 observation is worded as the read-only inference it is, the pre-registered-at-0 citation names the right libraries (client_golang for prometheus_* / alertmanager_* / promtail_* / jaeger_collector_*; VictoriaMetrics' own metrics library for vm_*, and vmalert's *_rules_errors_total per loaded rule — per-rule label sets, none on a rule-less vmalert), SRC.prometheus no longer reads as if prometheus_* were confirmed on the public tier (no Prometheus server there; only up / scrape_duration_seconds were), and otel/opentelemetry-collector-contrib:0.115.1 self-reporting service_version=0.115.0 is noted beside the stamp. The shipped lazy-counter policy deliberately deviates from the brief's wording (an always-present sibling + or vector(0)): it requires the per-kind sibling that registers with the counter and ends with or (count(<sibling>) * 0), because the sampler's bounded cascade queries aliases without a trusted inventory, where vector(0) would fabricate a healthy collector on a backend that has none — the canonical statement is the "Lazily-registered counters" paragraph of docs/MCP_INTEGRATION.md, the contracts header points at it.
  • Review follow-ups (honesty re-audit of the slice). Six places where the panel could read as assurance for something never sampled, plus the names the review checked against upstream source:
    • Alias table. vmalert's ruler counters are vmalert_recording_rules_errors_total / vmalert_alerting_rules_errors_total (plural — the singular names could never answer on the recorded VictoriaMetrics + vmalert reference stack). query_latency_p99 now reads max(prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.99"}): Prometheus registers that metric as a summary, so the _bucket the reference pack (and the old row) quantiled over never exists — documented as known discrepancy 2 beside the scrape-duration one. The count rows carry a presence-guarded zero, count(up == 0) or (count(up) * 0) (and sum(up) / count(up) for the ratio), so a healthy stack reads 0 instead of empty while a backend without the metric still reads empty — or vector(0) was rejected because it would fabricate "0 down" where nothing is scraped. SRC.otelcol now says the otelcol_processor_dropped_* counters exist only on collectors before the processorhelper metric rework.
    • Sampler. A series-only instant vector (metric identities, no sample) is empty for every unit — a count row used to turn it into "1 target down". The metric-name inventory is evidence of presence, never of absence: only an inventory that carries up gates a row to not-in-inventory (with the inventory size in the reason); an inventory without up — the recorded 25-name fixture, an empty list, a capped server — gates nothing and the rows are queried on the bounded cascade (queried anyway in the reason of an empty answer). With a trusted inventory every eligible alias may be tried (the 2-per-row cap applies to the fallback cascade only). hasToolsList is whether the tools/list RPC succeeded, so an empty advertised list reads not-attempted (tier), not 24 × failed. New export stackInventoryTrust(inventory); the sampler result carries inventory: { size, trusted, reason }.
    • Shapes + observers. Object shapes locate envelope-first (objectAt: ['data', ''], new locateObjectPayload(shapeId, response) the fetcher reuses): a Prometheus-API-style { status: 'success', data: {...} } wrapper used to satisfy status-object / health-object at its root, reading version null / cluster success and a wrapped { status: 'ERROR' } health verdict as ok. observeAlertmanager / observeGrafana return null only when none of their tools is advertised; an advertised tool that fails (HTTP error, timeout, bad shape) yields a non-null result with error, annotated as mcp.observed.alertmanager.error / mcp.observed.grafana.error, carried on summary.alertmanager.error / summary.grafana.error with a … status probe failed — <error> warning, and rendered — probe failed: <error> — — not exposed is reserved for tools absent from tools/list.
    • Studio. The datasources row never prints 0 unhealthy for a surface nothing checked: N · health not checked (grafana_datasource_health not exposed or did not answer) when every health is unknown, otherwise N · M error: <names> · K unchecked: <names> (summary.grafana.healthChecked counts the verdicts). The not-attempted row renders summary.stack.reason; a family with no observed row reads — not attempted: call budget exhausted instead of not attempted: not attempted.
    • Recorder. Committed fixtures' _recorded provenance carries tool + timestamp (+ query) only — the maintainer's MCP hostname stays in the git-ignored review copy; every query-string value of MCP_URL is redacted alongside MCP_AUTH (a URL-borne token echoed by an upstream error body comes out <redacted>); the report prints the fetcher's inventory-trust verdict and the inventory size beside the not-in-inventory count.
    • Docs + suites. MCP_INTEGRATION.md sampling policy rewritten for the trust rule (the old sentence claimed a trimmed fixture made every alias eligible while the code gated everything); the annotation reference lists mcp.observed.alertmanager / grafana.* / grafana.error; new cases in test-fetch-live.mjs (series-only count rows, trusted vs untrusted vs empty inventory, no per-row cap with an inventory, advertised-but-failing and envelope observers, error annotations), test-contract-shapes.mjs (envelope location), test-stack-self-metrics.mjs (pinned upstream names, guarded zeros, discrepancy 2), test-record-fixtures.mjs (URL-only token, no host in provenance, inventory-trust line).
  • Suites: tools/test-stack-self-metrics.mjs (npm run test:stack) — unique ids, allowed sets, every alias's requires ⊆ the metric names its expr reads, every referenceSli resolves in its reference pack, resolver purity; tools/test-fetch-live.mjs — sampler cases (inventory eligibility, product preference, data/empty/failed, NaN → empty, both envelopes, bounded cascade, global cap, restricted tier, no tools/list, quiet() family), observer cases, the annotation contract (schema-valid, no Verified/Scaffold change, adapted layers unchanged) and an end-to-end run against the fake JSON-RPC server; server/test-smoke.mjs (draft summary with and without metrics_query, live-status fields); tools/test-journey.mjs (stack facts, record, markdown); new tools/test-record-fixtures.mjs — the recorder against a fake MCP replaying the fixtures: report content and drift, no writes in report mode, every --write file's trimming and shape, the token absent from every output and file, the restricted tier, missing MCP_URL; tools/test-contract-shapes.mjs — recorded precedence and the recorded-stack/ scan; tools/test-contract-guard.mjs now scans the recorder too. Docs: MCP_INTEGRATION.md "Stack self-metrics (registry / sampling / surfaces)" with every mcp.stack.* / mcp.observed.* key listed once in the annotation reference; README "Fetch Live From MCP".

Live evidence honesty: the fetcher stops asserting what it never observed

The top product risk is false assurance — a live pack reporting Verified / aligned / fresh for an artefact that is not doing its job. This entry removes the known false-assurance paths in tools/fetch-live-pack.mjs, makes the on-wire liveness signals visible, lets journeys gate on the vantage, and stops two authoring-only differences from reading as drift. Existing users should expect alignment percentages and grades to drop against the same MCP — placeholders that used to count as Verified now park as Scaffold, burn-rate alerts the platform never had no longer align with the ones the repo declares, and scrape jobs whose targets are all down no longer attest scraping. The lower number is the honest one.

False-assurance fixes

  • Burn-rate alerts are mapped from discovered rules, never synthesised. The fetcher used to emit a two-window burn-rate alert for every inferred SLO and stamp mcp.verified.policy.burn_rate_alerts whenever any alert name came back. spec.policy.burn_rate_alerts is now built from the alerting rules the MCP exposes — compiler-labelled rules (slo/burn_rate/window_short/window_long/severity) and <slo>_burn_<N>x_<short>_<long>-named rules, grouped per SLO, deduped, short window first — and each entry gets an indexed mcp.verified.policy.burn_rate_alerts[<i>] stamp (the unindexed key is gone). A group whose SLO shares the SLI base of an inferred placeholder re-identifies that SLO to the discovered id and replaces its placeholder objective/window from the slo_objective/slo_window annotations; groups naming an SLO nobody inferred, or carrying a single window, land in mcp.discovered.alert_rules_unmapped; forecast rules are skipped; severities outside the SEVn enum are mapped from the Prometheus vocabulary or the burn factor and listed in mcp.discovered.alert_rules_severity_inferred. When nothing maps, the schema-forced placeholder is stamped mcp.scaffold.policy.burn_rate_alerts[0].
  • Every placeholder the fetcher invents is Scaffold, never Verified. mcp.scaffold.<symbol> (the live counterpart of crawler.scaffold.<symbol>; tools/lib/adapter.mjs sourceOf now honours both prefixes) marks the hard-coded spec.otel block (system_health answering used to verify it), collector receivers/processors, the logs/traces exporters, the fallback logs-elastic backend, the platform-overview dashboard stub, the SEV1 → Teams route, the baselines, and the per-service / platform_availability SLI+SLO guesses. pipelines.exporters.metrics is Verified only when scrape targets or a metric inventory evidenced it; metrics-prom only on a *_build_info capture; traces-jaeger only when the topology names jaeger or traces_services answered. SLIs inferred from real recorded rules stay Verified; an SLO re-identified by a discovered burn-rate group becomes Verified.
  • MTTD is no longer derived from anomaly thresholds. spec.baselines.mttd_target_p50 was the smallest anomalies_baselines thresholdMs (e.g. 90ms) stamped Verified — a latency-anomaly threshold is not a time-to-detect target. Baselines are now the platform default for the declared criticality (measurement_source: platform-default, stamped Scaffold) until a §5.10 measurement exists; mcp.baselinesComputed still counts the anomaly baselines returned, which is evidence the tool answered, not a measurement.

Liveness evidence (on-wire health)

  • A scrape job NAME used to be enough to stamp mcp.verified.telemetry.scrape and mcp.verified.pipelines.exporters.metrics, and a rule's presence was enough to verify it, even when the same payload said every target was down or the rule was failing to evaluate. The scrape_configs probe now adapts to per-job target lists ([{job, targets:[{instance, health, lastScrape, lastError}]}], tolerant of otel-mcp-server's metrics_targets and Prometheus /api/v1/targets): only jobs with at least one target up (or of unknown health) land in mcp.discovered.scrape_jobs and attest scraping; jobs whose every target is down go to mcp.discovered.scrape_jobs_down; every target's health is kept in mcp.observed.scrape_targets (JSON, 200 entries, lastError trimmed to 200 chars). When no job is up the metrics exporter falls back to its scaffold marker.
  • The recording_rules / alert_rules adapters carry health, lastError, lastEvaluation, evaluationTime (and state, activeAt for alerting rules); the fetcher strips them before spec.queries and writes mcp.observed.recording_rules / mcp.observed.alert_rules plus mcp.discovered.recording_rules_unhealthy / alert_rules_unhealthy. Recording rules are stamped per index (mcp.verified.queries.recording_rules[<i>] — the symbol the adapter actually reads; they used to project Declared despite the group stamp) and only when healthy or health-less; a burn-rate group fed by an unhealthy alerting rule still maps but earns no indexed stamp. Absent health is never treated as unhealthy; legacy job-name string[] probe results still count as health-less scrape evidence.
  • Draft summary (POST /api/draft-from-mcp) gains scrapeJobsDown, recordingRulesUnhealthy, alertRulesUnhealthy; the studio lists N scrape jobs down: … and N rules unhealthy: … when non-empty.

Probe honesty (what the MCP could and could not answer)

  • A probe family whose every candidate is absent from tools/list was recorded as unsupported by the probe loop but never emitted; the studio read it as "probed, none found" and warned that the family "returned empty". The pack now stamps mcp.probesUnsupported (a restricted MCP tier) beside mcp.probesFailed (a hole of unknown size), and mcp.probeErrors.<family> carries the last erroring candidate's message (trimmed to 200 chars). fetchMcp returns its probeFailures map; buildCanonicalPack accepts it.
  • partialLiveEvidence (vendored, zero-import) gains additive keys only: unsupported, errors, and vantage (full / partial / restricted / lost / none); partial keeps meaning outright failures. GET /api/live-status returns probesFailed and probesUnsupported; the MCP badge goes to its error state for failed probe families and names a restricted tier in its title; the draft summary lists unsupported families as "— not exposed by this MCP" (with the probe error inline for failed ones) and warns once about the restricted tier; the partial-evidence banners append the not-exposed families.

Journeys: studio grade parity, vantage-aware gate, vantage-lost records

  • One score for one comparison. tools/lib/journey.mjs attaches the requirement-chain comparison (comparePackBranches) to the diff as traceabilityGraph before grading — exactly what the studio's /api/diff does — so the CLI and the studio score Drift-free on the same construct. The run record says which construct scored it (grade.driftConstruct: requirement-chain / diff-buckets) and carries the chain rollup (traceability).
  • The vantage is part of the verdict. Run records gain probes {attempted, succeeded, empty, failed, unsupported}, probeErrors, vantage, toolsExposedCount, scrapeJobsDown and unhealthyRules, read from Pack B's fetcher annotations (a file-sourced B reports empty lists, 0, null, none). Two new gate keys: failOnPartialEvidence: true breaches when any probe family failed or the vantage is entirely lost; maxUnhealthy: N breaches when scrape jobs down + unhealthy rules exceed N. Empty and unsupported families never breach on their own. The markdown report prints Live probes and On-wire health lines; the README gate example shows both keys.
  • Vantage loss is a point in history, not a hole. When Pack B is an mcp: source and the fetch itself fails, the journey writes an outcome: vantage-lost record (error, both sources, timing, no fabricated grade) before rethrowing — exit code stays 2, the studio route still answers 502. Configuration errors (missing file, unset authEnv) leave no record. packc journey list and the studio Journeys view render the outcome without a fake alignment undefined%.

Canon: two authoring-only differences stop reading as drift

  • tools/lib/promql-canon.mjs removes the spaces around the symbolic binary operators + * / % ^ == != <= >= < > =~ !~ (outside strings, parser-proven expressions only), so a / b ≡ a/b and rate(x[5m]) > 0.5 ≡ rate(x[5m])>0.5. - is left alone (unary/binary ambiguity), keyword operators keep the space that bounds them, label-value strings stay byte-identical, and a space inside an operator (a < = b) is never glued into a different one. Decision-bearing differences (> vs >=, thresholds, =~ vs !~) still read as drift.
  • tools/lib/artefact-model.mjs behaviorFor strips a leading ref: from slo, sli, trigger, error_budget_policy, and from expr when the whole value is a reference (ref:slis.x), via a pure stripRefFields copy of the spec — identity already stripped it, so a burn alert declared slo: ref:x paired with the live slo: x and then reported itself drifted on its own binding. Both modules are vendored downstream (docs/VENDORING.md): additive behaviour only, no new imports, signatures unchanged — re-copy the pair together.

Review follow-ups (false-assurance re-audit)

  • Placeholders never pair. The studio construct (diffPacks + comparePackBranches, which the journey now grades on) still paired a live Scaffold placeholder with a declared artefact: a declared burn-rate alert on the live pack's first SLO read aligned when it carried the compiler's default windows (exactly the placeholder's) or drifted otherwise — never declared, not live — and BAK-01 / PIP-PRC-01/02 / the SEV1 route aligned or drifted against scaffold backends, processors and the fallback route. Only the diff-bucket grade path parked them. diffPacks now removes source: Scaffold artefacts from both sides before pairing into a new scaffold bucket (summary.scaffold; a repo placeholder is not a declaration either, so a real live route the repo only had a placeholder for reads live, not declared); comparePackBranches refuses a Scaffold live node as evidence (declared_only), never emits it as live_only, drops a live-only branch rooted on a scaffold SLO, and does not count a scaffold route/dashboard as proof the live connector can see that kind. Journey facts (alignmentPct, aligned, declaredNotLive → maxDeclaredNotLive) follow; the run record carries drift.scaffold. The studio drift drill, verdict and prototype views read the new bucket beside the older isScaffoldDiffEntry filter.
  • Tiered SLOs on one SLI no longer produce a dangling ref. mapDiscoveredBurnAlerts resolved groups in one pass and only tracked re-id candidates as claimed, so an x_99_9 group could re-identify the SLO an x_99 group had already bound exactly — spec.slos then held x_99_9 while burn_rate_alerts[0].slo still said x_99 (POL-01 Verified with a ref pointing at nothing; the schema's pattern-only Ref never caught it). Exact matches now bind and claim first; base matches re-identify only unclaimed placeholders; the leftover group is reported in mcp.discovered.alert_rules_unmapped. Exact matches apply the slo_objective/slo_window evidence and drop the SLO's scaffold marker like a re-id does; a re-id to an id that is not a schema Slug (length included) is refused.
  • Health follows the evidence down the chain. An SLI inferred from recorded rules of which one is reported unhealthy is no longer stamped mcp.verified.slis.<id> (new mcp.discovered.slis_unhealthy); an SLO bound by a burn group fed by an unhealthy rule is not stamped either; rule health is keyed by name with any-unhealthy-wins semantics (a duplicated name reported err in one group and ok in another stamped both spec entries Verified while being named unhealthy); the requirement traceability chain lists an unhealthy live alerting rule with verified: false / health: err, does not let it (or a scaffold burn-rate entry) close missing_alert_evidence, and counts withAlerts on evidence only — the studio chain rows and the requirement-trace drawer show (unhealthy).
  • Under-reporting fixed too. Discovered dashboards are stamped per id (mcp.verified.dashboards.<id>, the symbol the adapter reads) beside the aggregate key, so they project Verified instead of Declared. mcp.probeErrors.<family> is written only for families whose outcome is failed — a family whose later candidate answered no longer carries the earlier candidate's error while sitting in probesSucceeded.
  • Fresh names the vantage. The grade's Fresh criterion keeps its staleness pass/fail (scoring is out of scope) but its detail now says vantage lost: no probe family answered (failed: …; not exposed: …) or vantage partial: failed …, so a refresh that saw nothing does not read as a fresh look at production. .journey-outcome.is-lost uses a new --warn text token instead of a hard-coded colour.

Fixtures re-pinned (npm run test:contracts:update): tools/fixtures/mcp/adapted/scrape_configs.json is now per-job target health instead of a job-name string[] (the alertmanager job is present with its down target and lastError); recording_rules.json / alert_rules.json carry the observation fields (health, lastError, lastEvaluation, evaluationTime, and state / activeAt). No crawler or compiler golden changed.

Suites: tools/test-fetch-live.mjs (placeholder + mapping paths, every scaffold key and its adapter projection, the excluded down job / all-down / legacy / adapter-shape cases, per-index rule stamps, the unsupported/failed split and per-family error mapping), tools/test-diagnostic-grade.mjs (every vantage), server/test-smoke.mjs (live-status fields on a planted live pack), tools/test-journey.mjs (both gate keys, record fields, studio parity, the vantage-lost record via a closed loopback port, the no-record rule), tools/test-promql-canon.mjs and tools/test-diff.mjs (operators, fences, ref: ≡ bare). Docs: MCP_INTEGRATION.md (annotation reference, what the fetcher invents, on-wire liveness, burn mapping, journeys), DIFF.md, ADAPTER.md, VENDORING.md, README.

Studio Diagnose views rejoin the behavioural engine

  • Diagnose → Compare cards classify from the diff again. The side-by-side view derived its in-both / only-in chips by recomputing keys client-side as defines || id — a lossy approximation of the old diff keyspace that stopped matching entirely once the engine moved to behavioural identity keys (kind::{identity}): the summary band said N matched while all cards below rendered unclassified and the "In both" / "A − B" slice filters returned nonsense. Cards and slice filters now classify by the artefact ids embedded in the diff entries themselves (unique per pack side) — no client-side key math to drift again.
  • Traceability (Advanced) is diff-driven. Its repo-vs-live buckets were re-derived locally by raw id match + whole-object JSON equality (only annotations/_sub stripped), so behaviourally-identical artefacts with different ids double-counted as declared-not-verified and verified-not-declared, and endpoint/expression-formatting differences flagged Stale. The four buckets now re-bin the server diff — aligned = inBoth-aligned, stale = inBoth-drifted (with the engine's delta fields named on each row) — and reconcile with the drift drill's unlensed totals: out-of-scope artefacts and scaffold placeholders are parked the same way (counts + live-scope control shown), while the drill additionally applies the active product lens. Panels stay compared at dashboard granularity (the engine's rule), now stated in the view. The view fetches the diff itself when it isn't loaded yet. Suppress/resolve prefs now key on behavioural identity keys (collision entries append the artefact symbol so #NN ordinal reshuffles don't migrate a suppression), so previously-saved entries no longer match and reset once.
  • Stale-diff hardening. state.diff is stamped with the selection it was computed for (packs + envs + scope + service) and every consumer treats a mismatch as no diff, refetching instead of rendering — previously a Pack A/env switch could leave the views classifying the new pack against the old diff's positional ids. Late responses for a superseded selection are discarded, concurrent renders share one in-flight fetch, and the three loading branches only re-render on progress (an unregistered Pack A now gets an honest error instead of a render loop).
  • prettyDiffKey learned the behavioural keyspace (kind::{identity-json} with #NN occurrence ordinals) — backend, pipeline-stage, storage, mesh/collection, product-keyed, scrape-job, imports, and remediation identities render as readable labels instead of raw JSON, and forecast alerts are no longer mislabelled as burn-rate alerts.

Account menu: profile, change password, sign out

  • The identity chip in the studio header is now an account menu: who you're signed in as (and the auth mode), change password… (stand-alone mode — OIDC passwords belong to the IdP), and sign out. It renders in identity postures only; the open posture (OBSERVOGRAM_AUTH=off or no users file) has no accounts to manage.
  • Signed-in users can finally change their password: /auth/change-password now also accepts a session. The current password is required — that knowledge is the CSRF defence — and wrong guesses share the login damper (5 failures → 30s lockout). The mid-login forced-change flow (flow cookie, skip) is behaviourally preserved, with one deliberate alignment: a user deleted mid-flow now gets "unknown user" before password validation, matching the session path.
  • Login now clears a leftover pwchange flow cookie. An abandoned forced change (admin/admin typed on a shared browser, then closed) used to leave a 10-minute flow cookie that shadowed the session on /auth/change-password — the account menu's link could silently target the other account. A normal login expires it.
  • Known limitation (stateless sessions, pre-existing model): rotating a password does not revoke other outstanding sessions — an already-stolen cookie rides out its TTL (default 8h). Real revocation needs a per-user epoch in users.json checked at session verification.
  • Suite: 15 new assertions in server/test-auth-local.mjs — the session variant of the page (current-password field, no skip control; the flow variant is pinned to lack the field), wrong-current 401 in JSON and HTML, short-replacement 400, rotation in JSON and HTML (302 home), old password dead / new signs in, deleted-user GET/POST with a live session, the stale-flow-cookie clear on login, the 429 lockout, and the damper being shared with /auth/login.

"Skip for now" on the forced password change

  • The first-boot change-password page now offers Skip for now — ask again next sign-in: a session is issued without replacing admin/admin, the forced-change flags stay set, and every subsequent sign-in re-prompts. New POST /auth/change-password/skip, credentialed by the same signed flow cookie as the change itself.
  • Deliberately narrow: skipping works only while the record still holds the seeded default — an admin-set temporary password (mustChange without seededDefault) stays a forced change, and the loopback guard is unaffected (a skipped default still refuses to bind beyond loopback).
  • Suite: 12 new assertions in server/test-auth-local.mjs — skip control rendering, no-cookie rejection, session issuance with flags intact (JSON and browser-form paths), network-bind refusal after a skip, re-prompt at next sign-in, and the temporary-password 403 (JSON and HTML, skip control absent).

Explicit collision reporting + collision-proof coverage

  • New top-level collisions on the diffPacks() result ({ layer, kind, key, aCount, bCount }) and graph.collisions on buildDependencyGraph() ({ key, kind, count }): every identity key held by more than one artefact, surfaced explicitly so callers can fail loud instead of inferring collisions from #NN occurrence suffixes. Additive — no existing bucket, key, or summary count changes.
  • Alert-route identity deliberately stays severity-keyed. Widening it with channel kinds was prototyped and declined: the live connector and crawler fabricate a channel kind when routing cannot be introspected (fetch-live's SEV1 msteams placeholder, unmapped-receiver stubs), so channel-kind identity would split declared routes from their live placeholders and report them falsely missing in production. Channel changes stay decision-bearing drift on the paired route; a new regression test pins the placeholder pairing.
  • otel/baselines empty identities and pipeline-stage {name} identity are now documented invariants (spec-singular objects; collector batch/2-style duplicates survive via occurrence ordinals), in code comments and docs/DIFF.md.
  • Vendoring contract hardened (docs/VENDORING.md): tools/lib/artefact-model.mjs, promql-canon.mjs, and promql.mjs join the vendorable table and the staying-current diff command — they were always diff.mjs's import closure, but a downstream copy that missed them could silently mix engine generations (the root cause behind a downstream studio's 2026-08 collision report, filed against code replaced on 2026-06-09).
  • Compiler: duplicate-severity routes now emit a valid, complete Alertmanager config. Bringing the crawled pack into strict CI validation exposed the same severity-collision bug in compileAlertmanager(): one receiver per route named by severity, so N same-severity routes emitted N identically-named receivers (amtool check-config rejects the file) — and their identical-matcher sibling routes meant only the first could ever fire. Duplicates now get ordinal receiver names (svc-sev2-2, …) and identical-matcher groups chain with continue: true, so every declared channel notifies. Regression-tested in tools/test-compile.mjs.
  • Suite: the collision fixture grew same-named pipeline stages and a fourth same-severity route; tools/test-packs.mjs now asserts the self-diff invariant (every artefact preserved, alignment/Jaccard 1.0) for every bundled pack; examples/krystaline.repo.pack.Carlos.yaml renamed to krystaline-repo-carlos.pack.yaml so the one crawled real-world pack (324 artefacts) finally enters the suite, with its genuine collisions pinned in PACK_EXPECTATIONS; tools/test-traceability-graph.mjs gained duplicate-identity-key coverage.

Chronicled retroactively: the identity-collision safety net (2026-06-09)

  • Never changelogged when it landed (commit 4c5bae4, between 0.3.0 and this line): diffPacks groups identity keys instead of last-write-wins Map.set, pairs exact behavioural matches first, and preserves duplicates with stable #NN occurrence suffixes; canonicalize() began treating empty arrays like absent fields in the same commit. buildDependencyGraph (created the same day in 6fcb7ef) has had the grouped behaviour from birth. If a vendored diff.mjs still contains aByKey.set(keyOf(a), a), it predates 2026-06-09 — re-vendor the full engine closure.

Ships like Grafana: default admin on first boot

  • First boot with nothing configured now seeds a default admin user (password admin) and the login page is live — the studio is a signed-in app out of the box, no CLI setup. The password change is forced at first sign-in (new /auth/change-password, signed short-lived flow cookie, no session until the change lands). Softened later in this release: skippable per session for the seeded default — see the "Skip for now" entry above.
  • The default credential never crosses loopback: a network boot never seeds admin/admin (it refuses to start and names the options), and start() refuses to bind beyond loopback while an already-seeded default is unchanged. Container/network boots set OBSERVOGRAM_ADMIN_PASSWORD=<secret> to seed a real credential (no forced change); the same env var also replaces a still-default admin record, so a workspace seeded on loopback can be moved behind the network without a deadlock.
  • The seed backs off from any expressed intent: OIDC configured, an existing users file, OBSERVOGRAM_API_TOKEN (the token-only 10B contract is unchanged), or armed tenancy.
  • New OBSERVOGRAM_AUTH=off — the one switch that disables identity entirely (no login, no seeding, /auth/* answers 404): the pre-0.5 open posture, kept for dev shells, scripts and CI. This is a default-behaviour change: npm start on a fresh workspace now lands on a login page.
  • Stand-alone auth arming is now per-request (routes register once, posture follows the users file), so the seed at start() and npm run users both take effect without a re-import; npm run users -- passwd clears a pending forced-change flag.
  • Suite: server/test-auth-local.mjs grew 18 assertions — seed shape, 401-by-default, loopback guard, the full forced-change flow, OBSERVOGRAM_ADMIN_PASSWORD (fresh seed + rescue), never-seed-on-network-boot, and the off switch; server/test-smoke.mjs runs under OBSERVOGRAM_AUTH=off and is the open-posture regression.

Tomograph → Observogram (rebrand, backwards-compatible)

  • Every user-facing name is now Observogram: package name and npm bin (observogram; packc unchanged), studio shell, login page, CLI help, Docker image (observogram:<version>), k8s manifests, docs, and diagrams.
  • Nothing breaks on upgrade — the legacy spellings keep working via tools/lib/brand-env.mjs (drop planned for 0.6):
    • Env vars: every knob reads OBSERVOGRAM_* first, then the legacy TOMOGRAPH_* (WORKSPACE, API_TOKEN[_LABEL], OIDC_*, SESSION_*, USERS_FILE, INSECURE_NO_AUTH, ALLOW_LOCAL_MCP, STRICT_SNAPSHOT, BUILD, DEBUG, DIFF_SCOPE, MCP_TIMEOUT_MS, GRAFANA_*).
    • Workspace: the default directory is .observogram/, but an existing .tomograph/ keeps being used as-is (no migration needed, no data orphaned).
    • HTTP: the studio sends X-Observogram-Org / X-Observogram-CSRF; the server accepts the X-Tomograph-* spellings too and echoes X-Observogram-Org.
    • Sessions: the cookie is observogram_session; sessions issued pre-rebrand under tomo_session stay valid (same HMAC secret), and logout clears both.
    • Pack annotations: writers emit observogram.* (diff.scopeMode, retrofeed.*, services); readers accept the tomograph.* namespace from pre-rebrand packs, new key wins when both exist.
  • Legacy-compat is CI-asserted (workspace env fallback, org/CSRF headers, session cookie, annotation namespace).

Vendorable verdict engines (downstream-studio decoupling)

  • studio/diagnostic-grade.mjs is now zero-dependency (L4_SUBGROUPS inlined) — vendorable verbatim, like tools/lib/diff.mjs and tools/lib/protocols.mjs.
  • studio/verdict-ui.mjs no longer reads global state: buildVerdictModel() / projectGrade() take { pack, packB, diff, compareBId, catalogEntry } explicitly and loadRunHistory() accepts an injectable fetchFn.
  • New studio/compare-catalog.mjs — catalogEntryFor() + LAYERS_FOR_DIFF extracted from the 3000-line compare-view (which re-exports them), so verdict modules import the small catalog module instead of the whole view.
  • See the new docs/VENDORING.md for the downstream contract.

The studio host seam (no view imports app.mjs for orchestration)

  • New studio/host.mjs — a zero-import module holding the app-level callbacks (loadPackB, openDeployModal, renderMainView, renderTabs); app.mjs fills it once at boot via initHost(). Every view module (compare, compile, layers, drawer, journeys, references, atlas, all protos) now calls these through host (imported as appHost) instead of importing app.mjs — app-specific helpers remain direct imports until each view adopts the model-passing convention.
  • New tools/test-studio-graph.mjs in npm test — links the entire studio ES-module graph headlessly, so a dangling import/export anywhere in the studio fails CI (node --check is parse-only and cannot catch it).
  • New studio/verdict-ui.css — the .mc-* widget styles verdict-ui.mjs emits, split out of app.css along the vendoring seam (the file header documents the theme variables it expects).
  • New docs/UI_CONVENTIONS.md — the adopt-on-touch conventions: host seam, loader/renderer split with injectable fetchFn, render(container, model, host) signature, and per-zone CSS prefixes.
  • Rebrand follow-through: the last "tomogram" vocabulary is now "observogram" (Discover's scan title/tagline, hero-asset path assets/observogram-hero.png, docs).

Fixed: comparing legacy imports (and any cross-service packs)

  • The PACK B picker filtered out every pack whose service didn't match the active one — two legacy imports (each deriving its own service from the old pack id) could never be compared. Cross-service packs are now selectable under an "other services" group; same-service packs and live aggregates still lead the list, and the diff's service scope keeps cross-service comparisons honest.

Stage 2 tenancy — workspace-per-org

  • New server/tenancy.mjs: org registry in <workspace>/orgs.json (the file existing arms tenancy, mirroring users.json), per-request org context via AsyncLocalStorage, idempotent flat → orgs/default/ boot migration. Requires identity; refuses to boot otherwise (fail-closed).
  • workspaceRoot() is context-aware: registry, deploys, snapshots, journeys, runs all answer from <workspace>/orgs/<orgId>/ inside a request. The in-memory upload registry and workspace index cache are keyed per org.
  • Org selection via X-Observogram-Org (default: first membership); membership enforced in middleware; bearer token = deployment-level service account. New GET /api/orgs; /auth/me carries memberships; roles recorded for Stage 3 (not yet enforced).
  • New CLI npm run orgs -- create|remove|add-member|remove-member|list.
  • Studio: active org resolved before the first catalog fetch; ORG chip (switcher when multi-org) in the OBSERVA bar.
  • New suite server/test-tenancy.mjs — the isolation gate: org B reads/writes nothing of org A (API + filesystem-path assertions), migration, per-org reset, fail-closed posture.

Previous pack format (layered JSON) supported again

  • New tools/lib/legacy.mjs — detects the pre-v1.2 layered "studio-shape" JSON and upconverts it to a canonical v1.2 manifest. Lossless (every legacy artefact kept verbatim in legacy.artefact.* annotations), honest (every schema-forced placeholder marked crawler.scaffold.* → projects as Scaffold, never Declared), deterministic.
  • POST /api/validate upconverts legacy uploads transparently; the studio toast reports the conversion (N artefacts mapped, M scaffolds).
  • New CLI npm run upconvert-legacy <file> [-o out.pack.json].
  • The four original layered JSON packs restored from git history as working examples: examples/legacy/ (+ README documenting the format).
  • New suite tools/test-legacy-pack.mjs gates all four examples on every npm test; the validator's gatekeeper error now points legacy packs at the converter.