Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .changeset/liveness-ten-preview-claims.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
---

docs(liveness): re-verify the last ten preview-only `live` claims — eight were wrong (#3686)

Closes the preview-claim sweep. Of the 13 properties a 2026-06 pass marked
`live` citing only a metadata-admin preview panel, **10 were wrong** (77%).
Corrected: `action.shortcut`, `action.bulkEnabled`, `flow.active`,
`skill.triggerPhrases`, `tool.category`, `tool.requiresConfirmation`,
`tool.active`, `tool.builtIn` → `dead` + `authorWarn`; `action.execute` and
`flow.status` keep `live` with their real runtime readers cited. Also fixes
`flow.json`'s file-level note, which claimed "status/active gate nothing" —
true when written, falsified a month later by `497bda853`. Ledger-only;
releases nothing.
55 changes: 44 additions & 11 deletions packages/spec/liveness/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,46 @@ three were re-verified in 2026-07 and **all three were wrong or misleading**:
the verdict was right for the wrong reason and hid a five-surface silent
no-op (evidence corrected; the gap itself fixed in objectui#2863)

The remaining ten preview-only claims (`action.execute`/`shortcut`/`bulkEnabled`,
`flow.status`/`active`, `skill.triggerPhrases`, `tool.category`/
`requiresConfirmation`/`active`/`builtIn`) are **unverified** — treat them as
suspect until someone cites a real runtime reader. When in doubt, the honest
status is `dead` + `authorWarn`: an author who gets a warning for a property
that turns out to work loses nothing; an author who gets silence for a property
that does nothing ships a bug.
**All thirteen have now been re-verified (2026-07, #3686). Final tally: 3 stand
as `live`, 10 were wrong** — a 77% error rate for the preview-renderer standard:

| Verdict | Properties |
|---|---|
| `live`, evidence corrected to the real reader | `action.execute` (ActionRunner + the spec transform), `action.disabled` (six render surfaces), `flow.status` (engine gates binding + execution since `497bda853`) |
| corrected to `dead` + `authorWarn` | `action.shortcut`, `action.bulkEnabled`, `flow.active`, `skill.triggerPhrases`, `tool.category`, `tool.requiresConfirmation`, `tool.active`, `tool.builtIn`, `skill.permissions`*, `agent.knowledge` |

\* `skill.permissions` was subsequently pruned outright — it was never enforced.

Note the two failure directions the sweep exposed. Most entries **overstated**
liveness. But `flow.status` was *understated*: the file-level note still said
"status/active gate nothing", true when written and falsified a month later by
`497bda853`. **A ledger entry is a claim with a timestamp; code moves under it
in both directions.**

When in doubt, the honest status is `dead` + `authorWarn`: an author who gets a
warning for a property that turns out to work loses nothing; an author who gets
silence for a property that does nothing ships a bug.

### How to verify a claim without fooling yourself

Three false conclusions were published during this sweep, all from the same
mistake: **a strong negative claim ("nothing reads X") resting on a search whose
result set was silently truncated or filtered.** Concretely:

1. `… | head -3` hid the real hit further down the list.
2. A pathspec glob `packages/*/src` never matched the nested
`packages/app-shell/src/layout/…`.
3. **On macOS, `git grep -E` silently does not honour `\b`** — `git grep -cE
"\.active\b" flow.zod.ts` returns *nothing* on a file that provably contains
three `.active` occurrences (`git grep -cw active` finds them). Any absence
conclusion drawn from a `\b` pattern on this platform is a false negative.

So: a grep can only prove **presence**. To prove absence, either close the call
graph by hand (declaration → registration → accessor → *caller*, which is how
`action.shortcut` and `tool.active` were settled) or — cheapest and most
decisive for this ledger — **author the property, boot the app and look**. That
is how the `app.badge`/`separator` question was finally settled after two wrong
grep-based rounds.

## Runtime proofs — prove-it-runs (ADR-0054)

Expand Down Expand Up @@ -225,14 +258,14 @@ EOF
|---|---|---|---|---|---|
| object | 40 | – | 0 | 1 | aspirational tier (versioning/softDelete/search/recordName/keyPrefix) + tags/active/abstract REMOVED (#2377) — tombstoned in UNKNOWN_KEY_GUIDANCE; `enable.trash`/`mru` REMOVED (#2377 close-out) — tombstoned in the now-`.strict()` ObjectCapabilities; `isSystem` + `enable.searchable` CORRECTED to live (#2377 — sharing default-model + global-search opt-out; 2026-06 audit missed both readers); `tenancy.strategy`/`crossTenantAccess` REMOVED post-15.0 (#2763) |
| field | 55 | – | 0 | – | healthy — full dead set (vectorConfig/fileAttachmentConfig/dependencies, then referenceFilters/columnName/index) REMOVED (#2377); columnName also dropped the ADR-0062 D7 lint + StorageNameMapping column helpers |
| flow | 27 | – | 4 | – | dead = description/template/nodes.outputSchema/errorHandling.fallbackNodeId (engine uses fault edges) |
| action | 35 | 1 | 0 | – | `disabled` went LIVE via metadata-admin authoring UI (2026-06 audit missed objectui); `type:'form'` CORRECTED to live (objectui ActionRunner.executeForm, #2377); dead `timeout` REMOVED (#2377) |
| flow | 26 | – | 5 | – | dead = description/template/nodes.outputSchema/errorHandling.fallbackNodeId (engine uses fault edges) + `active` CORRECTED to dead 2026-07 (deprecated no-op — `status` is what gates binding/execution since 497bda853; the file `_note` claiming otherwise is fixed) |
| action | 33 | 1 | 2 | – | `type:'form'` CORRECTED to live (objectui ActionRunner.executeForm, #2377); dead `timeout` REMOVED (#2377); `disabled` live for real since objectui#2863 (six surfaces); `shortcut` + `bulkEnabled` CORRECTED to dead 2026-07 — registered into ActionEngine but their accessors have no non-test caller (#3686 sweep) |
| hook | 11 | – | 2 | – | model-healthy; only label/description dead (benign) |
| permission | 32 | – | 0 | – | CRUD/FLS/RLS live; dead `contextVariables` REMOVED (ADR-0105 D11 — RLS resolves only the `current_user.*` built-ins plus runtime-staged `rlsMembership` sets) |
| position | 4 | – | – | – | (role's ADR-0090 successor) fully live |
| agent | 13 | 5 | 1 | – | dead `tenantId` + `planning.strategy`/`allowReplan` REMOVED (#2377) — only `planning.maxIterations` live; autonomy tier experimental; `knowledge` CORRECTED to dead 2026-07 — `search_knowledge` takes `sourceIds` from the LLM's tool-call args, never from the agent record (#1878 §3 recheck) |
| tool | 9 | 1 | 1 | – | `permissions` dead — tool invocation not permission-gated by it |
| skill | 9 | – | | – | `permissions` REMOVED 2026-07 never permission-gated anything (the cloud SkillRegistry reads only `active`/`triggerConditions`/`tools`); owner call was prune, not enforce. Gate at the agent (`access`/`permissions`, #1884) or on the tools' underlying actions |
| tool | 5 | 1 | 5 | – | the whole authoring surface is inert: `permissions` (not permission-gated), plus `category`/`requiresConfirmation`/`active`/`builtIn` CORRECTED to dead 2026-07 (#3686 sweep). ⚠️ `requiresConfirmation` is SAFETY-shaped and unenforced on every path — ADR-0033 already resolved to delete it |
| skill | 8 | – | 1 | – | `permissions` REMOVED 2026-07 (never gated anything — owner call was prune, #3704); `triggerPhrases` CORRECTED to dead — phrases are never matched against user messages; activation is `triggerConditions` + the agent's `skills[]` + explicit /skill-name pinning (#3686 sweep) |
| dataset | 19 | – | 0 | – | `measures.certified` (declared-but-unenforced governance flag) REMOVED in 16.0 (#2377) |
| page | 16 | – | – | 1 | fully live + one planned |
| view | 70 | 0 | 5 | – | list/form drilled via `children` (#2998 Track B); dead = list.{responsive,performance} + form.{data,defaultSort,aria}, all but aria authorWarn'd; form.{buttons,defaults} now live — objectui ObjectForm folds them onto its flat props (framework#1894 / #2998); audit-era DEAD lines superseded by re-verification (submitBehavior, sharing.lockedBy, list ViewData providers, and the ADR-0021 chart shape — all live now); level-2 dead residue (userActions.buttons, addRecord.mode/formView, tabs[].order) noted on parents — one drill level only |
Expand Down
20 changes: 12 additions & 8 deletions packages/spec/liveness/action.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@
},
"execute": {
"status": "live",
"evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ActionPreview.tsx:154",
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
"evidence": "objectui packages/core/src/actions/ActionRunner.ts:704 (executeScript reads action.execute) + framework packages/spec/src/ui/action.zod.ts:577-580 (.transform lowers execute -> target)",
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Verdict stands but for a different reason: TWO independent runtime readers exist. NOTE a live divergence — the spec transform keeps `target` when both are set (action.test.ts:1032), while ActionRunner does `execute || target`, so an action declaring both runs different code client- vs server-side."
},
"params": {
"status": "live",
Expand Down Expand Up @@ -105,14 +105,18 @@
"note": "EVIDENCE CORRECTED 2026-07 (#1878 §3 recheck): previously cited the metadata-admin PREVIEW renderer, which only echoes the authored value. At that time the predicate was actually enforced on ONE of six surfaces (#1885 wired action:button alone) — the 'live' verdict was right for the wrong reason and hid a real silent no-op on the other five. objectui#2863 wired the rest; showcase specimen `showcase_archive_task` (#3643) dogfoods it."
},
"shortcut": {
"status": "live",
"evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ActionPreview.tsx:158",
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
"status": "dead",
"evidence": "registered into ActionEngine.shortcuts[] (objectui packages/core/src/actions/ActionEngine.ts:150) but getShortcuts()/handleShortcut() have no non-test caller and no keydown listener feeds them",
"authorWarn": true,
"authorHint": "Not wired — nothing dispatches keydown events to actions. objectui's keyboard stack (useKeyboardShortcuts / KeyboardProtocol) is hand-registered and never touches ActionEngine; register the key there and have its handler invoke the action by name.",
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Also declared in action.form.ts:69, so pruning would need a form change + a 4-locale i18n regen."
},
"bulkEnabled": {
"status": "live",
"evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ActionPreview.tsx:159",
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
"status": "dead",
"evidence": "registered into ActionEngine (objectui ActionEngine.ts:151) but getBulkActions()/executeBulk() have no non-test caller; the multi-select toolbar is driven by the LIST VIEW's bulkActions/bulkActionDefs (plugin-grid ObjectGrid.tsx:1602-1612)",
"authorWarn": true,
"authorHint": "Not wired — the multi-select toolbar reads the list view's `bulkActions` / `bulkActionDefs`, not this flag. Declare the action on the view instead of setting bulkEnabled here.",
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Distinct from the live bulk fast-path (core/src/actions/bulkFastPath.ts executeBulkBatch), which never consults this flag. Also in action.form.ts:79."
},
"ai": {
"status": "live",
Expand Down
14 changes: 8 additions & 6 deletions packages/spec/liveness/flow.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"type": "flow",
"_note": "FlowSchema. Seeded from docs/audits/2026-06-flowschema-property-liveness.md. Consumers: service-automation engine + node executors. runAs is marker-driven experimental (spec). status/active gate nothing (engine uses an in-memory flowEnabled map via toggleFlow).",
"_note": "FlowSchema. Seeded from docs/audits/2026-06-flowschema-property-liveness.md. Consumers: service-automation engine + node executors. runAs is marker-driven experimental (spec). CORRECTED 2026-07: the previous note claimed 'status/active gate nothing' — that was true when the audit ran but became FALSE with commit 497bda853 (2026-07-05, 'honor flow status for enable/disable'). `status` now gates binding AND execution; `active` remains a deprecated no-op.",
"props": {
"name": {
"status": "live",
Expand Down Expand Up @@ -28,13 +28,15 @@
},
"status": {
"status": "live",
"evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/FlowPreview.tsx:105",
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
"evidence": "packages/services/service-automation/src/engine.ts:1374-1382 (obsolete/invalid -> flowEnabled=false), :1387 (gates activateFlowTrigger), :1704 + :3097 (execute-time 'Flow is disabled' guard)",
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Verdict stands, evidence replaced: it is genuinely engine-consumed since 497bda853. The previously cited FlowPreview line number did not even contain a status read."
},
"active": {
"status": "live",
"evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/FlowPreview.tsx:105",
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
"status": "dead",
"evidence": "spec already marks it '(Deprecated: use status)'; the only reader anywhere is a display fallback in objectui FlowPreview.tsx:157 (`d.status ?? (d.active ? ...)`)",
"authorWarn": true,
"authorHint": "Deprecated no-op — writing `active: false` does NOT stop a flow. Use `status: 'obsolete'` (or 'invalid') to unbind and disable it, `status: 'active'` to arm it.",
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Sharper than a plain no-op: the spec default is `false` while the engine treats an unset flow as enabled, so the key reads as if it disabled the flow when it does nothing."
},
"runAs": {
"status": "live",
Expand Down
8 changes: 5 additions & 3 deletions packages/spec/liveness/skill.json
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,11 @@
"note": "inactive skills dropped."
},
"triggerPhrases": {
"status": "live",
"evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/SkillPreview.tsx:44",
"note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file."
"status": "dead",
"evidence": "cloud service-ai/src/skill-registry.ts:266 copies it into toSummary() and assistant-routes.ts:98,129 serve it — but nothing consumes the field; skill activation is triggerConditions + the agent's skills[] allowlist + explicit /skill-name pinning (skill-registry.ts:128-175, assistant-routes.ts:189-196)",
"authorWarn": true,
"authorHint": "Phrases are never matched against the user's message. Routing is `triggerConditions` (AND of context field/operator/value) intersected with the agent's `skills[]`, plus explicit /skill-name pinning; put routing intent in triggerConditions, and describe intent in description/instructions (the LLM does see those).",
"note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. A runtime path does read it, but only to hand it back to API clients — a dead-end projection, not consumption."
}
}
}
Loading
Loading