diff --git a/.changeset/liveness-ten-preview-claims.md b/.changeset/liveness-ten-preview-claims.md new file mode 100644 index 0000000000..88e8a6cdf2 --- /dev/null +++ b/.changeset/liveness-ten-preview-claims.md @@ -0,0 +1,14 @@ +--- +--- + +docs(liveness): re-verify the last ten preview-only `live` claims — eight were wrong (#3686) + +Closes the preview-claim sweep. Of the 13 properties a 2026-06 pass marked +`live` citing only a metadata-admin preview panel, **10 were wrong** (77%). +Corrected: `action.shortcut`, `action.bulkEnabled`, `flow.active`, +`skill.triggerPhrases`, `tool.category`, `tool.requiresConfirmation`, +`tool.active`, `tool.builtIn` → `dead` + `authorWarn`; `action.execute` and +`flow.status` keep `live` with their real runtime readers cited. Also fixes +`flow.json`'s file-level note, which claimed "status/active gate nothing" — +true when written, falsified a month later by `497bda853`. Ledger-only; +releases nothing. diff --git a/packages/spec/liveness/README.md b/packages/spec/liveness/README.md index d2fc48e1f7..f74680b7b4 100644 --- a/packages/spec/liveness/README.md +++ b/packages/spec/liveness/README.md @@ -61,13 +61,46 @@ three were re-verified in 2026-07 and **all three were wrong or misleading**: the verdict was right for the wrong reason and hid a five-surface silent no-op (evidence corrected; the gap itself fixed in objectui#2863) -The remaining ten preview-only claims (`action.execute`/`shortcut`/`bulkEnabled`, -`flow.status`/`active`, `skill.triggerPhrases`, `tool.category`/ -`requiresConfirmation`/`active`/`builtIn`) are **unverified** — treat them as -suspect until someone cites a real runtime reader. When in doubt, the honest -status is `dead` + `authorWarn`: an author who gets a warning for a property -that turns out to work loses nothing; an author who gets silence for a property -that does nothing ships a bug. +**All thirteen have now been re-verified (2026-07, #3686). Final tally: 3 stand +as `live`, 10 were wrong** — a 77% error rate for the preview-renderer standard: + +| Verdict | Properties | +|---|---| +| `live`, evidence corrected to the real reader | `action.execute` (ActionRunner + the spec transform), `action.disabled` (six render surfaces), `flow.status` (engine gates binding + execution since `497bda853`) | +| corrected to `dead` + `authorWarn` | `action.shortcut`, `action.bulkEnabled`, `flow.active`, `skill.triggerPhrases`, `tool.category`, `tool.requiresConfirmation`, `tool.active`, `tool.builtIn`, `skill.permissions`*, `agent.knowledge` | + +\* `skill.permissions` was subsequently pruned outright — it was never enforced. + +Note the two failure directions the sweep exposed. Most entries **overstated** +liveness. But `flow.status` was *understated*: the file-level note still said +"status/active gate nothing", true when written and falsified a month later by +`497bda853`. **A ledger entry is a claim with a timestamp; code moves under it +in both directions.** + +When in doubt, the honest status is `dead` + `authorWarn`: an author who gets a +warning for a property that turns out to work loses nothing; an author who gets +silence for a property that does nothing ships a bug. + +### How to verify a claim without fooling yourself + +Three false conclusions were published during this sweep, all from the same +mistake: **a strong negative claim ("nothing reads X") resting on a search whose +result set was silently truncated or filtered.** Concretely: + +1. `… | head -3` hid the real hit further down the list. +2. A pathspec glob `packages/*/src` never matched the nested + `packages/app-shell/src/layout/…`. +3. **On macOS, `git grep -E` silently does not honour `\b`** — `git grep -cE + "\.active\b" flow.zod.ts` returns *nothing* on a file that provably contains + three `.active` occurrences (`git grep -cw active` finds them). Any absence + conclusion drawn from a `\b` pattern on this platform is a false negative. + +So: a grep can only prove **presence**. To prove absence, either close the call +graph by hand (declaration → registration → accessor → *caller*, which is how +`action.shortcut` and `tool.active` were settled) or — cheapest and most +decisive for this ledger — **author the property, boot the app and look**. That +is how the `app.badge`/`separator` question was finally settled after two wrong +grep-based rounds. ## Runtime proofs — prove-it-runs (ADR-0054) @@ -225,14 +258,14 @@ EOF |---|---|---|---|---|---| | object | 40 | – | 0 | 1 | aspirational tier (versioning/softDelete/search/recordName/keyPrefix) + tags/active/abstract REMOVED (#2377) — tombstoned in UNKNOWN_KEY_GUIDANCE; `enable.trash`/`mru` REMOVED (#2377 close-out) — tombstoned in the now-`.strict()` ObjectCapabilities; `isSystem` + `enable.searchable` CORRECTED to live (#2377 — sharing default-model + global-search opt-out; 2026-06 audit missed both readers); `tenancy.strategy`/`crossTenantAccess` REMOVED post-15.0 (#2763) | | field | 55 | – | 0 | – | healthy — full dead set (vectorConfig/fileAttachmentConfig/dependencies, then referenceFilters/columnName/index) REMOVED (#2377); columnName also dropped the ADR-0062 D7 lint + StorageNameMapping column helpers | -| flow | 27 | – | 4 | – | dead = description/template/nodes.outputSchema/errorHandling.fallbackNodeId (engine uses fault edges) | -| action | 35 | 1 | 0 | – | `disabled` went LIVE via metadata-admin authoring UI (2026-06 audit missed objectui); `type:'form'` CORRECTED to live (objectui ActionRunner.executeForm, #2377); dead `timeout` REMOVED (#2377) | +| flow | 26 | – | 5 | – | dead = description/template/nodes.outputSchema/errorHandling.fallbackNodeId (engine uses fault edges) + `active` CORRECTED to dead 2026-07 (deprecated no-op — `status` is what gates binding/execution since 497bda853; the file `_note` claiming otherwise is fixed) | +| action | 33 | 1 | 2 | – | `type:'form'` CORRECTED to live (objectui ActionRunner.executeForm, #2377); dead `timeout` REMOVED (#2377); `disabled` live for real since objectui#2863 (six surfaces); `shortcut` + `bulkEnabled` CORRECTED to dead 2026-07 — registered into ActionEngine but their accessors have no non-test caller (#3686 sweep) | | hook | 11 | – | 2 | – | model-healthy; only label/description dead (benign) | | permission | 32 | – | 0 | – | CRUD/FLS/RLS live; dead `contextVariables` REMOVED (ADR-0105 D11 — RLS resolves only the `current_user.*` built-ins plus runtime-staged `rlsMembership` sets) | | position | 4 | – | – | – | (role's ADR-0090 successor) fully live | | agent | 13 | 5 | 1 | – | dead `tenantId` + `planning.strategy`/`allowReplan` REMOVED (#2377) — only `planning.maxIterations` live; autonomy tier experimental; `knowledge` CORRECTED to dead 2026-07 — `search_knowledge` takes `sourceIds` from the LLM's tool-call args, never from the agent record (#1878 §3 recheck) | -| tool | 9 | 1 | 1 | – | `permissions` dead — tool invocation not permission-gated by it | -| skill | 9 | – | – | – | `permissions` REMOVED 2026-07 — never permission-gated anything (the cloud SkillRegistry reads only `active`/`triggerConditions`/`tools`); owner call was prune, not enforce. Gate at the agent (`access`/`permissions`, #1884) or on the tools' underlying actions | +| tool | 5 | 1 | 5 | – | the whole authoring surface is inert: `permissions` (not permission-gated), plus `category`/`requiresConfirmation`/`active`/`builtIn` CORRECTED to dead 2026-07 (#3686 sweep). ⚠️ `requiresConfirmation` is SAFETY-shaped and unenforced on every path — ADR-0033 already resolved to delete it | +| skill | 8 | – | 1 | – | `permissions` REMOVED 2026-07 (never gated anything — owner call was prune, #3704); `triggerPhrases` CORRECTED to dead — phrases are never matched against user messages; activation is `triggerConditions` + the agent's `skills[]` + explicit /skill-name pinning (#3686 sweep) | | dataset | 19 | – | 0 | – | `measures.certified` (declared-but-unenforced governance flag) REMOVED in 16.0 (#2377) | | page | 16 | – | – | 1 | fully live + one planned | | view | 70 | 0 | 5 | – | list/form drilled via `children` (#2998 Track B); dead = list.{responsive,performance} + form.{data,defaultSort,aria}, all but aria authorWarn'd; form.{buttons,defaults} now live — objectui ObjectForm folds them onto its flat props (framework#1894 / #2998); audit-era DEAD lines superseded by re-verification (submitBehavior, sharing.lockedBy, list ViewData providers, and the ADR-0021 chart shape — all live now); level-2 dead residue (userActions.buttons, addRecord.mode/formView, tabs[].order) noted on parents — one drill level only | diff --git a/packages/spec/liveness/action.json b/packages/spec/liveness/action.json index 9ecb3e23e6..5d8b1508de 100644 --- a/packages/spec/liveness/action.json +++ b/packages/spec/liveness/action.json @@ -44,8 +44,8 @@ }, "execute": { "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ActionPreview.tsx:154", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "evidence": "objectui packages/core/src/actions/ActionRunner.ts:704 (executeScript reads action.execute) + framework packages/spec/src/ui/action.zod.ts:577-580 (.transform lowers execute -> target)", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Verdict stands but for a different reason: TWO independent runtime readers exist. NOTE a live divergence — the spec transform keeps `target` when both are set (action.test.ts:1032), while ActionRunner does `execute || target`, so an action declaring both runs different code client- vs server-side." }, "params": { "status": "live", @@ -105,14 +105,18 @@ "note": "EVIDENCE CORRECTED 2026-07 (#1878 §3 recheck): previously cited the metadata-admin PREVIEW renderer, which only echoes the authored value. At that time the predicate was actually enforced on ONE of six surfaces (#1885 wired action:button alone) — the 'live' verdict was right for the wrong reason and hid a real silent no-op on the other five. objectui#2863 wired the rest; showcase specimen `showcase_archive_task` (#3643) dogfoods it." }, "shortcut": { - "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ActionPreview.tsx:158", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "status": "dead", + "evidence": "registered into ActionEngine.shortcuts[] (objectui packages/core/src/actions/ActionEngine.ts:150) but getShortcuts()/handleShortcut() have no non-test caller and no keydown listener feeds them", + "authorWarn": true, + "authorHint": "Not wired — nothing dispatches keydown events to actions. objectui's keyboard stack (useKeyboardShortcuts / KeyboardProtocol) is hand-registered and never touches ActionEngine; register the key there and have its handler invoke the action by name.", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Also declared in action.form.ts:69, so pruning would need a form change + a 4-locale i18n regen." }, "bulkEnabled": { - "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ActionPreview.tsx:159", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "status": "dead", + "evidence": "registered into ActionEngine (objectui ActionEngine.ts:151) but getBulkActions()/executeBulk() have no non-test caller; the multi-select toolbar is driven by the LIST VIEW's bulkActions/bulkActionDefs (plugin-grid ObjectGrid.tsx:1602-1612)", + "authorWarn": true, + "authorHint": "Not wired — the multi-select toolbar reads the list view's `bulkActions` / `bulkActionDefs`, not this flag. Declare the action on the view instead of setting bulkEnabled here.", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Distinct from the live bulk fast-path (core/src/actions/bulkFastPath.ts executeBulkBatch), which never consults this flag. Also in action.form.ts:79." }, "ai": { "status": "live", diff --git a/packages/spec/liveness/flow.json b/packages/spec/liveness/flow.json index 16507d4c78..ffca156e88 100644 --- a/packages/spec/liveness/flow.json +++ b/packages/spec/liveness/flow.json @@ -1,6 +1,6 @@ { "type": "flow", - "_note": "FlowSchema. Seeded from docs/audits/2026-06-flowschema-property-liveness.md. Consumers: service-automation engine + node executors. runAs is marker-driven experimental (spec). status/active gate nothing (engine uses an in-memory flowEnabled map via toggleFlow).", + "_note": "FlowSchema. Seeded from docs/audits/2026-06-flowschema-property-liveness.md. Consumers: service-automation engine + node executors. runAs is marker-driven experimental (spec). CORRECTED 2026-07: the previous note claimed 'status/active gate nothing' — that was true when the audit ran but became FALSE with commit 497bda853 (2026-07-05, 'honor flow status for enable/disable'). `status` now gates binding AND execution; `active` remains a deprecated no-op.", "props": { "name": { "status": "live", @@ -28,13 +28,15 @@ }, "status": { "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/FlowPreview.tsx:105", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "evidence": "packages/services/service-automation/src/engine.ts:1374-1382 (obsolete/invalid -> flowEnabled=false), :1387 (gates activateFlowTrigger), :1704 + :3097 (execute-time 'Flow is disabled' guard)", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Verdict stands, evidence replaced: it is genuinely engine-consumed since 497bda853. The previously cited FlowPreview line number did not even contain a status read." }, "active": { - "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/FlowPreview.tsx:105", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "status": "dead", + "evidence": "spec already marks it '(Deprecated: use status)'; the only reader anywhere is a display fallback in objectui FlowPreview.tsx:157 (`d.status ?? (d.active ? ...)`)", + "authorWarn": true, + "authorHint": "Deprecated no-op — writing `active: false` does NOT stop a flow. Use `status: 'obsolete'` (or 'invalid') to unbind and disable it, `status: 'active'` to arm it.", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Sharper than a plain no-op: the spec default is `false` while the engine treats an unset flow as enabled, so the key reads as if it disabled the flow when it does nothing." }, "runAs": { "status": "live", diff --git a/packages/spec/liveness/skill.json b/packages/spec/liveness/skill.json index 20e36176f3..1bc349b904 100644 --- a/packages/spec/liveness/skill.json +++ b/packages/spec/liveness/skill.json @@ -42,9 +42,11 @@ "note": "inactive skills dropped." }, "triggerPhrases": { - "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/SkillPreview.tsx:44", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "status": "dead", + "evidence": "cloud service-ai/src/skill-registry.ts:266 copies it into toSummary() and assistant-routes.ts:98,129 serve it — but nothing consumes the field; skill activation is triggerConditions + the agent's skills[] allowlist + explicit /skill-name pinning (skill-registry.ts:128-175, assistant-routes.ts:189-196)", + "authorWarn": true, + "authorHint": "Phrases are never matched against the user's message. Routing is `triggerConditions` (AND of context field/operator/value) intersected with the agent's `skills[]`, plus explicit /skill-name pinning; put routing intent in triggerConditions, and describe intent in description/instructions (the LLM does see those).", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. A runtime path does read it, but only to hand it back to API clients — a dead-end projection, not consumption." } } } diff --git a/packages/spec/liveness/tool.json b/packages/spec/liveness/tool.json index 2312bed0bd..b4a1a2f113 100644 --- a/packages/spec/liveness/tool.json +++ b/packages/spec/liveness/tool.json @@ -32,14 +32,18 @@ "note": "keys folded into description only; no validation exists." }, "category": { - "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ToolPreview.tsx:99", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "status": "dead", + "evidence": "only reader is a serializer pass-through in cloud service-ai/src/routes/tool-routes.ts:48 (GET /api/v1/ai/tools); no client consumes it, nothing groups/filters/routes by it", + "authorWarn": true, + "authorHint": "Display-only — no runtime groups, filters or routes tools by category.", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed." }, "requiresConfirmation": { - "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ToolPreview.tsx:101", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "status": "dead", + "evidence": "zero reads off any TOOL definition in any repo; every requiresConfirmation read is action.ai.requiresConfirmation (a different, live property). Ignored by the LLM tool set (vercel-adapter.ts:263-272), ToolRegistry.execute (tool-registry.ts:183-210), POST /ai/tools/:name/execute (tool-routes.ts:91,105) and the MCP bridge (framework packages/mcp/src/mcp-server-runtime.ts:176, which uses a hardcoded destructive-name list)", + "authorWarn": true, + "authorHint": "NOT ENFORCED — nothing pauses for confirmation on this flag, on any path. To gate a destructive operation, put it behind an action with `ai.requiresConfirmation` + the HITL approval queue, which is the only path that actually stops execution.", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. SAFETY-SHAPED: tool.form.ts:36-42 places it in a section titled 'Access & safety' with helpText 'Ask user to approve before executing (for destructive actions)', and skills/objectstack-ai/SKILL.md:547 tells authors to rely on it — so the authoring surface actively teaches a guarantee nothing delivers. ADR-0033 (:31, :60) already recorded it as never-enforced and resolved to DELETE the placeholder (the draft/publish workspace is the real approval gate); that deletion was never executed. Prune tracked separately." }, "permissions": { "status": "dead", @@ -48,14 +52,18 @@ "authorHint": "Not part of AIToolDefinition and no consumer — tool invocation is NOT permission-gated by this list. Gate the underlying action via requiredPermissions / permission sets (ADR-0066)." }, "active": { - "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ToolPreview.tsx:102", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "status": "dead", + "evidence": "AIToolDefinition (packages/spec/src/contracts/ai-service.ts:157-191) has no `active` field; ToolRegistry.getAll() returns everything (cloud tool-registry.ts:159-161) and selection is by name only — contrast agent.active (agent-runtime.ts:212,501,510) and skill.active (skill-registry.ts:93,110), which ARE enforced", + "authorWarn": true, + "authorHint": "`active: false` does NOT withdraw the tool — it still reaches the LLM tool set and POST /ai/tools/:name/execute still runs it (unlike agent.active / skill.active, which are enforced). To withdraw a tool, drop it from the skill/agent that references it.", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed. Also never set by any registration path." }, "builtIn": { - "status": "live", - "evidence": "objectui: packages/app-shell/src/views/metadata-admin/previews/ToolPreview.tsx:103", - "note": "LIVE via objectui metadata-admin authoring UI — the 2026-06 audit missed objectui; verified by reading the file." + "status": "dead", + "evidence": "not part of AIToolDefinition; whole-repo search across all three repos finds only a test assertion and the objectui preview pill — nothing branches on it", + "authorWarn": true, + "authorHint": "A display/authoring hint only — no runtime branches on it; it never affects registration, selection or execution.", + "note": "RE-VERIFIED 2026-07 (#3686 preview-claim sweep): the prior `live` verdict cited only a metadata-admin PREVIEW panel, which echoes what the author typed." } } }