Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .changeset/ai-surface-affinity-lint.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
"@objectstack/lint": minor
"@objectstack/spec": patch
---

feat(lint): `validate-ai-surface-affinity` — skill ↔ agent surface affinity is now linted (#3820)

An agent binds a product surface (`'ask'` | `'build'`, ADR-0063 §1) and a skill
declares which surface it belongs to (`'ask'` | `'build'` | `'both'`, §3). The
runtime refuses an incompatible binding with a **load error at chat time** —
after parse, validate, and deploy all passed cleanly. The new rule reports that
contradiction statically, and joins `REFERENCE_INTEGRITY_RULES`, so
`objectstack validate`, `lint`, and `compile` all pick it up with no CLI
changes.

Scope is deliberately narrow (zero false positives by construction): only
bindings where **both** the agent and the skill are declared in the same stack
are checked. `agent.skills[]` names that don't resolve in-stack (kernel skills
are runtime-registered and statically invisible) are skipped — resolving those
namespaces is #3820 D0/D2, decided by ADR-0109 (Proposed).

The spec side is doc-truth only, no schema shape changes:

- `stack.agents` is documented as **platform-internal** (ADR-0063 §2 — the
kernel ships exactly two agents; third parties extend via skills), replacing
prose that still described the withdrawn ADR-0040 per-app-copilot model.
- `stack.tools` is documented as declaration-only pending the ADR-0109 tool
authoring model.
- `app.defaultAgent` is re-documented as a surface-binding knob (`'ask'`
implicit / `'build'` for authoring surfaces), not a custom-agent slot.
- `SkillSchema` now states that a per-skill `permissions` field deliberately
does not exist (ADR-0049) — authoring one is silently stripped; access is
gated by `agent.access` / `agent.permissions` and per-tool authz.
2 changes: 1 addition & 1 deletion content/docs/references/ui/app.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ const result = ActionNavItem.parse(data);
| **sharing** | `{ enabled?: boolean; publicLink?: string; password?: string; allowedDomains?: string[]; … }` | optional | Public sharing configuration |
| **embed** | `{ enabled?: boolean; allowedOrigins?: string[]; width?: string; height?: string; … }` | optional | Iframe embedding configuration |
| **mobileNavigation** | `{ mode?: Enum<'drawer' \| 'bottom_nav' \| 'hamburger'>; bottomNavItems?: string[] }` | optional | Mobile-specific navigation configuration |
| **defaultAgent** | `string` | optional | Name of the default AI agent for this app (used by the ambient chat endpoint) |
| **defaultAgent** | `string` | optional | Platform agent bound to this app's ambient chat ('ask' is the implicit default; 'build' for authoring surfaces) — ADR-0063 §1 |
| **aria** | `{ ariaLabel?: string; ariaDescribedBy?: string; role?: string }` | optional | ARIA accessibility attributes for the application |
| **protection** | `{ lock: Enum<'none' \| 'no-overlay' \| 'no-delete' \| 'full'>; reason: string; docsUrl?: string }` | optional | Package author protection block — lock policy for this app. |
| **_lock** | `Enum<'none' \| 'no-overlay' \| 'no-delete' \| 'full'>` | optional | Item-level lock — controls overlay & delete (ADR-0010). |
Expand Down
38 changes: 38 additions & 0 deletions docs/adr/0109-ai-tool-authoring-model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# ADR-0109: The AI tool authoring model — third-party tools are bindings to executable primitives

**Status**: Proposed (2026-07-28) — the "D0" decision of issue #3820; blocks the `skill.tools` branch of the R7 reference-integrity rule.
**Deciders**: ObjectStack Protocol Architects
**Builds on**: [ADR-0063](./0063-two-kernel-agents-skills-are-the-extension-primitive.md) (skills + tools are the third-party extension primitive), [ADR-0064](./0064-tool-scoping-to-agent.md) (an agent's tools are its skills' tools), [ADR-0078](./0078-no-silently-inert-metadata.md) (no silently inert metadata)
**Consumers**: `@objectstack/spec` (`ai/tool.zod.ts`, `stack.zod.ts`), `@objectstack/lint` (the R7 rule), `../cloud/service-ai` (ToolRegistry), `../cloud/service-ai-studio`

---

## TL;DR

`stack.tools` today is a **declaration with no executable half and no reader**: `ToolSchema` carries no handler binding, the metadata `tool` type is written once at boot (Studio visibility mirror, `service-ai-studio/plugin.ts`) and read by **nothing**, `'tools'` is missing from `composeStacks`' `CONCAT_ARRAY_FIELDS`, and the executable tool set lives exclusively in the runtime-registered in-process `ToolRegistry`. ADR-0063 §2 says third parties extend via "skills **and tools**" — but only the skills half is real. That is the ADR-0078 prohibited state, and it makes any reference-integrity rule over `skill.tools[]` unbuildable: there is no registry of facts to resolve against (16/16 of HotCRM's skill→tool references miss `stack.tools`; 6 of those 16 nonetheless resolve — against the runtime registry lint cannot see).

Decision: **a third-party tool is a *binding* to an executable primitive the platform already has — an action or a flow — never a free-floating executable of its own.** Kernel/plugin tools stay runtime-registered, and their names become facts via a curated, conformance-tested registry in spec.

## Decision

1. **Third-party tools bind, they don't implement.** `ToolSchema` gains a required-for-app-packages `binding` (`{ type: 'action' | 'flow', name: string }`). The runtime materialises a callable from the bound primitive (the same mechanism that already materialises `action_<name>` tools from object action lists); `parameters` may narrow, but never widen, what the bound primitive accepts. Handlers, authz, and audit stay where they already are — on the action/flow. A tool record without a `binding` is legal only for runtime-registered (kernel/plugin) tools, whose record is a visibility mirror, not a definition.
2. **Platform tool names become facts, not guesses.** Spec exports a curated `PLATFORM_PROVIDED_TOOL_NAMES: ReadonlySet<string>` (the `PLATFORM_PROVIDED_OBJECT_NAMES` / #3657 precedent), kept honest by conformance tests in the *owning* packages (service-ai / service-ai-studio assert "every tool I register is listed; every listed name with my prefix is registered"). The lint → spec dependency direction is preserved.
3. **Reference integrity becomes decidable, and R7's tool branch unblocks.** `skill.tools[]` (wildcard-aware) resolves against: bound tools declared in-stack (→ checkable through to the action/flow, which existing rules already resolve) ∪ `PLATFORM_PROVIDED_TOOL_NAMES` ∪ wildcard families. Resolution ladder per ADR-0072: unknown name → **warning** at first (ADR-0078 ratchet), **error** once the registry has soaked one release.
4. **The inert surface is closed.** `'tools'` joins `CONCAT_ARRAY_FIELDS`; the boot-time mirror writes gain provenance so a stack-declared record cannot shadow a platform tool's mirror (today `if (exists) skip` lets it); the runtime's "silently dropped" posture for unresolved references stays (graceful degradation) but logs, because the *authoring-time* signal now comes from lint — loud at the producer, tolerant at the consumer (Prime Directive #12).

## Rejected alternative

**Tools as first-class executables** (a `handler` on ToolSchema): re-invents actions/flows inside the AI layer, splits authz/audit across two systems, and turns every tool into a second place to define business logic. The platform's ontology is already executable; the AI layer needs a *view* of it, not a rival.

## Consequences

- "Skills + tools" (ADR-0063 §2) becomes a shipped contract: a third party can author a skill whose tools are real, checkable, and governed by the same authz as the UI button that runs the same action.
- AI-authored metadata gets a closed vocabulary: a hallucinated tool name fails lint at draft time instead of shipping as a dead capability the copilot claims to have (the HotCRM failure, 10 fictional tools across 6 skills).
- Costs: a spec change (`binding`), the registry + conformance tests, a materialisation path in the runtime; the R7 tool-branch rule stays blocked until 1–2 land.

## Follow-up

- [ ] `binding` on `ToolSchema` + `'tools'` into `CONCAT_ARRAY_FIELDS` (spec).
- [ ] `PLATFORM_PROVIDED_TOOL_NAMES` + owning-package conformance tests (spec, cloud).
- [ ] Materialise bound tools in the ToolRegistry; provenance-guard the boot mirror (cloud).
- [ ] R7 tool branch: `validate-ai-references` skill→tool resolution, warning-first (lint; #3820).
9 changes: 9 additions & 0 deletions packages/lint/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,15 @@ export type {
TranslationRefSeverity,
} from './validate-translation-references.js';

export {
validateAiSurfaceAffinity,
AI_SKILL_SURFACE_MISMATCH,
} from './validate-ai-surface-affinity.js';
export type {
AiSurfaceAffinityFinding,
AiSurfaceAffinitySeverity,
} from './validate-ai-surface-affinity.js';

// One entry point for the reference-resolution rules above (#3583 §5 D5).
// Adding a rule to `REFERENCE_INTEGRITY_RULES` runs it on `validate`, `lint`
// and `compile` at once — the CLI call sites do not change.
Expand Down
10 changes: 8 additions & 2 deletions packages/lint/src/reference-integrity-suite.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ describe('reference-integrity suite — membership', () => {
'validateNavAccess',
'validateTranslationReferences',
'validateFlowTemplatePaths',
'validateAiSurfaceAffinity',
]);
});

Expand Down Expand Up @@ -107,6 +108,10 @@ describe('reference-integrity suite — every member actually runs', () => {
// validateTranslationReferences: a field the object does not declare.
{ en: { objects: { crm_lead: { label: 'Lead', fields: { assigned_to: { label: 'Owner' } } } } } },
],
// validateAiSurfaceAffinity: an 'ask' agent binding a 'build' skill — the
// runtime throws on this at chat time (ADR-0064 §3).
agents: [{ name: 'helper', surface: 'ask', skills: ['metadata_authoring'] }],
skills: [{ name: 'metadata_authoring', surface: 'build', tools: [] }],
flows: [
{
name: 'lead_followup',
Expand Down Expand Up @@ -139,6 +144,7 @@ describe('reference-integrity suite — every member actually runs', () => {
expect(rules).toContain('nav-object-ungranted');
expect(rules).toContain('translation-target-unknown');
expect(rules).toContain('flow-template-unknown-field');
expect(rules).toContain('ai-skill-surface-mismatch');
});

it('carries a gating flow-template finding through the suite (#3810)', () => {
Expand All @@ -160,9 +166,9 @@ describe('reference-integrity suite — every member actually runs', () => {
expect(typeof f.message).toBe('string');
expect(typeof f.hint).toBe('string');
}
// Object references run first, flow template paths last.
// Object references run first, AI surface affinity last.
expect(findings[0].rule).toBe('object-reference-unknown');
expect(findings[findings.length - 1].rule).toBe('flow-template-unknown-field');
expect(findings[findings.length - 1].rule).toBe('ai-skill-surface-mismatch');
});

it('returns nothing for an empty stack', () => {
Expand Down
2 changes: 2 additions & 0 deletions packages/lint/src/reference-integrity-suite.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ import { validateChartBindings } from './validate-chart-bindings.js';
import { validateNavAccess } from './validate-nav-access.js';
import { validateTranslationReferences } from './validate-translation-references.js';
import { validateFlowTemplatePaths } from './validate-flow-template-paths.js';
import { validateAiSurfaceAffinity } from './validate-ai-surface-affinity.js';

export type ReferenceIntegritySeverity = 'error' | 'warning';

Expand Down Expand Up @@ -94,6 +95,7 @@ export const REFERENCE_INTEGRITY_RULES: readonly ReferenceIntegrityRule[] = [
{ name: 'validateNavAccess', run: validateNavAccess },
{ name: 'validateTranslationReferences', run: validateTranslationReferences },
{ name: 'validateFlowTemplatePaths', run: validateFlowTemplatePaths },
{ name: 'validateAiSurfaceAffinity', run: validateAiSurfaceAffinity },
];

/**
Expand Down
139 changes: 139 additions & 0 deletions packages/lint/src/validate-ai-surface-affinity.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.

import { describe, it, expect } from 'vitest';
import {
validateAiSurfaceAffinity,
AI_SKILL_SURFACE_MISMATCH,
} from './validate-ai-surface-affinity.js';

describe('validate-ai-surface-affinity', () => {
// ── The false-positive floor, read per branch (#3806 lesson) ──────────────
// `examples/` declares zero agents/skills, so it proves nothing about this
// rule. The floor is instead the real shipped corpus the issue measured:
// HotCRM's two agents × six skills, none of which declares a `surface`
// (raw config — no Zod defaults applied). Every binding must pass.
it('is clean on the HotCRM-shaped corpus (no surface fields anywhere)', () => {
const stack = {
agents: [
{
name: 'sales_copilot',
skills: [
'live_data',
'lead_qualification',
'email_drafting',
'revenue_forecasting',
'customer_360',
],
},
{
name: 'service_copilot',
skills: ['case_triage', 'customer_360', 'email_drafting'],
},
],
skills: [
{ name: 'live_data', tools: ['describe_object'] },
{ name: 'lead_qualification', tools: ['analyze_lead'] },
{ name: 'email_drafting', tools: ['generate_email'] },
{ name: 'revenue_forecasting', tools: ['forecast_revenue'] },
{ name: 'customer_360', tools: ['search_knowledge'] },
{ name: 'case_triage', tools: ['triage_case'] },
],
};
expect(validateAiSurfaceAffinity(stack)).toEqual([]);
});

it("flags an 'ask' agent referencing a 'build' skill", () => {
const stack = {
agents: [{ name: 'helper', surface: 'ask', skills: ['metadata_authoring'] }],
skills: [{ name: 'metadata_authoring', surface: 'build', tools: [] }],
};
const findings = validateAiSurfaceAffinity(stack);
expect(findings).toHaveLength(1);
expect(findings[0]).toMatchObject({
severity: 'error',
rule: AI_SKILL_SURFACE_MISMATCH,
where: 'agent "helper" · skills',
path: 'agents[0].skills[0]',
});
expect(findings[0].message).toContain("'ask'");
expect(findings[0].message).toContain("'build'");
});

it("flags a 'build' agent referencing an 'ask' skill (symmetric)", () => {
const stack = {
agents: [{ name: 'builder', surface: 'build', skills: ['data_explorer'] }],
skills: [{ name: 'data_explorer', surface: 'ask', tools: [] }],
};
const findings = validateAiSurfaceAffinity(stack);
expect(findings).toHaveLength(1);
expect(findings[0].path).toBe('agents[0].skills[0]');
});

it("accepts surface: 'both' on either agent surface", () => {
const stack = {
agents: [
{ name: 'asker', surface: 'ask', skills: ['schema_tools'] },
{ name: 'builder', surface: 'build', skills: ['schema_tools'] },
],
skills: [{ name: 'schema_tools', surface: 'both', tools: [] }],
};
expect(validateAiSurfaceAffinity(stack)).toEqual([]);
});

it("defaults an absent surface to 'ask' on BOTH sides (mirrors the runtime)", () => {
// Agent without surface (→ 'ask') referencing a build skill: must flag.
const stack = {
agents: [{ name: 'helper', skills: ['metadata_authoring'] }],
skills: [{ name: 'metadata_authoring', surface: 'build', tools: [] }],
};
expect(validateAiSurfaceAffinity(stack)).toHaveLength(1);

// Skill without surface (→ 'ask') referenced by a build agent: must flag.
const inverse = {
agents: [{ name: 'builder', surface: 'build', skills: ['plain'] }],
skills: [{ name: 'plain', tools: [] }],
};
expect(validateAiSurfaceAffinity(inverse)).toHaveLength(1);
});

it('skips references that do not resolve in-stack (kernel/runtime-registered skills)', () => {
// `ask`/`build` kernel skills (schema_reader, data_explorer, …) register at
// boot and are statically invisible — resolving them against `stack.skills`
// alone is #3820 D0/D2 territory, not this rule's.
const stack = {
agents: [{ name: 'helper', surface: 'ask', skills: ['schema_reader', 'data_explorer'] }],
skills: [],
};
expect(validateAiSurfaceAffinity(stack)).toEqual([]);
});

it('reports every mismatched binding, with stable paths', () => {
const stack = {
agents: [
{
name: 'helper',
surface: 'ask',
skills: ['ok_skill', 'authoring', 'unresolved', 'design'],
},
],
skills: [
{ name: 'ok_skill', surface: 'ask', tools: [] },
{ name: 'authoring', surface: 'build', tools: [] },
{ name: 'design', surface: 'build', tools: [] },
],
};
const findings = validateAiSurfaceAffinity(stack);
expect(findings.map((f) => f.path)).toEqual(['agents[0].skills[1]', 'agents[0].skills[3]']);
});

it('tolerates junk shapes without throwing', () => {
expect(validateAiSurfaceAffinity({})).toEqual([]);
expect(validateAiSurfaceAffinity({ agents: 'nope', skills: 42 } as never)).toEqual([]);
expect(
validateAiSurfaceAffinity({
agents: [null, { name: 'x', skills: [null, 7, 'y'] }],
skills: [null, { surface: 'build' }, { name: 'y', surface: 'build' }],
} as never),
).toHaveLength(1);
});
});
Loading
Loading