Skip to content

docs(harness): teach the LLM call-surface rule in the system prompt - #680

Closed
gwitwer wants to merge 2 commits into
mainfrom
docs/sap-2775-harness-llm-usage-rule
Closed

docs(harness): teach the LLM call-surface rule in the system prompt#680
gwitwer wants to merge 2 commits into
mainfrom
docs/sap-2775-harness-llm-usage-rule

Conversation

@gwitwer

@gwitwer gwitwer commented Aug 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Teaches the LLM call-surface rule in the Studio harness's DEFAULT_SYSTEM_PROMPT (packages/harness/src/profiles/default.ts) — the embedded coding agent that writes Sapiom agents is the highest-reach mis-selection source, since it's the one actually authoring the step code that would misuse a one-shot LLM call as an agent loop (or vice versa). This is the token-budgeted sibling of the fuller MCP-instructions section already shipped in #679: same rule, ~8 lines instead of the full section, because this prompt is injected fresh every session via --append-system-prompt and is a live cost on every turn.

Content shipped

**Calling LLMs from agent code:** one-shot call → `ctx.sapiom.llm.run`; a
platform-driven multi-turn loop → `ctx.sapiom.models.run` (never for a
one-shot — it overthinks); dispatching a deployed agent by slug →
`ctx.sapiom.agents.run`. Structured output = tool-use/schema output, read
only `type === 'text'` blocks — never "reply with only JSON". Omit `model`
(recommended) or pin the `smart` label; raw provider ids are never
honored. Results disclose the served class + lane. Debugging a run: the
Run Inspector, or the per-step I/O endpoint documented below.
Guide: https://docs.sapiom.ai/guides/choose-a-call-surface.

Placed right after the existing "The two MCPs" block, anchored near the pre-existing "sapiom (remote, HTTP) ... models" mention (default.ts:15-18) — corrected that bullet's vague "models" list item to "LLM calls (see below)" pointing at the new block, so it no longer reads as a single undifferentiated capability.

Doc-link convention

This prompt had no existing docs.sapiom.ai citation to follow — checked, there isn't one. Introduced the first one here (https://docs.sapiom.ai/guides/choose-a-call-surface), matching the terse inline-link style the rest of the prompt already uses for other conventions (e.g. .sapiom/harness-context.json), rather than inventing a new citation format.

Changes

  • packages/harness/src/profiles/default.ts — new "Calling LLMs from agent code" block; corrected the adjacent MCP-capability bullet.
  • packages/harness/src/core/inject/system-prompt.test.ts — extended the existing content-guard assertions (this file's established pattern — it already asserts specific phrases like "The Canvas follows that selection") to cover the new section's key phrases.

Testing

cd packages/harness
npx vitest run src/core/inject/system-prompt.test.ts

Result: 4 passed (unchanged count — added assertions to an existing test rather than a new one).

Full package sanity check (needed pnpm build at the repo root first — packages/harness transitively depends on several unbuilt workspace packages, e.g. @sapiom/analytics-core, @sapiom/mcp, @sapiom/agent-core, in a fresh worktree):

pnpm build   # from repo root
cd packages/harness && npx vitest run

Result: 141 test files passed, 2095 tests passed. eslint clean on both touched files.

Correction (round 2) — internal service naming

agent-core's scaffold terminology guard (scaffold.test.ts, surfaced while working #683) bans internal service naming from customer-facing content — this prompt isn't shipped through that same guard, but the ruling was to apply the same fix everywhere in the teaching stack for consistency. This file's terse debugging line never spelled out a literal internal endpoint path to begin with, but reworded it to match the other surfaces' "see the guide" pointer: "Debugging a run: the Run Inspector, or the per-step I/O endpoint documented below." Re-ran npx vitest run src/core/inject/system-prompt.test.ts — still 4 passed.

Related

Refs: internal tracker
Companion (fuller) MCP-instructions version: #679

Checklist

  • Section kept to ~8 lines (token-budget rationale above)
  • Consistent with the docs(mcp): teach the LLM call-surface rule in the authoring instructions #679 rule content (same facts, terser form)
  • Adjacent "models" mention corrected rather than left misleading
  • Test coverage extended following this file's existing content-guard pattern
  • Full package build + test suite green
  • Round-2 correction: debugging line reworded for internal-naming consistency

🤖 Generated with Claude Code

https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

@gwitwer @claude
docs(harness): teach the LLM call-surface rule in the system prompt
e6406f9 
The #1 mis-selection source is the embedded coding agent itself picking the
wrong LLM call surface while writing a Sapiom agent's step code — this is
the highest-reach fix among the SAP-2775 surfaces, since it's the one
already reading the codebase. Adds a compact "Calling LLMs from agent code"
block: ctx.sapiom.llm.run (one-shot) vs ctx.sapiom.models.run (multi-turn
loop — never for a one-shot, it overthinks) vs ctx.sapiom.agents.run
(dispatch a deployed agent); structured output via tool-use/schema output,
read only type==='text' blocks, never string-parse JSON; omit `model`
(recommended) or pin the `smart` label — raw provider ids are never
honored; results disclose the served class + lane; and the per-step /io
endpoint / Run Inspector for debugging. Kept to ~8 lines (the prompt is
token-budgeted every session) versus the fuller MCP-instructions version
(sapiom-js#679) — same rule, terser form.

Corrected the existing "sapiom (remote, HTTP)" bullet's "models" mention to
point at the new block instead of naming a single vague capability, and
introduced this package's first docs.sapiom.ai citation (no prior
convention existed here) pointing at the canonical guide page.

Extended system-prompt.test.ts's existing content-guard assertions (the
established pattern in this file) to cover the new section's key phrases.

Refs: SAP-2775

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc
@gwitwer gwitwer mentioned this pull request Aug 23, 2026
Closed
4 tasks
@gwitwer @claude
fix(harness): point the debugging line at the guide instead of a lite…
fe8fa96 
…ral path

Ratified across the teaching stack (sapiom-js#683's scaffold terminology
guard bans workflow(s)/orchestration(s) from customer-facing content): the
per-step debugging endpoint's literal path lives in the canonical guide
(docs-internal#133) only. This prompt's terse debugging line never spelled
out the literal `/v1/workflows/...` path to begin with, but reworded it for
consistency with the other teaching surfaces' "see the guide" pointer.

Refs: SAP-2775, SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc
@gwitwer gwitwer mentioned this pull request Aug 23, 2026
Merged
@gwitwer

gwitwer commented Aug 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Consolidated into #685 — content unchanged plus the terminology scrub. Closing this in favor of that.

@gwitwer gwitwer closed this Aug 23, 2026
@sapiom sapiom deleted a comment from github-actions Bot Aug 23, 2026
@sapiom sapiom deleted a comment from github-actions Bot Aug 23, 2026
gwitwer added a commit that referenced this pull request Aug 23, 2026
@gwitwer @claude
feat(tools,mcp,harness,skills): teach the LLM call-surface rule acros…
5a8eeea 
…s all authoring surfaces (SAP-2764/2775/2776) (#685)

* docs(mcp): teach the LLM call-surface rule in the authoring instructions

Add the same "LLM calls & agent loops" section shipped in the companion
Sapiom-repo PR: ctx.sapiom.llm.run (one-shot, read only type==='text' blocks
— never string-parse JSON out of a response that may carry a thinking
block) vs ctx.sapiom.models.run / models.coding.run (platform-driven
multi-turn loops) vs ctx.sapiom.agents.run (dispatch a deployed agent by
slug); the "you never pick a model" deadlineMinutes/class-label contract;
and the step-io endpoint for debugging a run. Highest-reach usability fix
in the project — this is the bundled offline fallback AUTHORING_INSTRUCTIONS
served when the live @sapiom/mcp startup fetch to the backend fails.

The new section is byte-identical to the backend's DEFAULT_MCP_INSTRUCTIONS
copy of it (checksum-verified). Note: the two files as a WHOLE were already
not byte-identical before this change for unrelated, pre-existing reasons
(this file's title, its older "Two ways to use Sapiom" aliasing section, and
its ctx.shared 256 KiB quota paragraph from SAP-2790/#677 are absent from or
worded differently than the backend copy) — out of scope here; flagged
separately.

Refs: SAP-2775
Companion Sapiom-repo PR: sapiom/Sapiom#4519

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(mcp): correct the model-pinning claim in the LLM-usage-rule section

The "You may pin a class label (small / medium / large / smart)" sentence
was wrong against shipped behavior: today's AGENTS_LLM_V2_KNOWN_LABELS
allowlist is smart, minimax-m3, m2.7, opus, haiku, sonnet, m3-test —
small/medium/large are the SKU spec's future vocabulary and would
warn-and-default today, not pin. Replaced with present-truth guidance:
omit `model` entirely (recommended), or pin the one label that's
contract-attested to work (`smart`) — raw provider model ids are never
honored. Kept byte-identical to the companion Sapiom-repo fix (checksum
33ced1cf7be83ab72ced6e8837ba585131710bb553c064dcde64d68288011314 for the
shared section in both).

Refs: SAP-2775
Companion Sapiom-repo PR: sapiom/Sapiom#4519

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* docs(harness): teach the LLM call-surface rule in the system prompt

The #1 mis-selection source is the embedded coding agent itself picking the
wrong LLM call surface while writing a Sapiom agent's step code — this is
the highest-reach fix among the SAP-2775 surfaces, since it's the one
already reading the codebase. Adds a compact "Calling LLMs from agent code"
block: ctx.sapiom.llm.run (one-shot) vs ctx.sapiom.models.run (multi-turn
loop — never for a one-shot, it overthinks) vs ctx.sapiom.agents.run
(dispatch a deployed agent); structured output via tool-use/schema output,
read only type==='text' blocks, never string-parse JSON; omit `model`
(recommended) or pin the `smart` label — raw provider ids are never
honored; results disclose the served class + lane; and the per-step /io
endpoint / Run Inspector for debugging. Kept to ~8 lines (the prompt is
token-budgeted every session) versus the fuller MCP-instructions version
(sapiom-js#679) — same rule, terser form.

Corrected the existing "sapiom (remote, HTTP)" bullet's "models" mention to
point at the new block instead of naming a single vague capability, and
introduced this package's first docs.sapiom.ai citation (no prior
convention existed here) pointing at the canonical guide page.

Extended system-prompt.test.ts's existing content-guard assertions (the
established pattern in this file) to cover the new section's key phrases.

Refs: SAP-2775

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* feat(tools): served class/lane disclosure, structured output, label ergonomics (SAP-2764)

Reissues #673 under the corrected contract, plus the strong-defaults
ergonomics items. Additive only — no published type or behavior changes
for existing consumers.

- models.run (ModelRunOutcome) and models.coding.run (CodingRunOutcome):
  optional servedClass/lane (wire served_class/lane) — the billing class
  the run's label resolved to and the lane it executed in. Verified fresh
  against origin/dev (sapiom/Sapiom@b40e580c3): both wire shapes carry
  these fields (coding always null today, no cost_usd key on coding at
  all — it never had a real one to preserve). Left ModelRunOutcome.costUsd
  untouched (still number, non-nullable) per the additive-only mandate;
  the server's own stale-#4482-row null guard is a narrow, pre-existing
  edge case this PR does not attempt to fix in the type.
- llm.run/redeem/callSession: LlmDisclosure (wire shape) + readDisclosure()
  (camelCased LlmDisclosureResult), mirroring #673's reviewed design minus
  the still-contested `degradation` reservation (out of scope here).
- llm.run gains an optional `output: { name, schema }` — the blessed
  tool-calling pattern for structured output, automated (appends a forced
  tool + tool_choice). run()'s return type is unchanged either way; read
  the parsed value with the new structuredOf(). New textOf() reads the
  plain-text reply, skipping a `thinking` block that may precede it.
- model/label fields across llm.run, llm.submit, llm.createSession,
  models.run, and models.coding.run: soft-union type ("smart" |
  (string & Record<never, never>), the lint-safe spelling already used by
  content-generation's LiteralUnion) for autocomplete, with JSDoc settled
  on "routing label" terminology — omit-recommended, "smart" if pinning,
  raw provider ids never honored.

Refs: SAP-2764

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* docs(skills): teach the LLM call-surface rule + naming conventions

Adds the LLM call-surface rule to the canonical sapiom-agent-authoring
skill — the highest-reach teaching surface of the whole SAP-2775/2776 stack,
since it ships into every scaffolded agent project: ctx.sapiom.llm.run
(one-shot) vs ctx.sapiom.models.run (platform-driven multi-turn loop, never
for a one-shot) vs ctx.sapiom.agents.run (dispatch a deployed agent by
slug), a worked "wrong vs right" example against the "reply with only
JSON" + string-parsing mistake (forced tool call + explicit content-block
`type` filtering instead), the omit-or-pin-`smart`-label rule, and a
debugging pointer. Also adds a "Naming Conventions" section settling the
platform's overloaded agent/run/task/session/dispatch vocabulary and
"label" as the author-facing term for a model value, with a rule that new
capabilities must not add a sixth meaning to an already-overloaded word.

Synced identically across all four copies skill-sync.test.ts guards: the
canonical source, both scaffold templates (default, coding-pause), and the
Claude Code plugin copy — verified with `npx jest skill-sync` (7 passed).

Folds in the models/index.ts:414-421 stale-comment fix flagged on SAP-2776:
the "Default agent ... `agent.run` / `agent.launch`" header (and two
adjacent comments in the same block) named a namespace that's actually
exported as `models` — corrected throughout, including the top-of-file
module docstring's own self-description and code example, which had the
same problem in an even more prominent spot.

Caught by the scaffold's own terminology guard (scaffold.test.ts's
"copies the %s template with exact Agent terminology" — bans
workflow(s)/orchestration(s) from anything shipped into a scaffold): an
earlier draft of the Naming Conventions "agent" row explained the internal
vs. customer-facing naming split by naming the internal term directly, and
the debugging pointer cited the literal `/v1/workflows/...` REST path —
both rephrased to convey the same information without the banned words.
Flagging in the PR body that this same constraint may affect four earlier,
already-open PRs in this teaching stack that used the identical debugging
pointer text, none of which have this guard test.

Refs: SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(mcp): drop internal `workflows`-service naming from the LLM-usage-rule section

sapiom-js's own scaffold terminology guard (agent-core's scaffold.test.ts)
bans workflow(s)/orchestration(s) from anything shipped to a customer — the
debugging line's literal `GET /v1/workflows/executions/:id/steps/:stepId/io`
path, and the "one monolithic workflow" phrase, both leak that internal
service naming. Ruling: the literal path's one deliberate home is the
canonical guide (docs-internal#133); every other teaching surface points
there instead of repeating it. Replaced both phrases; kept the shared
section byte-identical with the Sapiom-repo backend copy (re-verified
checksum). Updated the drift-guard test's assertion to match.

Refs: SAP-2775, SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(harness): point the debugging line at the guide instead of a literal path

Ratified across the teaching stack (sapiom-js#683's scaffold terminology
guard bans workflow(s)/orchestration(s) from customer-facing content): the
per-step debugging endpoint's literal path lives in the canonical guide
(docs-internal#133) only. This prompt's terse debugging line never spelled
out the literal `/v1/workflows/...` path to begin with, but reworded it for
consistency with the other teaching surfaces' "see the guide" pointer.

Refs: SAP-2775, SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* chore(changeset): add missing changesets for the mcp + harness prompt content

#679/#680 (now consolidated here) touched @sapiom/mcp's AUTHORING_INSTRUCTIONS
fallback and @sapiom/harness's DEFAULT_SYSTEM_PROMPT without a changeset —
both are user-facing content changes to changeset-tracked packages, and this
repo's convention (checked prior harness/mcp commits) is one changeset per
such change. Added both; left the pre-existing @sapiom/tools (#682) and
@sapiom/agent-core + @sapiom/tools (#683) changesets as separate, non-
overlapping entries per package.

Refs: SAP-2764, SAP-2775, SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(skills,mcp,changeset): sanitize internal service naming from shipped content

This repo is public. A review found internal naming that had made it into
shipped/publishable content, beyond the earlier workflow/orchestration scrub:

- sapiom-agent-authoring/SKILL.md's Naming Conventions table named an
  internal service by its internal name ("agent-runs") and an internal
  design-doc label ("Surface B") — reworded to describe the capability by
  what it does, not its internal name. Also dropped a dangling reference to
  a "writing-agents" skill and a PROMPT.md path that don't exist in this
  repo (they're specific to a different, private repo) — a broken pointer
  shipping to every scaffold, found while fixing the naming leak in the
  same table cell.
- packages/mcp/src/instructions.ts's module doc comment named a private
  companion repo's file path directly; TypeScript's declaration emit
  preserves this comment on the exported symbol, so it would ship in the
  published .d.ts. Genericized to "a private companion repo" (pre-existing,
  predates this branch's other changes).
- The disclosure changeset (ships permanently in CHANGELOG.md on release)
  named an internal ticket id — trimmed.

Synced the skill fix across all four copies skill-sync.test.ts guards;
re-verified checksums and re-ran the affected test suites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(sap-2775): resolve review-round findings on the LLM call-surface teaching stack

Fixes five content bugs raised across two bot-review rounds on this PR, plus
two disclosure wording trims:

- Structured-output teaching was self-contradictory (forced tool-use output
  has no text block; the result lives in tool_use). Fixed in the MCP
  fallback instructions, the harness system prompt, and the skill's worked
  example, with new tests pinning the "tool_use" wording so this can't
  regress silently again.
- structuredOf/textOf/readDisclosure are now reachable from ctx.sapiom.llm
  (client.ts type + runtime, plus the run_local stub), additive only, with
  new tests exercising all three through the client.
- Removed the skill's "is landing" blockquote for a convenience that ships
  in this same PR.
- Fixed the "llm.submit/redeem is a multi-turn surface" mistake — it's a
  single call with deferred capacity.
- Reverted LlmSessionCreateSpec.model to string (it pins an exact alias,
  mutually exclusive with the real label field).
- Trimmed two Naming Conventions rows per a documentation-disclosure ruling.

Skill edits synced across all four copies (skill-sync guard).

* fix(sap-2775): round-4 review fixes — prompt wording, example, naming clarity

- Harness prompt's debugging pointer said "documented below" with nothing
  below documenting it (the next line is the guide link). Changed to
  "documented in the guide", matching the MCP fallback copy's wording.
  Content-guard test now pins the corrected phrase and forbids the old one.
- Skill's "Right" worked example passed `model: "smart"` in the mainline
  path while the surrounding text recommends omitting `model`. Example now
  omits it, with pinning shown as a one-line variant comment.
- Naming Conventions "label" row clarified: a result's `servedClass` field
  is a disclosure field reporting the resolved billing class, not a
  contradiction of "never call a label a class" (author-facing input vs.
  server-reported output are different axes).

Synced across all four SKILL.md copies (skill-sync + MD5 verified).

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gwitwer