feat(tools): expose servedModel / costUsd disclosure on execution results (SAP-2764) - #673
feat(tools): expose servedModel / costUsd disclosure on execution results (SAP-2764)#673gwitwer wants to merge 4 commits into
Conversation
…ecution results Additive, optional-safe result fields per the DisclosureFields contract (Sapiom plans/model-execution-surface/interfaces.md): - models.run: ModelRunOutcome.servedModel (wire served_model) — the deployment that actually served, distinct from modelUsed which stays the requested label; costUsd doc'd as the served-priced real cost; degradation reserved (SAP-2768). - models.coding.run: CodingRunOutcome gains servedModel/costUsd (nullable — coding cannot observe them yet) + reserved degradation. - llm.run / llm.redeem / llm.callSession: LlmDisclosure type + readDisclosure() helper for the served_model/cost_usd fields the gateway injects into raw /v2 non-streaming bodies (camelCased, old-server safe). All fields optional/nullable: existing consumers compile and run unchanged; missing wire fields map to null (unknown) — never fabricated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9
…et, old-server coverage - Published JSDoc no longer names a serving vendor or internal deployment codename (opaque placeholder in specs), and no longer points at a private-repo planning doc or internal ticket ids — fields are described by their wire semantics, attributed to the server. - readDisclosure now returns the named, exported LlmDisclosureResult (camelCase); LlmDisclosure stays exported as the documented raw wire shape (matching the snake_case bodies llm.run returns). - Added the missing minor changeset for @sapiom/tools, including the changed documented meaning of the existing ModelRunOutcome.costUsd. - Old-server compatibility is now tested on both mappers: models.run maps absent disclosure fields to null, and a new coding-result spec covers null/absent/present disclosure on models.coding.run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9
…; forbid 'fireworks' in copy check - degradation stays reserved (required by the driving ticket) but is now typed `unknown` on all three public types: consumers must narrow before reading, so the future concrete type is purely additive — addresses the review's narrowing-is-breaking concern without un-shipping the reservation. - Changeset wording now says exactly which fields map to null on old servers vs stay absent. - provider-neutral-copy-check gains 'fireworks' so the round-1 leak class cannot regress silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9
…4 contract correction
The deliverable changed (Dave, 2026-08-21): disclosing the raw served model
contradicts the SKU contract and SAP-1891, and a provider-priced cost must
never reach a customer. The disclosure vocabulary is size × lane.
- ModelRunOutcome/CodingRunOutcome: servedModel → servedClass + lane (wire
served_class/lane); costUsd deprecated + nullable (servers now report null —
the metering pipeline owns customer dollars; old servers' numbers still map).
- LlmDisclosure/readDisclosure: served_class + lane (LlmDisclosureResult is
{ servedClass, lane }); streams carry the same data as
x-sapiom-served-class / x-sapiom-lane headers.
- degradation reservation unchanged (unknown-typed).
- Changeset rewritten to match, including the costUsd number → number|null
widening.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9
|
Reworked in ca7c37b to the corrected disclosure contract (see the server-side discussion): the disclosure vocabulary is now billing class × lane — |
|
Closing — the disclosure contract changed: customer results carry served class + lane + token usage, never the provider model or provider-priced cost (per the platform's SKU contract). These |
|
Addressed in 7c4ef1c — point-by-point: 🔒 Vendor/codename in shipped JSDoc — Fixed. The JSDoc example is gone (field described by its semantics: "server-assigned identifier of the deployment"), and all spec fixtures use the opaque 🔒 Private-repo pointer / internal-tracker link — Fixed. The contract path and ticket ids are out of the published JSDoc (fields are documented by their wire semantics instead), and the PR body no longer carries the internal tracker link or the private companion-branch description. Missing changeset — Fixed. Added a minor changeset for Cut JSDoc asserts backend behavior — Fixed. All disclosure docs now state what the wire field is and attribute interpretation to the server ("as reported by the server", "server-assigned identifier"), with no client-side guarantee implied, and no internal ticket ids in published docs.
Old-server path untested — Fixed. |
|
Round 2 addressed in f316c8a:
Changeset precision — Fixed: it now says exactly which fields map to Checker gap — Added the deployment vendor's name from the round-1 finding to the provider-neutral copy checker's forbidden list, so that leak class can't regress silently; check passes on the tree. Test execution — For the record since the sandbox blocked jest for you: |
…s all authoring surfaces (SAP-2764/2775/2776) (#685) * docs(mcp): teach the LLM call-surface rule in the authoring instructions Add the same "LLM calls & agent loops" section shipped in the companion Sapiom-repo PR: ctx.sapiom.llm.run (one-shot, read only type==='text' blocks — never string-parse JSON out of a response that may carry a thinking block) vs ctx.sapiom.models.run / models.coding.run (platform-driven multi-turn loops) vs ctx.sapiom.agents.run (dispatch a deployed agent by slug); the "you never pick a model" deadlineMinutes/class-label contract; and the step-io endpoint for debugging a run. Highest-reach usability fix in the project — this is the bundled offline fallback AUTHORING_INSTRUCTIONS served when the live @sapiom/mcp startup fetch to the backend fails. The new section is byte-identical to the backend's DEFAULT_MCP_INSTRUCTIONS copy of it (checksum-verified). Note: the two files as a WHOLE were already not byte-identical before this change for unrelated, pre-existing reasons (this file's title, its older "Two ways to use Sapiom" aliasing section, and its ctx.shared 256 KiB quota paragraph from SAP-2790/#677 are absent from or worded differently than the backend copy) — out of scope here; flagged separately. Refs: SAP-2775 Companion Sapiom-repo PR: sapiom/Sapiom#4519 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * fix(mcp): correct the model-pinning claim in the LLM-usage-rule section The "You may pin a class label (small / medium / large / smart)" sentence was wrong against shipped behavior: today's AGENTS_LLM_V2_KNOWN_LABELS allowlist is smart, minimax-m3, m2.7, opus, haiku, sonnet, m3-test — small/medium/large are the SKU spec's future vocabulary and would warn-and-default today, not pin. Replaced with present-truth guidance: omit `model` entirely (recommended), or pin the one label that's contract-attested to work (`smart`) — raw provider model ids are never honored. Kept byte-identical to the companion Sapiom-repo fix (checksum 33ced1cf7be83ab72ced6e8837ba585131710bb553c064dcde64d68288011314 for the shared section in both). Refs: SAP-2775 Companion Sapiom-repo PR: sapiom/Sapiom#4519 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * docs(harness): teach the LLM call-surface rule in the system prompt The #1 mis-selection source is the embedded coding agent itself picking the wrong LLM call surface while writing a Sapiom agent's step code — this is the highest-reach fix among the SAP-2775 surfaces, since it's the one already reading the codebase. Adds a compact "Calling LLMs from agent code" block: ctx.sapiom.llm.run (one-shot) vs ctx.sapiom.models.run (multi-turn loop — never for a one-shot, it overthinks) vs ctx.sapiom.agents.run (dispatch a deployed agent); structured output via tool-use/schema output, read only type==='text' blocks, never string-parse JSON; omit `model` (recommended) or pin the `smart` label — raw provider ids are never honored; results disclose the served class + lane; and the per-step /io endpoint / Run Inspector for debugging. Kept to ~8 lines (the prompt is token-budgeted every session) versus the fuller MCP-instructions version (sapiom-js#679) — same rule, terser form. Corrected the existing "sapiom (remote, HTTP)" bullet's "models" mention to point at the new block instead of naming a single vague capability, and introduced this package's first docs.sapiom.ai citation (no prior convention existed here) pointing at the canonical guide page. Extended system-prompt.test.ts's existing content-guard assertions (the established pattern in this file) to cover the new section's key phrases. Refs: SAP-2775 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * feat(tools): served class/lane disclosure, structured output, label ergonomics (SAP-2764) Reissues #673 under the corrected contract, plus the strong-defaults ergonomics items. Additive only — no published type or behavior changes for existing consumers. - models.run (ModelRunOutcome) and models.coding.run (CodingRunOutcome): optional servedClass/lane (wire served_class/lane) — the billing class the run's label resolved to and the lane it executed in. Verified fresh against origin/dev (sapiom/Sapiom@b40e580c3): both wire shapes carry these fields (coding always null today, no cost_usd key on coding at all — it never had a real one to preserve). Left ModelRunOutcome.costUsd untouched (still number, non-nullable) per the additive-only mandate; the server's own stale-#4482-row null guard is a narrow, pre-existing edge case this PR does not attempt to fix in the type. - llm.run/redeem/callSession: LlmDisclosure (wire shape) + readDisclosure() (camelCased LlmDisclosureResult), mirroring #673's reviewed design minus the still-contested `degradation` reservation (out of scope here). - llm.run gains an optional `output: { name, schema }` — the blessed tool-calling pattern for structured output, automated (appends a forced tool + tool_choice). run()'s return type is unchanged either way; read the parsed value with the new structuredOf(). New textOf() reads the plain-text reply, skipping a `thinking` block that may precede it. - model/label fields across llm.run, llm.submit, llm.createSession, models.run, and models.coding.run: soft-union type ("smart" | (string & Record<never, never>), the lint-safe spelling already used by content-generation's LiteralUnion) for autocomplete, with JSDoc settled on "routing label" terminology — omit-recommended, "smart" if pinning, raw provider ids never honored. Refs: SAP-2764 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * docs(skills): teach the LLM call-surface rule + naming conventions Adds the LLM call-surface rule to the canonical sapiom-agent-authoring skill — the highest-reach teaching surface of the whole SAP-2775/2776 stack, since it ships into every scaffolded agent project: ctx.sapiom.llm.run (one-shot) vs ctx.sapiom.models.run (platform-driven multi-turn loop, never for a one-shot) vs ctx.sapiom.agents.run (dispatch a deployed agent by slug), a worked "wrong vs right" example against the "reply with only JSON" + string-parsing mistake (forced tool call + explicit content-block `type` filtering instead), the omit-or-pin-`smart`-label rule, and a debugging pointer. Also adds a "Naming Conventions" section settling the platform's overloaded agent/run/task/session/dispatch vocabulary and "label" as the author-facing term for a model value, with a rule that new capabilities must not add a sixth meaning to an already-overloaded word. Synced identically across all four copies skill-sync.test.ts guards: the canonical source, both scaffold templates (default, coding-pause), and the Claude Code plugin copy — verified with `npx jest skill-sync` (7 passed). Folds in the models/index.ts:414-421 stale-comment fix flagged on SAP-2776: the "Default agent ... `agent.run` / `agent.launch`" header (and two adjacent comments in the same block) named a namespace that's actually exported as `models` — corrected throughout, including the top-of-file module docstring's own self-description and code example, which had the same problem in an even more prominent spot. Caught by the scaffold's own terminology guard (scaffold.test.ts's "copies the %s template with exact Agent terminology" — bans workflow(s)/orchestration(s) from anything shipped into a scaffold): an earlier draft of the Naming Conventions "agent" row explained the internal vs. customer-facing naming split by naming the internal term directly, and the debugging pointer cited the literal `/v1/workflows/...` REST path — both rephrased to convey the same information without the banned words. Flagging in the PR body that this same constraint may affect four earlier, already-open PRs in this teaching stack that used the identical debugging pointer text, none of which have this guard test. Refs: SAP-2776 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * fix(mcp): drop internal `workflows`-service naming from the LLM-usage-rule section sapiom-js's own scaffold terminology guard (agent-core's scaffold.test.ts) bans workflow(s)/orchestration(s) from anything shipped to a customer — the debugging line's literal `GET /v1/workflows/executions/:id/steps/:stepId/io` path, and the "one monolithic workflow" phrase, both leak that internal service naming. Ruling: the literal path's one deliberate home is the canonical guide (docs-internal#133); every other teaching surface points there instead of repeating it. Replaced both phrases; kept the shared section byte-identical with the Sapiom-repo backend copy (re-verified checksum). Updated the drift-guard test's assertion to match. Refs: SAP-2775, SAP-2776 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * fix(harness): point the debugging line at the guide instead of a literal path Ratified across the teaching stack (sapiom-js#683's scaffold terminology guard bans workflow(s)/orchestration(s) from customer-facing content): the per-step debugging endpoint's literal path lives in the canonical guide (docs-internal#133) only. This prompt's terse debugging line never spelled out the literal `/v1/workflows/...` path to begin with, but reworded it for consistency with the other teaching surfaces' "see the guide" pointer. Refs: SAP-2775, SAP-2776 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * chore(changeset): add missing changesets for the mcp + harness prompt content #679/#680 (now consolidated here) touched @sapiom/mcp's AUTHORING_INSTRUCTIONS fallback and @sapiom/harness's DEFAULT_SYSTEM_PROMPT without a changeset — both are user-facing content changes to changeset-tracked packages, and this repo's convention (checked prior harness/mcp commits) is one changeset per such change. Added both; left the pre-existing @sapiom/tools (#682) and @sapiom/agent-core + @sapiom/tools (#683) changesets as separate, non- overlapping entries per package. Refs: SAP-2764, SAP-2775, SAP-2776 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * fix(skills,mcp,changeset): sanitize internal service naming from shipped content This repo is public. A review found internal naming that had made it into shipped/publishable content, beyond the earlier workflow/orchestration scrub: - sapiom-agent-authoring/SKILL.md's Naming Conventions table named an internal service by its internal name ("agent-runs") and an internal design-doc label ("Surface B") — reworded to describe the capability by what it does, not its internal name. Also dropped a dangling reference to a "writing-agents" skill and a PROMPT.md path that don't exist in this repo (they're specific to a different, private repo) — a broken pointer shipping to every scaffold, found while fixing the naming leak in the same table cell. - packages/mcp/src/instructions.ts's module doc comment named a private companion repo's file path directly; TypeScript's declaration emit preserves this comment on the exported symbol, so it would ship in the published .d.ts. Genericized to "a private companion repo" (pre-existing, predates this branch's other changes). - The disclosure changeset (ships permanently in CHANGELOG.md on release) named an internal ticket id — trimmed. Synced the skill fix across all four copies skill-sync.test.ts guards; re-verified checksums and re-ran the affected test suites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc * fix(sap-2775): resolve review-round findings on the LLM call-surface teaching stack Fixes five content bugs raised across two bot-review rounds on this PR, plus two disclosure wording trims: - Structured-output teaching was self-contradictory (forced tool-use output has no text block; the result lives in tool_use). Fixed in the MCP fallback instructions, the harness system prompt, and the skill's worked example, with new tests pinning the "tool_use" wording so this can't regress silently again. - structuredOf/textOf/readDisclosure are now reachable from ctx.sapiom.llm (client.ts type + runtime, plus the run_local stub), additive only, with new tests exercising all three through the client. - Removed the skill's "is landing" blockquote for a convenience that ships in this same PR. - Fixed the "llm.submit/redeem is a multi-turn surface" mistake — it's a single call with deferred capacity. - Reverted LlmSessionCreateSpec.model to string (it pins an exact alias, mutually exclusive with the real label field). - Trimmed two Naming Conventions rows per a documentation-disclosure ruling. Skill edits synced across all four copies (skill-sync guard). * fix(sap-2775): round-4 review fixes — prompt wording, example, naming clarity - Harness prompt's debugging pointer said "documented below" with nothing below documenting it (the next line is the guide link). Changed to "documented in the guide", matching the MCP fallback copy's wording. Content-guard test now pins the corrected phrase and forbids the old one. - Skill's "Right" worked example passed `model: "smart"` in the mainline path while the surrounding text recommends omitting `model`. Example now omits it, with pinning shown as a one-line variant comment. - Naming Conventions "label" row clarified: a result's `servedClass` field is a disclosure field reporting the resolved billing class, not a contradiction of "never call a label a class" (author-facing input vs. server-reported output are different axes). Synced across all four SKILL.md copies (skill-sync + MD5 verified). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Summary
SDK half of a corrected server-side contract (internal tracker): expose the server's serving disclosure — the billing class + lane the request resolved to, in the SKU vocabulary the platform bills in. Never a model or provider id, never a provider price (an earlier draft's
servedModel/providercostUsdare gone per the corrected contract, shipped as a server-side companion change).What changed
models.run—ModelRunOutcome.servedClass+lane(wireserved_class/lane): the billing class (size) the run's label resolved to and the lane it executed in;modelUsedkeeps meaning the requested label.costUsdis deprecated and nownumber | null: servers report null (the customer dollar amount is the metering pipeline's, summed on run views); older servers' numbers still map through.models.coding.run—CodingRunOutcome.servedClass/lane/deprecatedcostUsd(all null from coding servers today; null = unknown, never fabricated).llm.run/llm.redeem/llm.callSession—LlmDisclosurenow describesserved_class/lane(injected top-level into raw /v2 non-streaming bodies; streams carry the same data asx-sapiom-served-class/x-sapiom-laneheaders);readDisclosure()returns the camelCaseLlmDisclosureResult { servedClass, lane }. The responsemodelfield keeps echoing the label.degradation(typedunknown) unchanged.costUsdnumber→number | nullwidening.Compatibility
null— covered on both mappers +readDisclosure.Testing
packages/tools: jest green (mapping incl. class/lane, old-server absent-fields, null-from-coding-servers,readDisclosurehappy/absent/malformed);pnpm buildclean; provider-neutral copy check passes.🤖 Generated with Claude Code
https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9