Skip to content

feat(tools): expose servedModel / costUsd disclosure on execution results (SAP-2764) - #673

Closed
gwitwer wants to merge 4 commits into
mainfrom
feat/SAP-2764
Closed

feat(tools): expose servedModel / costUsd disclosure on execution results (SAP-2764)#673
gwitwer wants to merge 4 commits into
mainfrom
feat/SAP-2764

Conversation

@gwitwer

@gwitwer gwitwer commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

SDK half of a corrected server-side contract (internal tracker): expose the server's serving disclosure — the billing class + lane the request resolved to, in the SKU vocabulary the platform bills in. Never a model or provider id, never a provider price (an earlier draft's servedModel/provider costUsd are gone per the corrected contract, shipped as a server-side companion change).

What changed

  • models.runModelRunOutcome.servedClass + lane (wire served_class/lane): the billing class (size) the run's label resolved to and the lane it executed in; modelUsed keeps meaning the requested label. costUsd is deprecated and now number | null: servers report null (the customer dollar amount is the metering pipeline's, summed on run views); older servers' numbers still map through.
  • models.coding.runCodingRunOutcome.servedClass/lane/deprecated costUsd (all null from coding servers today; null = unknown, never fabricated).
  • llm.run / llm.redeem / llm.callSessionLlmDisclosure now describes served_class/lane (injected top-level into raw /v2 non-streaming bodies; streams carry the same data as x-sapiom-served-class/x-sapiom-lane headers); readDisclosure() returns the camelCase LlmDisclosureResult { servedClass, lane }. The response model field keeps echoing the label.
  • Reserved degradation (typed unknown) unchanged.
  • Minor changeset rewritten, calling out the costUsd numbernumber | null widening.

Compatibility

  • All new fields optional/nullable → existing consumers compile/run unchanged; old servers map missing fields to null — covered on both mappers + readDisclosure.

Testing

  • packages/tools: jest green (mapping incl. class/lane, old-server absent-fields, null-from-coding-servers, readDisclosure happy/absent/malformed); pnpm build clean; provider-neutral copy check passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9

gwitwer and others added 4 commits August 21, 2026 01:57
…ecution results

Additive, optional-safe result fields per the DisclosureFields contract
(Sapiom plans/model-execution-surface/interfaces.md):

- models.run: ModelRunOutcome.servedModel (wire served_model) — the deployment
  that actually served, distinct from modelUsed which stays the requested
  label; costUsd doc'd as the served-priced real cost; degradation reserved
  (SAP-2768).
- models.coding.run: CodingRunOutcome gains servedModel/costUsd (nullable —
  coding cannot observe them yet) + reserved degradation.
- llm.run / llm.redeem / llm.callSession: LlmDisclosure type + readDisclosure()
  helper for the served_model/cost_usd fields the gateway injects into raw /v2
  non-streaming bodies (camelCased, old-server safe).

All fields optional/nullable: existing consumers compile and run unchanged;
missing wire fields map to null (unknown) — never fabricated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9
…et, old-server coverage

- Published JSDoc no longer names a serving vendor or internal deployment
  codename (opaque placeholder in specs), and no longer points at a
  private-repo planning doc or internal ticket ids — fields are described by
  their wire semantics, attributed to the server.
- readDisclosure now returns the named, exported LlmDisclosureResult
  (camelCase); LlmDisclosure stays exported as the documented raw wire shape
  (matching the snake_case bodies llm.run returns).
- Added the missing minor changeset for @sapiom/tools, including the changed
  documented meaning of the existing ModelRunOutcome.costUsd.
- Old-server compatibility is now tested on both mappers: models.run maps
  absent disclosure fields to null, and a new coding-result spec covers
  null/absent/present disclosure on models.coding.run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9
…; forbid 'fireworks' in copy check

- degradation stays reserved (required by the driving ticket) but is now typed
  `unknown` on all three public types: consumers must narrow before reading,
  so the future concrete type is purely additive — addresses the review's
  narrowing-is-breaking concern without un-shipping the reservation.
- Changeset wording now says exactly which fields map to null on old servers
  vs stay absent.
- provider-neutral-copy-check gains 'fireworks' so the round-1 leak class
  cannot regress silently.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9
…4 contract correction

The deliverable changed (Dave, 2026-08-21): disclosing the raw served model
contradicts the SKU contract and SAP-1891, and a provider-priced cost must
never reach a customer. The disclosure vocabulary is size × lane.

- ModelRunOutcome/CodingRunOutcome: servedModel → servedClass + lane (wire
  served_class/lane); costUsd deprecated + nullable (servers now report null —
  the metering pipeline owns customer dollars; old servers' numbers still map).
- LlmDisclosure/readDisclosure: served_class + lane (LlmDisclosureResult is
  { servedClass, lane }); streams carry the same data as
  x-sapiom-served-class / x-sapiom-lane headers.
- degradation reservation unchanged (unknown-typed).
- Changeset rewritten to match, including the costUsd number → number|null
  widening.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SZmJ79CfTQ4M1EeY5RD9s9
@gwitwer

gwitwer commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

Reworked in ca7c37b to the corrected disclosure contract (see the server-side discussion): the disclosure vocabulary is now billing class × laneservedClass/lane replace servedModel everywhere, and costUsd is deprecated + nullable (servers report null; the metering pipeline owns customer dollars). LlmDisclosureResult is now { servedClass, lane }. This also fully retires the round-1 confidentiality concern class: no field can carry a provider identifier anymore, by construction. PR body updated; changeset rewritten with the number → number | null widening called out.

@gwitwer

gwitwer commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator Author

Closing — the disclosure contract changed: customer results carry served class + lane + token usage, never the provider model or provider-priced cost (per the platform's SKU contract). These servedModel/costUsd types would expose a contract the server no longer serves. Reissue as servedClass/lane/usage result types as part of the upcoming SDK ergonomics PR (the platform's corrected model-execution contract, internal, is authoritative on the shape).

@gwitwer

gwitwer commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in 7c4ef1c — point-by-point:

🔒 Vendor/codename in shipped JSDoc — Fixed. The JSDoc example is gone (field described by its semantics: "server-assigned identifier of the deployment"), and all spec fixtures use the opaque deployment-a. node scripts/provider-neutral-copy-check.mjs passes. (Note: some pre-existing internal label strings in submit.spec.ts predate this PR and are untouched — flagging separately if you want them swept.)

🔒 Private-repo pointer / internal-tracker link — Fixed. The contract path and ticket ids are out of the published JSDoc (fields are documented by their wire semantics instead), and the PR body no longer carries the internal tracker link or the private companion-branch description.

Missing changeset — Fixed. Added a minor changeset for @sapiom/tools that explicitly calls out the changed documented meaning of the existing ModelRunOutcome.costUsd (server-reported cost, served-priced when disclosed) alongside the new fields.

Cut degradation — Respectfully keeping it: reserving the optional degradation field in these exact result shapes is an explicit Must of the driving requirement ("Reserve the optional degradation field in the same result shape, populated by a follow-up"), so removing it would un-ship a requirement. To blunt the narrowing concern, it's documented as "server-defined shape" — consumers are told the shape isn't theirs to depend on, and Record<string, unknown> only permits reads of unknown, so a later concrete type narrows what they can assume, not what compiles.

JSDoc asserts backend behavior — Fixed. All disclosure docs now state what the wire field is and attribute interpretation to the server ("as reported by the server", "server-assigned identifier"), with no client-side guarantee implied, and no internal ticket ids in published docs.

LlmDisclosure snake_case / unnameable return type — Fixed the second half: readDisclosure now returns the named, exported camelCase LlmDisclosureResult. Keeping LlmDisclosure exported in wire shape deliberately: unlike the rest of the package, llm.run/redeem/callSession return raw provider bodies (generic T, snake_case throughout), so the type consumers intersect with those bodies (llm.run<Message & LlmDisclosure>) must match the wire casing — a camelCase version would describe fields that don't exist on the object. Its docstring now says exactly that.

Old-server path untested — Fixed. run-launch.spec gains an absent-fields case for mapModelResult, and a new coding-result.spec.ts covers coding's mapper for all three cases (null fields as today's servers emit, absent fields as older servers emit, values passed through).

@gwitwer

gwitwer commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator Author

Round 2 addressed in f316c8a:

degradation reservation — Keeping the field, hardening the type. Reserving degradation in these exact result shapes is an explicit Must of the driving ticket, so cutting it would un-ship a requirement — but your substantive concern (narrowing an untyped bag later breaks anyone who indexed into it) is real, so it's now typed unknown on all three public types (and the wire interfaces). Nothing can index into unknown without narrowing first, which makes the future concrete type purely additive — the breaking-change vector you flagged no longer exists. The changeset states the shape is server-defined and not yet stable.

Changeset precision — Fixed: it now says exactly which fields map to null on old servers (servedModel/costUsd via the mappers and readDisclosure) vs stay absent (degradation, raw body fields).

Checker gap — Added the deployment vendor's name from the round-1 finding to the provider-neutral copy checker's forbidden list, so that leak class can't regress silently; check passes on the tree.

Test execution — For the record since the sandbox blocked jest for you: npx jest --maxWorkers=1 on the three touched spec files → 13 passed; full packages/tools suite was green before round 2 (567 tests) and pnpm build is clean after it.

@sapiom sapiom deleted a comment from github-actions Bot Aug 23, 2026
@sapiom sapiom deleted a comment from github-actions Bot Aug 23, 2026
gwitwer added a commit that referenced this pull request Aug 23, 2026
…s all authoring surfaces (SAP-2764/2775/2776) (#685)

* docs(mcp): teach the LLM call-surface rule in the authoring instructions

Add the same "LLM calls & agent loops" section shipped in the companion
Sapiom-repo PR: ctx.sapiom.llm.run (one-shot, read only type==='text' blocks
— never string-parse JSON out of a response that may carry a thinking
block) vs ctx.sapiom.models.run / models.coding.run (platform-driven
multi-turn loops) vs ctx.sapiom.agents.run (dispatch a deployed agent by
slug); the "you never pick a model" deadlineMinutes/class-label contract;
and the step-io endpoint for debugging a run. Highest-reach usability fix
in the project — this is the bundled offline fallback AUTHORING_INSTRUCTIONS
served when the live @sapiom/mcp startup fetch to the backend fails.

The new section is byte-identical to the backend's DEFAULT_MCP_INSTRUCTIONS
copy of it (checksum-verified). Note: the two files as a WHOLE were already
not byte-identical before this change for unrelated, pre-existing reasons
(this file's title, its older "Two ways to use Sapiom" aliasing section, and
its ctx.shared 256 KiB quota paragraph from SAP-2790/#677 are absent from or
worded differently than the backend copy) — out of scope here; flagged
separately.

Refs: SAP-2775
Companion Sapiom-repo PR: sapiom/Sapiom#4519

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(mcp): correct the model-pinning claim in the LLM-usage-rule section

The "You may pin a class label (small / medium / large / smart)" sentence
was wrong against shipped behavior: today's AGENTS_LLM_V2_KNOWN_LABELS
allowlist is smart, minimax-m3, m2.7, opus, haiku, sonnet, m3-test —
small/medium/large are the SKU spec's future vocabulary and would
warn-and-default today, not pin. Replaced with present-truth guidance:
omit `model` entirely (recommended), or pin the one label that's
contract-attested to work (`smart`) — raw provider model ids are never
honored. Kept byte-identical to the companion Sapiom-repo fix (checksum
33ced1cf7be83ab72ced6e8837ba585131710bb553c064dcde64d68288011314 for the
shared section in both).

Refs: SAP-2775
Companion Sapiom-repo PR: sapiom/Sapiom#4519

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* docs(harness): teach the LLM call-surface rule in the system prompt

The #1 mis-selection source is the embedded coding agent itself picking the
wrong LLM call surface while writing a Sapiom agent's step code — this is
the highest-reach fix among the SAP-2775 surfaces, since it's the one
already reading the codebase. Adds a compact "Calling LLMs from agent code"
block: ctx.sapiom.llm.run (one-shot) vs ctx.sapiom.models.run (multi-turn
loop — never for a one-shot, it overthinks) vs ctx.sapiom.agents.run
(dispatch a deployed agent); structured output via tool-use/schema output,
read only type==='text' blocks, never string-parse JSON; omit `model`
(recommended) or pin the `smart` label — raw provider ids are never
honored; results disclose the served class + lane; and the per-step /io
endpoint / Run Inspector for debugging. Kept to ~8 lines (the prompt is
token-budgeted every session) versus the fuller MCP-instructions version
(sapiom-js#679) — same rule, terser form.

Corrected the existing "sapiom (remote, HTTP)" bullet's "models" mention to
point at the new block instead of naming a single vague capability, and
introduced this package's first docs.sapiom.ai citation (no prior
convention existed here) pointing at the canonical guide page.

Extended system-prompt.test.ts's existing content-guard assertions (the
established pattern in this file) to cover the new section's key phrases.

Refs: SAP-2775

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* feat(tools): served class/lane disclosure, structured output, label ergonomics (SAP-2764)

Reissues #673 under the corrected contract, plus the strong-defaults
ergonomics items. Additive only — no published type or behavior changes
for existing consumers.

- models.run (ModelRunOutcome) and models.coding.run (CodingRunOutcome):
  optional servedClass/lane (wire served_class/lane) — the billing class
  the run's label resolved to and the lane it executed in. Verified fresh
  against origin/dev (sapiom/Sapiom@b40e580c3): both wire shapes carry
  these fields (coding always null today, no cost_usd key on coding at
  all — it never had a real one to preserve). Left ModelRunOutcome.costUsd
  untouched (still number, non-nullable) per the additive-only mandate;
  the server's own stale-#4482-row null guard is a narrow, pre-existing
  edge case this PR does not attempt to fix in the type.
- llm.run/redeem/callSession: LlmDisclosure (wire shape) + readDisclosure()
  (camelCased LlmDisclosureResult), mirroring #673's reviewed design minus
  the still-contested `degradation` reservation (out of scope here).
- llm.run gains an optional `output: { name, schema }` — the blessed
  tool-calling pattern for structured output, automated (appends a forced
  tool + tool_choice). run()'s return type is unchanged either way; read
  the parsed value with the new structuredOf(). New textOf() reads the
  plain-text reply, skipping a `thinking` block that may precede it.
- model/label fields across llm.run, llm.submit, llm.createSession,
  models.run, and models.coding.run: soft-union type ("smart" |
  (string & Record<never, never>), the lint-safe spelling already used by
  content-generation's LiteralUnion) for autocomplete, with JSDoc settled
  on "routing label" terminology — omit-recommended, "smart" if pinning,
  raw provider ids never honored.

Refs: SAP-2764

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* docs(skills): teach the LLM call-surface rule + naming conventions

Adds the LLM call-surface rule to the canonical sapiom-agent-authoring
skill — the highest-reach teaching surface of the whole SAP-2775/2776 stack,
since it ships into every scaffolded agent project: ctx.sapiom.llm.run
(one-shot) vs ctx.sapiom.models.run (platform-driven multi-turn loop, never
for a one-shot) vs ctx.sapiom.agents.run (dispatch a deployed agent by
slug), a worked "wrong vs right" example against the "reply with only
JSON" + string-parsing mistake (forced tool call + explicit content-block
`type` filtering instead), the omit-or-pin-`smart`-label rule, and a
debugging pointer. Also adds a "Naming Conventions" section settling the
platform's overloaded agent/run/task/session/dispatch vocabulary and
"label" as the author-facing term for a model value, with a rule that new
capabilities must not add a sixth meaning to an already-overloaded word.

Synced identically across all four copies skill-sync.test.ts guards: the
canonical source, both scaffold templates (default, coding-pause), and the
Claude Code plugin copy — verified with `npx jest skill-sync` (7 passed).

Folds in the models/index.ts:414-421 stale-comment fix flagged on SAP-2776:
the "Default agent ... `agent.run` / `agent.launch`" header (and two
adjacent comments in the same block) named a namespace that's actually
exported as `models` — corrected throughout, including the top-of-file
module docstring's own self-description and code example, which had the
same problem in an even more prominent spot.

Caught by the scaffold's own terminology guard (scaffold.test.ts's
"copies the %s template with exact Agent terminology" — bans
workflow(s)/orchestration(s) from anything shipped into a scaffold): an
earlier draft of the Naming Conventions "agent" row explained the internal
vs. customer-facing naming split by naming the internal term directly, and
the debugging pointer cited the literal `/v1/workflows/...` REST path —
both rephrased to convey the same information without the banned words.
Flagging in the PR body that this same constraint may affect four earlier,
already-open PRs in this teaching stack that used the identical debugging
pointer text, none of which have this guard test.

Refs: SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(mcp): drop internal `workflows`-service naming from the LLM-usage-rule section

sapiom-js's own scaffold terminology guard (agent-core's scaffold.test.ts)
bans workflow(s)/orchestration(s) from anything shipped to a customer — the
debugging line's literal `GET /v1/workflows/executions/:id/steps/:stepId/io`
path, and the "one monolithic workflow" phrase, both leak that internal
service naming. Ruling: the literal path's one deliberate home is the
canonical guide (docs-internal#133); every other teaching surface points
there instead of repeating it. Replaced both phrases; kept the shared
section byte-identical with the Sapiom-repo backend copy (re-verified
checksum). Updated the drift-guard test's assertion to match.

Refs: SAP-2775, SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(harness): point the debugging line at the guide instead of a literal path

Ratified across the teaching stack (sapiom-js#683's scaffold terminology
guard bans workflow(s)/orchestration(s) from customer-facing content): the
per-step debugging endpoint's literal path lives in the canonical guide
(docs-internal#133) only. This prompt's terse debugging line never spelled
out the literal `/v1/workflows/...` path to begin with, but reworded it for
consistency with the other teaching surfaces' "see the guide" pointer.

Refs: SAP-2775, SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* chore(changeset): add missing changesets for the mcp + harness prompt content

#679/#680 (now consolidated here) touched @sapiom/mcp's AUTHORING_INSTRUCTIONS
fallback and @sapiom/harness's DEFAULT_SYSTEM_PROMPT without a changeset —
both are user-facing content changes to changeset-tracked packages, and this
repo's convention (checked prior harness/mcp commits) is one changeset per
such change. Added both; left the pre-existing @sapiom/tools (#682) and
@sapiom/agent-core + @sapiom/tools (#683) changesets as separate, non-
overlapping entries per package.

Refs: SAP-2764, SAP-2775, SAP-2776

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(skills,mcp,changeset): sanitize internal service naming from shipped content

This repo is public. A review found internal naming that had made it into
shipped/publishable content, beyond the earlier workflow/orchestration scrub:

- sapiom-agent-authoring/SKILL.md's Naming Conventions table named an
  internal service by its internal name ("agent-runs") and an internal
  design-doc label ("Surface B") — reworded to describe the capability by
  what it does, not its internal name. Also dropped a dangling reference to
  a "writing-agents" skill and a PROMPT.md path that don't exist in this
  repo (they're specific to a different, private repo) — a broken pointer
  shipping to every scaffold, found while fixing the naming leak in the
  same table cell.
- packages/mcp/src/instructions.ts's module doc comment named a private
  companion repo's file path directly; TypeScript's declaration emit
  preserves this comment on the exported symbol, so it would ship in the
  published .d.ts. Genericized to "a private companion repo" (pre-existing,
  predates this branch's other changes).
- The disclosure changeset (ships permanently in CHANGELOG.md on release)
  named an internal ticket id — trimmed.

Synced the skill fix across all four copies skill-sync.test.ts guards;
re-verified checksums and re-ran the affected test suites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CDogyqvDnhy5iKgVpeZtWc

* fix(sap-2775): resolve review-round findings on the LLM call-surface teaching stack

Fixes five content bugs raised across two bot-review rounds on this PR, plus
two disclosure wording trims:

- Structured-output teaching was self-contradictory (forced tool-use output
  has no text block; the result lives in tool_use). Fixed in the MCP
  fallback instructions, the harness system prompt, and the skill's worked
  example, with new tests pinning the "tool_use" wording so this can't
  regress silently again.
- structuredOf/textOf/readDisclosure are now reachable from ctx.sapiom.llm
  (client.ts type + runtime, plus the run_local stub), additive only, with
  new tests exercising all three through the client.
- Removed the skill's "is landing" blockquote for a convenience that ships
  in this same PR.
- Fixed the "llm.submit/redeem is a multi-turn surface" mistake — it's a
  single call with deferred capacity.
- Reverted LlmSessionCreateSpec.model to string (it pins an exact alias,
  mutually exclusive with the real label field).
- Trimmed two Naming Conventions rows per a documentation-disclosure ruling.

Skill edits synced across all four copies (skill-sync guard).

* fix(sap-2775): round-4 review fixes — prompt wording, example, naming clarity

- Harness prompt's debugging pointer said "documented below" with nothing
  below documenting it (the next line is the guide link). Changed to
  "documented in the guide", matching the MCP fallback copy's wording.
  Content-guard test now pins the corrected phrase and forbids the old one.
- Skill's "Right" worked example passed `model: "smart"` in the mainline
  path while the surrounding text recommends omitting `model`. Example now
  omits it, with pinning shown as a one-line variant comment.
- Naming Conventions "label" row clarified: a result's `servedClass` field
  is a disclosure field reporting the resolved billing class, not a
  contradiction of "never call a label a class" (author-facing input vs.
  server-reported output are different axes).

Synced across all four SKILL.md copies (skill-sync + MD5 verified).

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot mentioned this pull request Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant