Skip to content

fix(agent-platform): read the janitor's args_schema in tool schemas - #69202

Merged
dmarticus merged 1 commit into
PostHog:masterfrom
Aidan945:fix/custom-tool-schema-key
Jul 9, 2026
Merged

fix(agent-platform): read the janitor's args_schema in tool schemas#69202
dmarticus merged 1 commit into
PostHog:masterfrom
Aidan945:fix/custom-tool-schema-key

Conversation

@Aidan945

@Aidan945 Aidan945 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Problem

Models see custom tools with no parameters. The janitor writes tools/<id>/schema.json as {description, args_schema} (writeToolSourceAndSchema), but the runner's loadCustomSchema parses {description, args} — so every typed-pipeline tool's schema silently resolves to a bare {type: "object"} and the model can only guess arguments from prose. A session transcript shows the model concluding "the tool's schema shows no explicit input parameters".

Both sides pass their own tests because the runner's fixtures hand-write schema.json with the runner's key — two hand-maintained copies of one wire contract. Among the readers of this file, the runner was the only one parsing args; readTypedBundle already reads args_schema.

Changes

File Change Why
agent-runner/src/loop/build-agent-tools.ts loadCustomSchema reads args_schema Direct fix; aligns the one outlier reader with the only production writer. No production bundle uses the legacy args key (confirmed by the maintainer), so there is a single path, no fallback
same file warn logs on both silent paths: unreadable schema.json (custom_tool.schema_unreadable, with the error) and a parsed schema missing both keys (custom_tool.schema_missing_args_schema) The degradation was invisible by construction; the second event covers the exact failure mode of this bug if the key ever drifts again
build-agent-tools.test.ts regression test authors through the real writeToolSourceAndSchema Welds writer to reader: if the canonical shape drifts again, this fails
4 files in agent-tests/src/cases/ hand-written args fixtures → args_schema e2e now rehearses the shape production writes — removes the masking that let this ship

The key rename is the direct fix; the two warn logs and fixture flips are the durable half — they make this class of drift visible and tested rather than silent. Deliberately not in this PR: single-sourcing the schema.json codec in agent-shared (natural follow-up; kept out to avoid colliding with in-flight storage-boundary work).

Sibling fix in the same authoring loop: #69201.

How did you test this code?

Ran locally against a full dev stack (all four agent services + Postgres/Redis/Kafka/SeaweedFS):

  • New regression test custom tool parameters survive the writeToolSourceAndSchema → loadCustomSchema round-trip, authored through the real writer against the real test bundle store. With the fix reverted it fails (parameters collapse to the bare {type:"object"} fallback) — the writer→reader weld no existing test provided.
  • pnpm vitest run src/loop/build-agent-tools.test.ts25 passed; full @posthog/agent-runner suite → all assertions pass; typescript:check, oxlint, oxfmt --check → clean on agent-runner and agent-tests.
  • End-to-end: before the fix, the model calls a schema'd tool with {}; after, it passes correct structured arguments. Reproduced both directions.
  • The fixture flips are behavior-neutral by construction (the reader accepts both keys); they execute fully in CI's e2e job — I couldn't run that suite on this machine (kafka host alias), so that's the one place CI adds coverage beyond what I ran.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

No docs change needed.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

I hit this while building agents against a local stack — a model calling my custom tool with empty arguments — and traced it from the transcript to the key mismatch; the fix and tests were developed in a Claude Code session (Fable 5) under my direction, with every executable claim above verified by running it. Key decision: fix the reader, not the writer — a census of every schema.json consumer showed the runner was the single outlier, and frozen bundles are immutable, so only the reader side heals revisions that already exist.

🤖 Generated with Claude Code

@Aidan945
Aidan945 marked this pull request as ready for review July 8, 2026 09:18
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 8, 2026 09:18

@dmarticus dmarticus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for this fix! I left some comments around just committing to a single direction for this change rather than supporting the legacy behavior, since we don't have any bundles in production that are actually malformed. I'll approve once that's addressed!

Comment thread products/agent_platform/services/agent-runner/src/loop/build-agent-tools.test.ts Outdated
The one production writer of custom-tool schema.json is the janitor's
typed pipeline (typed-bundle.ts writeToolSourceAndSchema), which writes
`{ description, args_schema }`; agent-shared's readTypedBundle already
reads `args_schema`. The runner's loadCustomSchema was the only reader
still parsing the legacy `args` key, so it silently fell back to a bare
`{ type: 'object' }` — the model never saw the parameter schema and could
only guess arguments from prose (a transcript shows it concluding the
tool has no parameters).

The runner now reads `args_schema` and nothing else, matching
readTypedBundle and the sole production writer's key. No production bundle
uses the legacy `args` key (confirmed against the production buckets), so
this commits to a single direction rather than two ways to spell the same
contract. The silent degradation is still guarded on both paths: an
unreadable schema.json (`custom_tool.schema_unreadable`, with the error)
and a parsed schema with no `args_schema`
(`custom_tool.schema_missing_args_schema`) — the latter being the exact
failure mode of this bug had the key drifted again, and now unambiguously
wrong.

Test fixtures use the production `args_schema` shape (agent-runner +
agent-tests). A single regression test welds the writer to the reader: it
authors through the real writer (writeToolSourceAndSchema) and asserts the
schema survives the writeToolSourceAndSchema → loadCustomSchema round-trip
(it collapses to a bare object schema without the fix).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Aidan945
Aidan945 force-pushed the fix/custom-tool-schema-key branch from 6597332 to 2a60baa Compare July 9, 2026 00:50
@Aidan945

Aidan945 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Makes sense. Good to get ahead of it. Dropped the fallback and the legacy test, so args_schema is the single path now.

@dmarticus dmarticus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work; thank you!

@dmarticus
dmarticus merged commit d9bda89 into PostHog:master Jul 9, 2026
192 checks passed
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 9, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-09 15:12 UTC Run
prod-us ✅ Deployed 2026-07-09 15:43 UTC Run
prod-eu ✅ Deployed 2026-07-09 15:40 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants