Skip to content

Normalize AI Copilot tool schemas for the provider - #366

Merged
AllTerrainDeveloper merged 2 commits into
WordPress:trunkfrom
juanlentino:fix/ai-copilot-tool-schema-normalization
Jul 18, 2026
Merged

Normalize AI Copilot tool schemas for the provider#366
AllTerrainDeveloper merged 2 commits into
WordPress:trunkfrom
juanlentino:fix/ai-copilot-tool-schema-normalization

Conversation

@juanlentino

Copy link
Copy Markdown
Contributor

Fixes #362.

Thanks @AllTerrainDeveloper for confirming the approach on the issue. This is the normalization fix; the fail-soft-per-tool and name-the-culprit ideas are left as the follow-ups you flagged.

The problem

desktop_mode_ai_run_search() advertises each built-in tool with its ability's input_schema passed straight through as the tool parameters (includes/ai-copilot/search.php). The Abilities API accepts the full breadth of JSON Schema, but the provider's tool-schema validator does not — and it rejects the whole request, not just the offending tool. So a single tool with a legal-but-unsupported schema makes Ask AI return a 400 for every query: unusable, not degraded, and the error names a tools.N index rather than the plugin, so it's painful to trace.

Three shapes, all valid JSON Schema, trigger it in the wild:

  1. type as an array — e.g. ['object','null'], an ability's GET/null run-path. The provider wants the literal "object" at the top level.
  2. A top-level oneOf / anyOf / allOf — e.g. "supply post_id OR slug". Rejected with "does not support oneOf, allOf, or anyOf at the top level".
  3. An empty properties — a no-args tool's [], where an object schema needs {}.

The fix

A small pure helper, desktop_mode_ai_normalize_tool_schema(), projects a schema onto the provider-supported subset:

  • forces top-level type to "object"
  • strips top-level oneOf / anyOf / allOf (nested combinators are left intact — those are real constraints the provider accepts)
  • coerces an empty properties array to an object
  • empty / non-array input becomes {"type":"object","properties":{}}

It's applied to every tool in the final list, after the desktop_mode_ai_tools filter — so it covers the complete set the provider receives: built-in abilities, command tools, and anything a plugin injected. No single tool, from any source, can 400 the request.

Nothing loses enforcement. This reshapes only the copy advertised to the model. WP_Ability::execute() still validates arguments against the real schema and permission_callback still gates execution — the model is simply told the constraint in prose (the tool description) instead of a schema construct the provider can't parse. The projection is idempotent, so a plugin that already normalizes on the filter is unaffected.

Tests

tests/phpunit/tests/aiToolSchemaNormalization.php covers each of the three shapes, nested-combinator preservation, empty/non-array input, and idempotence. The helper is pure, so these run without the network or the ability registry.

Notes

  • Fail-soft-per-tool (drop the offending tool and name it, rather than 400-ing) and naming the culprit are the resilience follow-ups you called out — happy to open a separate PR for those; they're a different concern from this correctness fix.
  • I tagged @since 0.9.6 as a guess against the current 0.9.5 release — please adjust to your intended version.
  • Verified the helper's behaviour directly and confirmed phpcs reports no new violations on the change. I couldn't run the full PHPUnit suite locally (needs the wp-env test DB), so I'd lean on CI for the test:php leg.

Prepared by @juanlentino with Claude Code assistance.

One tool with a valid-but-unsupported input_schema makes the AI Copilot return a
400 for every query: the provider's tool-schema validator rejects the entire tool
list, not just the offending tool, so the assistant is 100% unusable, not
degraded. Three shapes seen in the wild trigger it: a "type" array (an ability's
GET/null run-path), a top-level oneOf/anyOf/allOf, and an empty "properties" array
that encodes to JSON as [] where an object needs {}.

Add desktop_mode_ai_normalize_tool_schema() and apply it to every tool in the
final list, after the desktop_mode_ai_tools filter, so it covers built-in
abilities, command tools, and anything a plugin injected. This reshapes only the
model-facing copy: WP_Ability::execute() still validates arguments against the
real schema and permission_callback still gates execution, so no enforcement is
lost. Only the top level is constrained; nested combinators are preserved. The
projection is idempotent.

Adds unit coverage for all three shapes plus nested-combinator preservation and
idempotence.

Fixes WordPress#362.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A schema whose only content was a top-level combinator (or a bare type
union) normalized to {"type":"object"} with no properties key at all,
which leaves the provider-facing schema incomplete for the same strict
validator this fix targets. Default a missing properties key to an empty
object, same as the empty-array coercion, so the projection always emits
a complete object schema.
@AllTerrainDeveloper

AllTerrainDeveloper commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Reviewed this locally, the projection had one remaining gap: a schema whose only content is a top-level combinator (e.g. { oneOf: [ { properties: … }, { properties: … } ] } with nothing duplicated at the top level), or a bare { "type": ["object","null"] }, normalized to {"type":"object"} with no properties key at all, still an incomplete object schema for the same strict validator this PR targets, and the model would see a tool with zero argument info.

@AllTerrainDeveloper
AllTerrainDeveloper enabled auto-merge (squash) July 18, 2026 17:13
@AllTerrainDeveloper
AllTerrainDeveloper merged commit 9f2fe22 into WordPress:trunk Jul 18, 2026
5 checks passed
@AllTerrainDeveloper

Copy link
Copy Markdown
Collaborator

Just a note, that you are already appearing here:
https://next-admin.blog/contribute/

If you have any inconvenience, please tell it to me :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI Copilot: one third-party ability with a valid-but-unsupported input_schema 400s the entire assistant

2 participants