Skip to content

fix(agents): provider robustness — OpenAI strict schema, Anthropic refusals, retries, real request timeout - #467

Merged
epeicher merged 3 commits into
trunkfrom
fix/agents-answer-schema-openai-strict
Jul 31, 2026
Merged

fix(agents): provider robustness — OpenAI strict schema, Anthropic refusals, retries, real request timeout#467
epeicher merged 3 commits into
trunkfrom
fix/agents-answer-schema-openai-strict

Conversation

@epeicher

@epeicher epeicher commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #466.

Testing

CleanShot 2026-07-31 at 19 09 57@2x

What it does

Fixes four provider failures hit while testing #466:

  1. The 400 every agent chat hit on the OpenAI connector (below).
  2. Unexpected Anthropic API response: Missing the "content" key. — a model REFUSAL the Anthropic provider plugin crashes on before its own refusal handling.
  3. Agent stopped after 8 turns without a final answer. — the model spending the whole turn cap on tool calls.
  4. cURL error 28: Operation timed out after 30007 milliseconds — the AI Client's own 30-second request timeout, which feat(agents): enhance agent capabilities and HTTP timeout management #466's http_request_timeout raise cannot reach.

OpenAI: strict structured output 400

Bad Request (400) - Invalid schema for response_format 'response_schema': In context=('properties', 'call_to_actions', 'items'), 'required' is required to be supplied and to be an array including every key in properties. Missing 'style'.

Rationale

The OpenAI provider sends the answer schema verbatim with strict: true, and strict structured output has no optional fields: every key in properties must also appear in required. The agent answer schema listed style as optional in the call-to-action items (and call_to_actions as optional at the root), which OpenAI rejects before generating anything. #466 already covered the other strict rule (additionalProperties: false) and fixed the Copilot/drafts schemas; the agent answer schema's required lists were the remaining gap.

Implementation

  • desktop_mode_agent_answer_schema() declares full required lists: [id, label, style, reply] on the items, [text, call_to_actions] at the root. The lenient parse/sanitize path is untouched — a bad or missing style still defaults to secondary for pre-filter runtimes and non-strict providers.
  • desktop_mode_ai_normalize_response_schema() (the safety net feat(agents): enhance agent capabilities and HTTP timeout management #466 introduced for additionalProperties) now also repairs partial required lists in the same recursive walk, so a plugin-filtered schema addition cannot reintroduce the 400. Provider-safe: Google's provider already strips response-schema keys its API rejects.

Testing instructions

npm run test:php -- --filter='Tests_DesktopMode_AiResponseSchemaNormalization'   # 16 tests

Manual, with only the OpenAI provider plugin active (deactivate the Google/Anthropic providers to force routing): chat with any agent asking for a proposal that needs approval. Before this fix the answer is the 400 above; after, a structured answer arrives with working call-to-action buttons. Verified live against the OpenAI connector.

Anthropic: refusal handling, retry, forced final answer, real timeout

All reproduced live with only the Anthropic provider active. A raw-body trace of the failing "Missing content" case settled what it actually is:

TURN: status=200 stop=refusal content=[]
BODY: {"model":"claude-sonnet-5", ..., "content":[], "stop_reason":"refusal",
       "stop_details":{"type":"refusal","category":"bio", ...}}

A model refusal (a safety-classifier false positive over innocuous demo content), returned as a 200 with an empty content array — which the provider plugin throws on ("Missing the "content" key") before reaching its own refusal handling. Not a flap: retries usually reproduce it.

  • Refusal translation: desktop_mode_agent_humanize_generate_error() maps that parse error to an actionable message ("its safety system most likely declined this request — rephrase, or switch the provider in Settings → Connectors"), preserving the provider's original text in the error data.
  • Bounded retry: one retry per generate turn for transient signatures — a failed models-list fetch (No models found …), gateway 5xx, transport timeouts, and the empty-content case (kept because borderline refusals are stochastic and one request is cheap). desktop_mode_agent_generate_error_is_transient() is the pure, unit-tested classifier; deterministic rejections (schema 400s, rate limits) are never retried.
  • Forced final answer at the turn cap: when the model spends all DESKTOP_MODE_AGENT_RUNNER_MAX_TURNS turns calling tools, the runner makes one last TOOL-LESS generate over the transcript — with nothing to call, the model can only answer from what it already gathered. Falls back to the original max-turns error if even that fails; turns reports MAX + 1. The desktop_mode_agent_runner_generate filter doc notes it can now be invoked more than once per turn.
  • The 30s timeout: Core's WP_AI_Client_Prompt_Builder constructor pins an explicit 30-second timeout via the SDK's RequestOptions, bypassing the WordPress HTTP default that feat(agents): enhance agent capabilities and HTTP timeout management #466's wrapper raises — hence "timed out after 30007 milliseconds" despite the 180s allowance. The wrapper now also raises Core's wp_ai_client_default_request_timeout filter, with identical raise-only + finally-removal scoping.

Verified live with only the Anthropic provider active: the question that produced the 8-turn error now completes (turns 7 and 8 across runs), and a refused translation request surfaces the explained refusal instead of the parse error.

Open WordPress Playground Preview

@epeicher epeicher changed the title fix(agents): satisfy OpenAI strict structured output in the answer schema fix(agents): provider robustness — OpenAI strict schema, transient-error retry, forced final answer Jul 31, 2026
@epeicher epeicher changed the title fix(agents): provider robustness — OpenAI strict schema, transient-error retry, forced final answer fix(agents): provider robustness — OpenAI strict schema, Anthropic refusals, retries, real request timeout Jul 31, 2026
@epeicher
epeicher disabled the stack merge July 31, 2026 17:18
Base automatically changed from agents/tool-definitions-appendix to trunk July 31, 2026 17:19
epeicher added 3 commits July 31, 2026 19:19
OpenAI validates response_format json_schema with strict: true, and
strict mode has no optional fields: 'required' must list EVERY key in
'properties', or the whole request 400s ("'required' is required to
be supplied and to be an array including every key in properties.
Missing 'style'.").

- desktop_mode_agent_answer_schema(): 'style' joins the call-to-action
  items' required list and 'call_to_actions' joins the root's. The
  lenient parse/sanitize path is unchanged, so pre-filter runtimes and
  non-strict providers still work as before.
- desktop_mode_ai_normalize_response_schema() now also repairs partial
  'required' lists (recursively, same walk as the additionalProperties
  stamp), so a plugin-filtered schema addition cannot reintroduce the
  400 on OpenAI. Google's provider already strips response-schema keys
  it dislikes, so the repair is provider-safe.

Verified live against the OpenAI connector (Google/Anthropic provider
plugins deactivated to force routing): the previously failing chat now
returns a structured answer with working call-to-action buttons.
…t the turn cap

Two Anthropic-connector failures reproduced live, both one-off flaps
rather than deterministic rejections:

- 'Unexpected Anthropic API response: Missing the "content" key.' —
  the provider throws when a 200 arrives with an EMPTY content array
  (caught in a raw trace: status=200, no stop_reason, content=[]). A
  fresh request immediately succeeds.
- 'Agent stopped after 8 turns without a final answer.' — the model
  spends the whole cap calling tools (with thinking enabled) before
  answering; the run then discarded everything it gathered.

Fixes, both at the runner (the reliability boundary the HTTP-timeout
work established):

- One bounded retry per generate turn when the failure matches a
  transient signature (empty Anthropic content, a failed models-list
  fetch ('No models found'), gateway 5xx, transport timeout).
  Deterministic rejections (schema 400s, rate limits) are never
  retried. desktop_mode_agent_generate_error_is_transient() is the
  pure, unit-tested classifier.
- At the turn cap, one forced TOOL-LESS generate over the transcript:
  with nothing to call, the model can only produce a final answer from
  what it already gathered. Falls back to the original error when even
  that fails; turns reports MAX+1.
…refusals

- The 'cURL error 28: Operation timed out after 30007 milliseconds'
  failures were NOT the WordPress HTTP default: Core's
  WP_AI_Client_Prompt_Builder constructor pins an explicit 30-second
  timeout via the SDK's RequestOptions, which reaches the transport
  directly and bypasses the http_request_timeout raise entirely. The
  timeout wrapper now also raises Core's
  wp_ai_client_default_request_timeout filter (raise-only, removed in
  finally, same scoping) so long generations get the full allowance.
- The 'Unexpected Anthropic API response: Missing the "content" key.'
  errors are model REFUSALS, not flaps: a raw-body trace shows 200
  responses with stop_reason 'refusal', an empty content array, and
  stop_details.category 'bio' over innocuous demo content — the
  provider plugin crashes on the empty content before reaching its own
  refusal handling. desktop_mode_agent_humanize_generate_error() now
  translates that parse error into an actionable message (rephrase or
  switch provider), preserving the original detail in error data; the
  transient classifier's comments are corrected accordingly (one retry
  kept for borderline refusals).
@epeicher
epeicher force-pushed the fix/agents-answer-schema-openai-strict branch from 0c3ad74 to bf378bd Compare July 31, 2026 17:19
@epeicher
epeicher merged commit eef2046 into trunk Jul 31, 2026
5 checks passed
@epeicher
epeicher deleted the fix/agents-answer-schema-openai-strict branch July 31, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant