Skip to content

AI: surface an empty final answer as an error instead of a silent success - #530

Open
juanlentino wants to merge 1 commit into
WordPress:trunkfrom
juanlentino:fix/ai-empty-answer
Open

AI: surface an empty final answer as an error instead of a silent success#530
juanlentino wants to merge 1 commit into
WordPress:trunkfrom
juanlentino:fix/ai-empty-answer

Conversation

@juanlentino

@juanlentino juanlentino commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Scoped to the adapter change green-lit in #517 (plus the belt-and-braces runner guard).

What

A final turn with no function calls whose text cannot be extracted — toText() throws, or the response carries no text part (observed live: a hard agent task spends the entire max_tokens budget inside a thinking block, stop_reason: "max_tokens") — was swallowed into a "successful" generation with text: null. The agents runner then reported success with an empty answer, bypassing both the transient retry and every error surface.

openstation_ai_client_generate() now returns WP_Error( 'openstation_ai_empty_answer' ) for that case, with the original extraction detail preserved in the error data.

Consumers, walked

  • Search agentic loop: the duplicate local ! is_string( $text ) check (the old openstation_ai_empty branch) is removed — the centralized error reaches the loop's existing is_wp_error handling at both generate sites. Net behavior change: an empty-string final turn now reports openstation_ai_empty_answer instead of falling through to openstation_ai_result_parse.
  • Command follow-up: the deliberate degrade is preserved exactly as requested — this path matches the new code and still falls back to "Done." with the fallback flag, so a command that DID run never surfaces as a failure, and no openstation_ai_search_error fires for it (same as today).
  • Agents runner: belt-and-braces guard in the toolless-turn branch (trim-empty text ⇒ error, never a success with ''), plus a humanized mapping (openstation_agent_empty_answer) telling the user the model likely spent its output budget reasoning. The new code is deliberately NOT in the transient-retry signature list — the failure is task-shaped, not a flap; happy to add it if you'd rather retry once.
  • stop_reason isn't reachable from the SDK result surface this adapter touches, so per your note I didn't block on it — the provider's exception message lands in detail.

Tests

Three additions to Tests_OpenStation_AgentsRunner via the openstation_agent_runner_generate pre-filter seam: a text: null final turn is an error (not an empty success), whitespace-only text is the same failure, and the humanize mapping preserves the extraction detail. php -l clean on all four files.

Honest caveat: I couldn't run npm run test:php locally (no Docker on this machine) — the suite additions mirror the file's existing patterns and CI runs the same scripts.

Evidence for the mechanism (raw provider captures): #517

Follow-up issues for the other three threads from the maintainer reply (provider config filter, async runner shape, replayed placeholder) are being filed separately as requested.

🤖 Generated with Claude Code

Open WordPress Playground Preview

…cess

A final turn with no function calls whose text cannot be extracted
(toText() throws, or the response carries no text part — observed live
when a hard agent task spends the entire max_tokens budget inside a
thinking block) used to be swallowed into a "successful" generation
with text:null. The agents runner then reported success with an empty
answer, bypassing both the transient retry and the error surface, and
the chat rendered "The agent finished without a text answer" with zero
diagnostics.

openstation_ai_client_generate() now returns a WP_Error
(openstation_ai_empty_answer, original extraction detail preserved in
the error data) for that case, which:

- removes the duplicate local check in the search agentic loop (the
  centralized error reaches its existing is_wp_error handling),
- keeps the command follow-up's deliberate degrade: that path matches
  the new code and still falls back to "Done." so a command that DID
  run never surfaces as a failure,
- adds the belt-and-braces guard in the agents runner and a humanized
  message (openstation_agent_empty_answer) explaining the likely
  output-budget exhaustion.

Fixes WordPress#517 (adapter scope per maintainer guidance; provider config,
async runner, and the replayed placeholder are split into follow-up
issues).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@AllTerrainDeveloper AllTerrainDeveloper left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a NIT that can be dismissed if there is no way of checking the error number :)


$human = openstation_agent_humanize_generate_error( $raw );
$this->assertSame( 'openstation_agent_empty_answer', $human->get_error_code() );
$this->assertStringContainsString( 'output budget', $human->get_error_message() );

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey! I'm wondering if this can be replaced with an error number check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants