AI: surface an empty final answer as an error instead of a silent success - #530
Open
juanlentino wants to merge 1 commit into
Open
AI: surface an empty final answer as an error instead of a silent success#530juanlentino wants to merge 1 commit into
juanlentino wants to merge 1 commit into
Conversation
…cess A final turn with no function calls whose text cannot be extracted (toText() throws, or the response carries no text part — observed live when a hard agent task spends the entire max_tokens budget inside a thinking block) used to be swallowed into a "successful" generation with text:null. The agents runner then reported success with an empty answer, bypassing both the transient retry and the error surface, and the chat rendered "The agent finished without a text answer" with zero diagnostics. openstation_ai_client_generate() now returns a WP_Error (openstation_ai_empty_answer, original extraction detail preserved in the error data) for that case, which: - removes the duplicate local check in the search agentic loop (the centralized error reaches its existing is_wp_error handling), - keeps the command follow-up's deliberate degrade: that path matches the new code and still falls back to "Done." so a command that DID run never surfaces as a failure, - adds the belt-and-braces guard in the agents runner and a humanized message (openstation_agent_empty_answer) explaining the likely output-budget exhaustion. Fixes WordPress#517 (adapter scope per maintainer guidance; provider config, async runner, and the replayed placeholder are split into follow-up issues). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 7, 2026
AllTerrainDeveloper
approved these changes
Aug 8, 2026
AllTerrainDeveloper
left a comment
Collaborator
There was a problem hiding this comment.
There is a NIT that can be dismissed if there is no way of checking the error number :)
|
|
||
| $human = openstation_agent_humanize_generate_error( $raw ); | ||
| $this->assertSame( 'openstation_agent_empty_answer', $human->get_error_code() ); | ||
| $this->assertStringContainsString( 'output budget', $human->get_error_message() ); |
Collaborator
There was a problem hiding this comment.
Hey! I'm wondering if this can be replaced with an error number check
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scoped to the adapter change green-lit in #517 (plus the belt-and-braces runner guard).
What
A final turn with no function calls whose text cannot be extracted —
toText()throws, or the response carries no text part (observed live: a hard agent task spends the entiremax_tokensbudget inside a thinking block,stop_reason: "max_tokens") — was swallowed into a "successful" generation withtext: null. The agents runner then reported success with an empty answer, bypassing both the transient retry and every error surface.openstation_ai_client_generate()now returnsWP_Error( 'openstation_ai_empty_answer' )for that case, with the original extraction detail preserved in the error data.Consumers, walked
! is_string( $text )check (the oldopenstation_ai_emptybranch) is removed — the centralized error reaches the loop's existingis_wp_errorhandling at both generate sites. Net behavior change: an empty-string final turn now reportsopenstation_ai_empty_answerinstead of falling through toopenstation_ai_result_parse."Done."with thefallbackflag, so a command that DID run never surfaces as a failure, and noopenstation_ai_search_errorfires for it (same as today).''), plus a humanized mapping (openstation_agent_empty_answer) telling the user the model likely spent its output budget reasoning. The new code is deliberately NOT in the transient-retry signature list — the failure is task-shaped, not a flap; happy to add it if you'd rather retry once.stop_reasonisn't reachable from the SDK result surface this adapter touches, so per your note I didn't block on it — the provider's exception message lands indetail.Tests
Three additions to
Tests_OpenStation_AgentsRunnervia theopenstation_agent_runner_generatepre-filter seam: atext: nullfinal turn is an error (not an empty success), whitespace-only text is the same failure, and the humanize mapping preserves the extraction detail.php -lclean on all four files.Honest caveat: I couldn't run
npm run test:phplocally (no Docker on this machine) — the suite additions mirror the file's existing patterns and CI runs the same scripts.Evidence for the mechanism (raw provider captures): #517
Follow-up issues for the other three threads from the maintainer reply (provider config filter, async runner shape, replayed placeholder) are being filed separately as requested.
🤖 Generated with Claude Code