fix(core): clarify structured output errors after tool steps#1146
Merged
fix(core): clarify structured output errors after tool steps#1146
Conversation
🦋 Changeset detectedLatest commit: acdb306 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This comment has been minimized.
This comment has been minimized.
Contributor
📝 WalkthroughWalkthroughDetects when a structured output is requested but the model fails to emit a final output; Agent now inspects result.output immediately, collects finish/step/tool metadata, and throws a descriptive VoltAgentError (STRUCTURED_OUTPUT_NOT_GENERATED). Unhandled rejection logs also include a hint for this case. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Agent
participant Model
participant Tools
participant FallbackModel
Client->>Agent: generateText(input, { output: schema })
Agent->>Model: call generateText(...)
Note right of Model: may produce tool calls and/or final output
Model->>Tools: (optional) invoke tool(s)
Tools-->>Model: toolResults
Model-->>Agent: GenerateTextResult (lazy result.output getter)
Agent->>Agent: ensureStructuredOutputGenerated(result, output, tools, maxSteps)
alt result.output access throws NoOutputGeneratedError
Agent->>Agent: collect tool call metadata, finishReason, stepCount
Agent-->>Client: throw VoltAgentError(STRUCTURED_OUTPUT_NOT_GENERATED, metadata)
Note over Agent,Client: If configured, Agent may retry using FallbackModel
Agent->>FallbackModel: call generateText(...) (fallback)
FallbackModel-->>Agent: valid structured GenerateTextResult
Agent-->>Client: return parsed structured output
else result.output accessed successfully
Agent-->>Client: return structured output
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/core/src/agent/agent.ts`:
- Around line 1211-1216: The structured-output validation
(ensureStructuredOutputGenerated) currently runs after executeWithModelFallback
returns, which allows STRUCTURED_OUTPUT_NOT_GENERATED to bypass retries; move
the call into the fallback/retry loop so validation happens for each model
attempt: call ensureStructuredOutputGenerated (with result, output, tools,
maxSteps) inside executeWithModelFallback's per-attempt logic (or immediately
before an attempt returns success) and treat STRUCTURED_OUTPUT_NOT_GENERATED as
a retryable failure so the fallback loop can try the next model instead of
returning immediately.
In `@packages/core/src/voltagent.ts`:
- Around line 435-444: The unhandled-rejection hint check only detects
NoOutputGeneratedError (and AI_NoOutputGeneratedError) but not the wrapper
VoltAgentError; update the isNoOutputGeneratedError condition in voltagent.ts to
also consider instances of VoltAgentError whose code ===
"STRUCTURED_OUTPUT_NOT_GENERATED" (the wrapper created by Agent.generateText),
so the hint is shown when the structured-output error is wrapped.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: fc610b7f-5113-4d47-a8fe-b25460cf8fee
📒 Files selected for processing (4)
.changeset/curly-oranges-double.mdpackages/core/src/agent/agent.spec.tspackages/core/src/agent/agent.tspackages/core/src/voltagent.ts
Deploying voltagent with
|
| Latest commit: |
a6ec1ba
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://396ea85d.voltagent.pages.dev |
| Branch Preview URL: | https://fix-issue-1145-structured-ou.voltagent.pages.dev |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/core/src/agent/agent.ts`:
- Around line 1195-1200: The code is retrying STRUCTURED_OUTPUT_NOT_GENERATED
after generateText(...) has already executed step callbacks and side-effecting
tools, which can cause duplicate/stale ConversationBuffer, conversationSteps,
and tool outputs; modify the fallback/retry behavior so that before retrying
when ensureStructuredOutputGenerated(...) raises STRUCTURED_OUTPUT_NOT_GENERATED
you call resetOperationAttemptState(...) on the current OperationContext to
clear per-attempt state (ConversationBuffer, conversationSteps, tool execution
markers) OR instead mark STRUCTURED_OUTPUT_NOT_GENERATED as terminal once any
tool steps have executed; update the logic around
ensureStructuredOutputGenerated, the STRUCTURED_OUTPUT_NOT_GENERATED handling
(referenced at lines with STRUCTURED_OUTPUT_NOT_GENERATED), and where
generateText(...) completes to either resetAttempt state or make the error
non-retryable to prevent re-executing tools on the same OperationContext.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 3149c9c6-e547-43aa-a901-43a5d6e9e63a
📒 Files selected for processing (3)
packages/core/src/agent/agent.spec.tspackages/core/src/agent/agent.tspackages/core/src/voltagent.ts
Comment on lines
+1195
to
+1200
| this.ensureStructuredOutputGenerated({ | ||
| result: response, | ||
| output, | ||
| tools, | ||
| maxSteps, | ||
| }); |
Contributor
There was a problem hiding this comment.
Avoid recovering structured-output failures on a dirty attempt state.
By Line 1195, generateText(...) has already finished running step callbacks, so the failed attempt may have populated ConversationBuffer, conversationSteps, and executed tools. Because Lines 5264-5266 make STRUCTURED_OUTPUT_NOT_GENERATED retryable, the fallback loop can now rerun the request on the same OperationContext without any resetOperationAttemptState(...), which means a recovered attempt can re-execute side-effecting tools and later persist stale tool/result messages from the failed run.
Please either reset per-attempt state before retry/fallback for this error class, or keep this error terminal once tool steps have already executed.
Also applies to: 5264-5266
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/core/src/agent/agent.ts` around lines 1195 - 1200, The code is
retrying STRUCTURED_OUTPUT_NOT_GENERATED after generateText(...) has already
executed step callbacks and side-effecting tools, which can cause
duplicate/stale ConversationBuffer, conversationSteps, and tool outputs; modify
the fallback/retry behavior so that before retrying when
ensureStructuredOutputGenerated(...) raises STRUCTURED_OUTPUT_NOT_GENERATED you
call resetOperationAttemptState(...) on the current OperationContext to clear
per-attempt state (ConversationBuffer, conversationSteps, tool execution
markers) OR instead mark STRUCTURED_OUTPUT_NOT_GENERATED as terminal once any
tool steps have executed; update the logic around
ensureStructuredOutputGenerated, the STRUCTURED_OUTPUT_NOT_GENERATED handling
(referenced at lines with STRUCTURED_OUTPUT_NOT_GENERATED), and where
generateText(...) completes to either resetAttempt state or make the error
non-retryable to prevent re-executing tools on the same OperationContext.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Checklist
Please check if your PR fulfills the following requirements:
Bugs / Features
What is the current behavior?
When
Agent.generateTextis called with a structuredoutput(e.g.Output.object(...)) and tool-enabled runs end without a final output payload, callers can hit a vagueAI_NoOutputGeneratedErrorlater.What is the new behavior?
Agent.generateTextnow validates structured output availability immediately whenoutputis requested.If no final output is generated, it throws a descriptive
VoltAgentErrorwith:code: STRUCTURED_OUTPUT_NOT_GENERATEDstage: response_parsingAlso adds an unhandled-rejection hint in
VoltAgentlogs for theAI_NoOutputGeneratedErrorcase.fixes #1145
Notes for reviewers
packages/core/src/agent/agent.spec.tsfor missing structured output..changeset/curly-oranges-double.md.pnpm --filter @voltagent/core exec vitest run src/agent/agent.spec.ts --reporter=dotSummary by cubic
Improve structured output handling in
Agent.generateTextwhen tool-enabled runs end without a final payload. Calls now fail fast with a clear, retryableVoltAgentError, enabling fallback to the next model. Fixes #1145.outputis requested; if missing, throwVoltAgentError(STRUCTURED_OUTPUT_NOT_GENERATED) with finish reason, steps, max steps, configured tool count, and tool call count, and treat it as retryable for model fallback.VoltAgentlogs for missing structured outputs to guide emitting a final schema-matching response or splitting into two calls.Written for commit acdb306. Summary will update on new commits.
Summary by CodeRabbit
Bug Fixes
Tests