feat(usage): flag truncated/refused/failed finish_reasons as has_error#3071
Merged
chrarnoldus merged 7 commits intomainfrom May 7, 2026
Merged
feat(usage): flag truncated/refused/failed finish_reasons as has_error#3071chrarnoldus merged 7 commits intomainfrom
chrarnoldus merged 7 commits intomainfrom
Conversation
…has_error Introduce a shared zod enum + helper (isErrorFinishReason) for the set of finish_reason / stop_reason / status values we observe across OpenAI chat completions, OpenRouter, Anthropic Messages API, OpenAI Responses API, and Vercel AI SDK style responses. Reasons that indicate truncation, refusal, content filtering, upstream failure, or an interrupted in_progress stream now flip has_error to true in all three usage parsers (chat completions, messages, responses string path). Normal completion reasons (stop, end_turn, tool_use, completed, stop_sequence, tool_calls/tool-calls) and unclassified catch-alls (unknown, other, null) keep has_error driven only by status code and abort signals as before.
Contributor
Author
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (8 files)
Reviewed by gpt-5.5-2026-04-23 · 1,965,868 tokens |
…ror-classification
The zod schema had no runtime consumer — isErrorFinishReason uses a plain Set and the pipeline intentionally keeps finish_reason: string | null end-to-end so unknown upstream values flow through unchanged. Keeping the const arrays gives us the same type-level safety without adding a zod dependency nobody calls.
kilo-code-bot
Bot
changed the title
…ror-classification
The comment was mentioning specific provider APIs, which is noise when the list is sourced from production log distinct values. Keep the comment focused on where the data comes from and the unknown/other policy.
lambertjosh
approved these changes
May 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
apps/web/src/lib/ai-gateway/finishReason.tsmodule with two const arrays (NON_ERROR_FINISH_REASONS,ERROR_FINISH_REASONS) covering every distinctfinish_reason/stop_reason/ Responses APIstatusvalue observed in productionmicrodollar_usagelogs (OpenAI & OpenRouter chat completions, Vercel AI SDK, Anthropic Messages, OpenAI Responses), plus anisErrorFinishReason()helper.isErrorFinishReason(finish_reason)into thehasErrorcalculation of the OpenRouter chat completions stream + string parsers (processUsage.ts), the Anthropic Messages stream + string parsers (processUsage.messages.ts), and the OpenAI Responses stream parser (processUsage.responses.ts). The Responses API string parser'sstatus !== 'completed'rule is left alone — for a non-stream body it is intentionally stricter (flags null/missing status as an error, whichisErrorFinishReasondoes not).stop_reason: "refusal",finish_reason: "length" / "content_filter" / "failed" / "error"etc. is now recorded withhas_error: trueinmicrodollar_usageinstead of silently logged as a success. Unrecognised provider strings,unknown, andotherstay non-error so novel upstream values don't spike error rates.Verification
finish_reasonflows from the SSE event / response body intocoreProps.hasErrorfor both error and non-error values.finish_reason: 'stop', approved snapshots with'end_turn'and'completed') — all classified as non-error, so existing assertions are unaffected.Visual Changes
N/A
Reviewer Notes
lengthandmax_tokensare classified as errors (truncated output is something product/customers usually want surfaced). If you'd rather keep truncation as success, move those two entries fromERROR_FINISH_REASONStoNON_ERROR_FINISH_REASONSand the classifier test still passes unchanged.unknownandotherare non-errors on purpose so brand-new upstream values don't immediately spike error rates; pair with a future Sentry warning on values outside both lists if we want visibility on novel ones.pnpm typecheck/pnpm test/pnpm formatper the request not to run long-running tasks before pushing — please rely on CI.