fix(benchmark): gemini stream-json content/stats parse (#2159)#2221
Open
time-attack wants to merge 4 commits into
Open
fix(benchmark): gemini stream-json content/stats parse (#2159)#2221time-attack wants to merge 4 commits into
time-attack wants to merge 4 commits into
Conversation
GeminiAdapter was reading message.text and result.usage, so current CLI content/stats events produced empty $0 success rows. Accept content with an assistant role guard, stats token fallbacks, init model, and treat empty exit-0 output as an error (garrytan#2159). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
Exercise resultFromGeminiStream against current stream-json fixtures (including empty-success hardening). Recognize GEMINI_API_KEY and map IneligibleTierError to auth — personal OAuth free-tier is no longer supported by gemini CLI. Co-authored-by: Cursor <cursoragent@cursor.com>
Pass --skip-trust so temp/untrusted workdirs work in headless benchmarks (mirrors Codex --skip-git-repo-check). Revert VERSION, CHANGELOG, and package.json bumps — leave release bookkeeping to maintainers / /ship, matching community PR practice. Co-authored-by: Cursor <cursoragent@cursor.com>
time-attack
changed the title
Author
|
basically gemini updated their cli so we had to adjust to that. Google has stated they will switch to anti gravity for cli and depreciate later but the current version is not compatible with model benchmark, when google does the full migration we can handle that |
Author
|
/trunk merge |
|
An error occurred while submitting your PR to the queue: |
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2159:
GeminiAdapterwas reading legacymessage.text/result.usage, so current gemini CLIcontent+statsevents produced empty output, 0 tokens, and no error — a silent $0 success that mis-ranked gemini in benchmark tables.message.contentwhenrole === 'assistant'(ignore user-prompt echoes); keep legacytextfallbackresult.statswith legacyusagefallbacks; model frominiterrorrow (not a $0 success)--skip-trustfor headless/temp workdirs (required by current gemini CLI)GEMINI_API_KEY; map IneligibleTier / auth failures cleanlyNo
VERSION/CHANGELOG/package.jsonbump — leaving release metadata to maintainers.Test plan
bun test test/helpers/providers/gemini.test.ts(+ related free tests)GeminiAdapter.run()instances → realPONG/ALPHA/OK READY, positive tokens, no errorgstack-model-benchmark --prompt 'Reply with exactly the word: PONG' --models gemini --output jsonCloses #2159