Phase 2: evals harness + multi-provider lineup (Anthropic, OpenAI, Gemini) - #2
Merged
Conversation
…ble reports packages/evals: demo invoice/receipt schemas, dataset ground-truth mappers (CORD-v2 receipts, DocILE invoices), normalized value comparison, bbox grounding via IoU against annotated regions, cost tracking, and report generation from recorded runs only. The 25-receipt half is pinned by id + sha256 in data/manifest.json (fetched and verified from the canonical HF parquet at a fixed revision); the DocILE invoice half awaits the access token — loader and curation are ready. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The eval harness only ran Anthropic models. Add OpenAI and Google Gemini as selectable providers so runs can use a cheaper lineup. - models.ts: per-provider catalog, three vision-capable tiers each with list pricing sourced inline (2026-07). selectProviders/benchmarkModels are env-injectable and unit-tested. - Default: run every provider whose API key is set (ANTHROPIC_API_KEY / OPENAI_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY); setting only one benchmarks that provider alone. EVAL_PROVIDERS pins an explicit subset. - Backward compatible: only ANTHROPIC_API_KEY set → the original 3 models. - Add @ai-sdk/openai and @ai-sdk/google; 10 new tests (100 total pass). - Sync evals/core READMEs, ROADMAP, CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the Phase 2 eval harness to
main, plus provider selection so runs aren't locked to Anthropic.Eval harness (
packages/evals)docs/benchmark.md+ README table) generated exclusively from serialized run records — no hand-entered numbers.Multi-provider lineup
models.tsis now a per-provider catalog — Anthropic, OpenAI, Google Gemini — three vision-capable tiers each, with list pricing sourced inline (2026-07).ANTHROPIC_API_KEY/OPENAI_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY); setting only one benchmarks that provider alone — a cheaper way to iterate than the full Anthropic lineup.EVAL_PROVIDERSpins an explicit subset (e.g.openai,google); missing keys / unknown names error clearly.ANTHROPIC_API_KEYset → the original three models.@ai-sdk/openaiand@ai-sdk/google.Tests
100 passing (53 core + 47 evals), including 10 new
models.test.tscases for provider selection. Monorepo typecheck clean.Pricing sources: OpenAI · Google Gemini · Anthropic (platform.claude.com/docs/en/pricing).
🤖 Generated with Claude Code