You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The product is able to request Cloud providers such as Gemini, Claude or OpenAI throughout a provider adapter layer that auto-routes by detected provider and model. But every responses are normalized back into an OpenAI-chat shape (legacy). It lead to information loss and lossy round-trip.
Information loss (the big one)
OpenAI-chat shape can't hold what other providers return. You normalize → you discard:
If a future benchmark wants to score reasoning tokens used or refusal rate or cache hit rate, that signal is already thrown away at the adapter boundary. You'd be re-plumbing.
Lossy round-trip → silent empties
Mapping mismatches don't error, they yield empty answer_text. Gemini SAFETY block, Anthropic refusal, Responses reasoning-only output with no text → all normalize to "". Assertion fails with "missing text" —
misleading: looks like model got it wrong, actually got filtered/refused. Debugging points at wrong layer.
Usage-accounting skew
{prompt_tokens, completion_tokens} is OpenAI's vocabulary. Anthropic splits cache tokens; Responses counts reasoning tokens in output. Force them into 2 fields → leaderboard cost/token metrics become wrong or
incomparable across providers. A "tokens" column that means different things per provider is worse than no column.
Frozen to OpenAI's model
Every provider divergence becomes adapter glue. When OpenAI changes chat shape (they already forked to Responses), your "canonical" shape is the legacy one — you're normalizing OpenAI-new into OpenAI-old too.
Canonical = whoever you copied, and you copied the one that's already moving.
Streaming semantics flatten
SSE deltas differ: OpenAI choices[].delta.content, Anthropic content_block_delta, Gemini chunked candidates. Normalizing to "append text" loses event types (tool-call deltas, reasoning deltas, message-stop).
Fine if you only show text; lossy if you ever want event-level timing/inspection.
Currently, what has been done to mitigate:
Keep raw payload alongside normalized. You already store raw_responses (benchmark-runner.ts:871). Don't drop provider raw — normalization is a view, raw is the record. Future metrics recoverable without
re-running.
Add finish_reason + normalize_status to the normalized shape. One enum field distinguishes "empty because refusal/safety/length" from "empty answer". Kills downside Adding Playwright tests to check UI. #2.
Extend usage with optional fields (reasoning_tokens?, cache_read_tokens?) rather than crushing to 2. Leaderboard can ignore now, use later.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
The product is able to request Cloud providers such as Gemini, Claude or OpenAI throughout a provider adapter layer that auto-routes by detected provider and model. But every responses are normalized back into an OpenAI-chat shape (legacy). It lead to information loss and lossy round-trip.
OpenAI-chat shape can't hold what other providers return. You normalize → you discard:
If a future benchmark wants to score reasoning tokens used or refusal rate or cache hit rate, that signal is already thrown away at the adapter boundary. You'd be re-plumbing.
Mapping mismatches don't error, they yield empty answer_text. Gemini SAFETY block, Anthropic refusal, Responses reasoning-only output with no text → all normalize to "". Assertion fails with "missing text" —
misleading: looks like model got it wrong, actually got filtered/refused. Debugging points at wrong layer.
{prompt_tokens, completion_tokens} is OpenAI's vocabulary. Anthropic splits cache tokens; Responses counts reasoning tokens in output. Force them into 2 fields → leaderboard cost/token metrics become wrong or
incomparable across providers. A "tokens" column that means different things per provider is worse than no column.
Every provider divergence becomes adapter glue. When OpenAI changes chat shape (they already forked to Responses), your "canonical" shape is the legacy one — you're normalizing OpenAI-new into OpenAI-old too.
Canonical = whoever you copied, and you copied the one that's already moving.
SSE deltas differ: OpenAI choices[].delta.content, Anthropic content_block_delta, Gemini chunked candidates. Normalizing to "append text" loses event types (tool-call deltas, reasoning deltas, message-stop).
Fine if you only show text; lossy if you ever want event-level timing/inspection.
Currently, what has been done to mitigate:
re-running.
Must plan a refactor in a next version.
Beta Was this translation helpful? Give feedback.
All reactions