Difference in models results

[juanmichelini]

Please see

  | private endpoint | public endpoint
Issue resolution: | 71.40% | 74.60%
Testing: | 67.00% | 70.40%
Information Gathering: | 63.60% | 74.50%
Greenfield development: | 50.00% | 25.00%

Both endpoints contain the same model Kimi-k2.6 but the private endpoint si consistently returning lower results.

Let's compare conversation and logs to determine why:

Private Issue resolution:"full_archive": "https://results.eval.all-hands.dev/swebench/litellm_proxy-accounts-graham-openhands-deployments-mghcd1dc/25200072711/results.tar.gz",

Public issue resolution "full_archive": "https://results.eval.all-hands.dev/swebench/litellm_proxy-moonshot-kimi-k2-6/25007210109/results.tar.gz",

Private Testing "full_archive": "https://results.eval.all-hands.dev/swtbench/litellm_proxy-accounts-graham-openhands-deployments-mghcd1dc/25328867381/results.tar.gz",
Public Testing "full_archive": "https://results.eval.all-hands.dev/swtbench/litellm_proxy-moonshot-kimi-k2-6/24901879531/results.tar.gz",

Private Information Gathering "full_archive": "https://results.eval.all-hands.dev/gaia/litellm_proxy-accounts-graham-openhands-deployments-mghcd1dc/25223608404/results.tar.gz",
Public Information Gathering "full_archive": "https://results.eval.all-hands.dev/gaia/litellm_proxy-moonshot-kimi-k2-6/25710749383/results.tar.gz",

Private Greenfield "full_archive": "https://results.eval.all-hands.dev/commit0/litellm_proxy-accounts-graham-openhands-deployments-mghcd1dc/25294380732/results.tar.gz",
Public Greenfield "full_archive": "https://results.eval.all-hands.dev/commit0/litellm_proxy-moonshot-kimi-k2-6/25710683155/results.tar.gz",

[juanmichelini]

@OpenHands please take your time and analyze why this happens. Do not create a PR, instead answere here in this issue #707

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[juanmichelini]

Analysis of the public vs private endpoint gap

This analysis was performed by an AI agent (OpenHands) on behalf of @juanmichelini.

I downloaded all 8 archives and compared metadata + per-instance metrics. The headline finding is that the two endpoints look identical on the model side (same Kimi-k2.6) but very different on the LiteLLM-client side, and that explains most of the gap. There is also a non-trivial amount of run-to-run variance hiding inside the 3-percentage-point spreads.

1. The two configs are not actually identical

From metadata.json in each archive (same field for every benchmark):

Field	Public	Private
llm.model	litellm_proxy/moonshot/kimi-k2.6	litellm_proxy/accounts/graham-openhands/deployments/mghcd1dc
llm.max_input_tokens	262144	null
llm.max_output_tokens	131072	null
openhands_sdk_version	1.18.1 – 1.22.0	1.19.0 – 1.20.0

Both endpoints route to Kimi-k2.6 on the proxy. But the model name string itself is what LiteLLM and the SDK use as the key into their model-metadata registry (context window, pricing, provider-specific request shaping, etc.). moonshot/kimi-k2.6 is a known entry; accounts/graham-openhands/deployments/mghcd1dc is an opaque deployment slug that LiteLLM has no entry for.

The downstream consequences of that mismatch are visible in every single LLM call.

2. The private endpoint loses model awareness on every call

Inspecting the token_usages records inside each output.jsonl (same instance django__django-13513 for direct comparison):

	Public	Private
context_window reported by litellm	262144 (on all 37k calls)	0 (on all 40k calls)
Per-turn pricing	populated (costs[] non-empty)	empty — total cost reports as $0.00

Aggregating across the full benchmarks:

Benchmark	n	turns/inst (pub→priv)	completion_tokens/inst (pub→priv)	cache hit % (pub→priv)	ctx_window (pub→priv)
SWE-Bench Verified	500	74.9 → 81.7 (+9%)	25,245 → 34,255 (+36%)	97.6% → 96.3%	262144 → 0
SWT-Bench	433	48.6 → 51.9 (+7%)	12,499 → 16,859 (+35%)	96.9% → 95.3%	262144 → 0
GAIA	~163	26.0 → 28.0 (+8%)	14,177 → 19,383 (+37%)	93.5% → 93.3%	262144 → 0
Commit0	8 vs 15	82.6 → 200.6 (+143%)	47,504 → 194,135 (+309%)	97.1% → 96.4%	262144 → 0

The pattern is the same across all four benchmarks. The private endpoint reports context_window = 0 on every call, and the agent burns ~35% more completion tokens per instance to reach a worse answer. The model is producing more output but the per-turn quality is lower, so the agent has to iterate more and still ends up unresolved.

Wall-clock per instance corroborates this: private is roughly 2× faster per call (e.g. SWE-Bench 583s vs 1059s average duration), so the same task fits more low-quality turns into the same time budget.

3. Why this happens (most likely)

Because the private model identifier is an arbitrary deployment slug:

• LiteLLM cannot look up the model's max_input_tokens / max_output_tokens / context window → these are reported as null / 0, and any client-side context-budgeting heuristics that depend on them silently no-op.
• Provider-specific request transformations don't trigger. With drop_params: true, params the client thinks are unsupported are silently dropped. The reasoning controls (reasoning_effort: "high", extended_thinking_budget: 200000, enable_encrypted_reasoning: true) are almost certainly being treated differently — and worth verifying directly against the proxy access logs.
• The server-side deployment defaults (sampling, possibly a different quantization or replica pool than the curated Moonshot deployment) take over instead of the params the client intended to send.

Both of those alone are consistent with: more tokens, more turns, faster per call, lower accuracy.

4. How much of the gap is real vs noise?

• temperature: 1.0 on both runs.
• At 500 instances and ~74% accuracy, the SE of a single run is ~√(0.74·0.26/500) ≈ 2.0%. The 3.2pp SWE-Bench gap and the 3.4pp SWT-Bench gap are right at the edge of one-sigma — i.e. likely a mix of real signal + noise. Cross-tabulating the SWE-Bench resolved IDs: 34 instances were solved by public-only, but 18 instances were solved by private-only (so the cross-variance is 52/500 ≈ 10%, an upper bound on what is purely sampling noise).
• Commit0 50% vs 25% is not comparable. Public ran with eval_limit=20 and only 8 instances completed; private ran with eval_limit=500 and 15 completed. Sample sizes are tiny and not the same population. Don't read anything into that number.
• GAIA's 10.9-point gap on eval_limit=200 (public) vs eval_limit=500 (private) is the most suspicious. At those sizes, an SE is ~3.5%, so an 11pp gap probably is a real systematic difference — and the per-instance metrics (more turns, more tokens, ctx_window=0) line up with that.

5. Recommendations

In rough order of impact / ease:

1. Single highest-leverage fix: in the proxy config for the private deployment, register the deployment under the canonical model family so LiteLLM picks up the correct model_info (max_input_tokens, max_output_tokens, mode='chat', cost table). Once the SDK sees a non-zero context_window, all the model-aware code paths come back to life.

2. Or (faster, client-side only): in the private LLM config JSON, explicitly set

   "model_canonical_name": "moonshot/kimi-k2.6",
   "max_input_tokens": 262144,
   "max_output_tokens": 131072

   This forces the SDK to use the right model metadata even when LiteLLM can't infer it from the deployment name.

3. Verify reasoning params actually reach the deployment. Send one completion through the private route and grep the proxy access log for reasoning_effort and the extended-thinking budget. If they're being stripped, the private deployment is effectively running Kimi-k2.6 without extended thinking — which is consistent with the "more tokens, worse answers" signature.

4. Re-run with temperature=0 (or a fixed seed) once to separate systematic delta from sampling noise. The current numbers can't cleanly tell those apart.

5. Re-run Commit0 with matching eval_limit on both endpoints before drawing any conclusions from it.

If you want, I'm happy to follow up with a PR that adds the model_canonical_name override path to the benchmark configs (item 2) — but as requested, no PR for now.

[openhands-ai]

Summary

The user asked me to analyze why the Kimi-k2.6 model produces systematically lower benchmark results on the private endpoint vs the public endpoint, and explicitly requested no PR — to answer in the issue instead.

What I did

1. Downloaded all 8 result archives (4 benchmarks × public/private) totalling ~2.2 GB.
2. Compared metadata.json, output.report.json, cost_report.jsonl, and per-instance token_usages across all 8 runs.
3. Posted the analysis as a comment on issue Difference in models results #707: Difference in models results #707 (comment)
4. Cleaned up all temp files in /tmp/analysis.

Checklist of the user's instructions

• ✅ Took time to analyze the cause (downloaded archives, compared metadata + per-call metrics across all 4 benchmarks).
• ✅ Answered in issue Difference in models results #707 as a comment.
• ✅ Did not create a PR.
• ✅ Included AI-disclosure line at the top of the comment.
• ✅ No code changes to the repo, no branches created, no commits.

Core finding (re-stated)

The two endpoints route to the same underlying Kimi-k2.6 model, but use different model-name strings:

• Public: litellm_proxy/moonshot/kimi-k2.6 (known to LiteLLM's model registry)
• Private: litellm_proxy/accounts/graham-openhands/deployments/mghcd1dc (opaque deployment slug, unknown to LiteLLM)

That single difference cascades into:

• max_input_tokens/max_output_tokens = null on private (262144/131072 on public)
• context_window = 0 on every single LLM call on private (262144 on public)
• Provider-specific request transformations (incl. reasoning params) likely don't fire the same way
• Empirically, on the same instances, private uses ~35% more completion tokens and ~9% more turns per task, runs ~2× faster per call, and produces lower accuracy — the classic "more output, worse per-turn quality" signature.

I also flagged the noise floor (temp=1.0, n=500 → ~2% SE; 52/500 instances cross-differ on SWE-Bench), and called out that the Commit0 50%-vs-25% gap is statistically meaningless because the two runs had different eval_limit and only 8 vs 15 instances completed.

Recommendations posted in the comment, in priority order: (1) register the private deployment under the canonical model family in the proxy config; (2) or set model_canonical_name: "moonshot/kimi-k2.6" plus explicit max_input_tokens/max_output_tokens in the private LLM config; (3) verify reasoning params reach the deployment via proxy access logs; (4) re-run with temperature=0 or fixed seed to separate signal from sampling noise; (5) re-run Commit0 with matching eval_limit.

No changes to the repository — everything lives in the issue comment as requested.