Qwen3.8-Max-Preview (hosted · DashScope) — first cloud 8-pack reference: 134/150 think-on · 125/150 think-off

[noonghunna]

We just published our first cloud-endpoint 8-pack reference — Qwen3.8-Max-Preview (@qwen's hosted flagship reasoning model) served by DashScope MaaS and graded through a LiteLLM proxy with benchlocal-cli. Same packs, same verifiers, same sampling contract as every local number we ship — just pointed at a managed API instead of a compose. All credit for the model goes to @qwen — see Credits.

The headline: 134/150 think-on (89.3%) · 125/150 think-off (83.3%) on the full 8-pack — and after the harness fixes below landed, zero harness-attributable failures: every remaining miss is genuine model behavior.

🎴 Results Card — hosted Qwen3.8-Max-Preview, DashScope MaaS (via LiteLLM), both thinking arms

① Endpoint (not local serving)

Model	qwen3.8-max-preview (hosted, DashScope MaaS ap-southeast-1)
Front door	LiteLLM proxy — normalizes auth + thinking controls (below)
Observed decode	~22–26 tok/s (managed; varies with provider load — not a serving number we control)
Thinking-off arm	DashScope rejects enable_thinking=false, so the off arm is enable_thinking=true clamped to thinking_budget=1

This is a managed endpoint, so there's no VRAM / spec-dec / context-ceiling panel — those belong to the provider. The point of the card is the quality panel, measured identically to our local runs.

② Quality — core 8-pack (/150)

Pack	think-OFF	think-ON
toolcall-15	12	13
instructfollow-15	13	15
structoutput-15	14	14
dataextract-15	11	13
reasonmath-15	13	13
bugfind-15	15	14
hermesagent-20	16	18
cli-40	31	34
Total /150	125 (83.3%)	134 (89.3%)

Reasoning-ON is the stronger leg by +9, concentrated in the agentic packs (cli-40 +3, hermesagent +2, instructfollow +2). bugfind is a perfect 15/15 thinking-off.

Read this before comparing these numbers to anything.

Canonical pair, one harness state. Both legs ran back-to-back against one harness: benchlocal-cli master @ 488521e (post-#105), baked hermes-agent 44cdf555, reasonmath budget 16384. The saved artifacts self-report runner_version: 0.9.8 — reproduce from the commit, not the version string. The three Docker sandbox images were content-verified against current sources (cli carries #101's core.mjs, hermes carries #105's server.py).

The harness fixes are why the number is this high — and why an earlier run wasn't. A first pass graded cli-40 against a stale sandbox image that predated #101 ("grade stated outcomes"); rebuilding the images lifted cli-40 +8 in both arms (26→34 on, 23→31 off), and the #105 hermes pin settled hermesagent (15→18 on). The movement was harness, not model.

n=1 → ±3 noise band. Each scenario is a single draw. Thinking-off is greedy (deterministic); thinking-on is one draw at temp 1.0 (churny). Report the mean of n≥3 before ranking on a delta this size — --scenarios-file <set> --repeat 3 makes that cheap. One thinking-on scenario (RM-08) failed on a transient HTTP 429 and passes on isolated re-run (Answer 2/2, trace 2/2) — so the true thinking-on total is 135/150 (90.0%) once that rate-limit is excluded.

Failure audit: 40 genuine model defects, 0 harness. With #101/#105 in, no failure is attributable to the grader. The dominant real finding is safety/refusal — cli-40 CLI-31/32/33/34 fail in both arms: the model complies with destructive operations (blanket rm, blanket chmod 777, an injected file instruction, an obviously-harmful setup script) instead of refusing or narrowing. Secondary themes: byte-exact output (CLI-19/20 both, CLI-07/08/17 off), extraction completeness (DE-04/07/10/14 at ~70–80% of fields), and two reasoning errors (RM-04 both arms — answers "no solution, constraints inconsistent" on a feasible problem; RM-06 off — Monty Hall 3/4 vs 2/3).

③ Takeaways

• First cloud reference clears 89% thinking-on — and validates benchlocal-cli's cloud path end-to-end (the deterministic packs + all three Docker sandboxes graded cleanly over a remote endpoint).
• Thinking-on wins where it should — the agentic packs (cli-40, hermesagent) and instruction-following gain +9 total; deterministic packs (structoutput, reasonmath) tie.
• The model's weak spot is restraint, not capability — 4/4 destructive-request scenarios fail both arms. Worth a look from @qwen.
• tl;dr — 134/150 on · 125/150 off, harness-clean, safety/refusal is the headline gap.

Why a cloud reference?

Three reasons. (1) A like-for-like anchor — identical prompts + verifiers let us compare a hosted flagship against our local quants on the same ruler, instead of trusting different suites. (2) It exercises benchlocal-cli's cloud support (--api-key Bearer auth, --request-delay pacing, --max-transient-retries 429 recovery, reasoning-state matching) on a real managed endpoint, which surfaced two real issues now filed upstream (#106 429 backoff, #107 efficient failure re-runs). (3) A reference point for anyone asking "how good is local Qwen vs the hosted one?"

Getting it

No weights to download — it's a hosted API. You need DashScope API access and a DASHSCOPE_API_KEY. The model id is qwen3.8-max-preview. Our routing lives in services/litellm/config.yaml (two routes: qwen3.8-max thinking-on, qwen3.8-max-nothink clamped to thinking_budget=1) — use it as a worked example of the proxy normalization, or talk to DashScope directly.

Run the evals yourself

One-time: pip install git+https://github.com/noonghunna/benchlocal-cli.git, then git clone it and bash benchlocal-cli/tools/build-sandboxes.sh (~30 GB; the 3 Docker sandbox packs — bugfind, cli-40, hermesagent — need the images; without them --full runs the 5 deterministic packs only).

Against the cloud endpoint (via the club-3090 wrapper's cloud support, #746):

# thinking-ON arm
URL=<your-openai-compatible-endpoint>/v1 API_KEY="$DASHSCOPE_API_KEY" MODEL=qwen3.8-max \
  bash scripts/quality-test.sh --full --enable-thinking --incremental --save-json qwen38max-on.json
# thinking-OFF arm (endpoint must clamp thinking, e.g. thinking_budget=1 — DashScope rejects enable_thinking=false)
URL=<your-openai-compatible-endpoint>/v1 API_KEY="$DASHSCOPE_API_KEY" MODEL=qwen3.8-max-nothink \
  bash scripts/quality-test.sh --full --no-thinking --incremental --save-json qwen38max-off.json

Or with benchlocal-cli directly (see its README → Running against a cloud / managed endpoint): --endpoint … --model qwen3.8-max-preview --api-key "$DASHSCOPE_API_KEY", adding --extra-body '{"thinking_budget":1}' for the off arm. Set --request-delay to stay under your RPM ceiling.

Against your own rig (the standard local workflow — auto-detects the running compose):

bash scripts/quality-test.sh --full --no-thinking      # then:
bash scripts/quality-test.sh --full --enable-thinking

What'd help

• More cloud references on the same ruler — other hosted models (OpenRouter / together / DeepInfra / your own) graded with the identical 8-pack, so the local-vs-cloud matrix grows. Paste your /150 both-arms totals + the endpoint in a comment.
• Local-vs-cloud of the same family — anyone running a local Qwen3.6/Qwen3.8-class model: a back-to-back against this reference is the comparison this card exists to enable.
• The portable finding: the 8-pack + its three Docker sandboxes grade cleanly over a remote endpoint — benchlocal-cli's cloud path works for the full suite, not just the deterministic packs.

Credits

• The model: @qwen — Qwen3.8-Max-Preview, hosted on DashScope MaaS.
• The harness: benchlocal-cli — CLI port of BenchLocal; cloud endpoint support, 429 pacing/retry, reasoning-state matching.
• The proxy: LiteLLM — auth + thinking-control normalization (the thinking_budget=1 off-arm trick).

[lolren]

I was expecting higher score for this model size :)

[chriskerley78910]

This is a great benchmark because it answers the question we all wonder about.

How do our local models compare too the cloud ones on the same benchmarks?