🧠 VibeThinker-3B — 97% HumanEval+ at 3B on a single 3090, but no tool-calling (🐣 incubating)

[noonghunna]

🐣 Incubating — a verifiable-reasoning specialist, NOT a coding agent: it always reasons (<think>), has no tool-calling, and fails the standard functional gate by design (verify-full 5/9). Hidden from switch.sh --list (see --list --all); --force to launch. Posted because the findings are interesting — cross-rig numbers welcome (see "What'd help").

---

We just wired up llamacpp/vibethinker-3b-single — WeiboAI's VibeThinker-3B, a 3B verifiable-reasoning model (SFT+RL on Qwen2.5-Coder-3B). All credit for the model goes to WeiboAI; prithivMLmods for the Q8 GGUF; Qwen for the Qwen2.5-Coder-3B base — see Credits.

The headline: a 3B that scores 97% HumanEval+ / 83% LiveCodeBench-v6 / 100% GSM-Symbolic at ~166 TPS in ~6.2 GB — and a sharp quant finding: fp8 weights break this model, but Q8 doesn't.

⚠️ Don't reach for this to write code in an agent

With no tool-calling, it can't drive aider / opencode / cli-agent loops, edit files, or run multi-step tool workflows — it is not a coding-assistant / IDE model. For that, use a tool-calling config (the Qwen3.6-27B or Carnice composes).

Its lane is one-shot verifiable problem-solving: competition math and algorithmic coding (write-the-function / solve-the-problem, where a sandbox checks the answer). Think "competition solver," not "pair programmer."

🎴 Results Card — 1× RTX 3090, llama.cpp server-cuda, temp 0.6

① Serving — two engines

Config	Spec-dec	KV / ctx	decode TPS	prefill	VRAM
llamacpp/vibethinker-3b-single ⭐	none	q8_0 · 131K	166	~6,630 t/s	~6.2 GB
vllm/vibethinker-3b-single	none	fp8 · 131K	~110	—	~9.8 GB

llama.cpp Q8 wins on every axis. TTFT 26–43 ms; -b 4096 -ub 2048 = +20% prefill (-ub is the lever; -b 8192 adds nothing). No MTP head in the GGUF (plain qwen2 conversion) → no spec-dec, not needed at this speed.

② Quality — benchlocal, verifier-graded, temp 0.6, thinking-on

Core 8-pack (the packs an always-reasoning, no-tools model can run):

Pack	Score
instructfollow-15	15/15 (100%)
reasonmath-15	12/15 (80%)
structoutput-15	12/15 (80%)
dataextract-15	6/15 (40%)
toolcall-15 · hermesagent-20 · cli-40	N/A — no tool-calling

Reasoning + one-shot coding (sandbox executes the model's output — no tool-calls needed):

Pack	Score
gsm-symbolic-30	30/30 (100%) ⭐
humaneval-plus-30	29/30 (97%) ⭐
lcb-v6-30	25/30 (83%) (2 losses = token_limit on the hardest)

dataextract (40%) is a genuine ceiling, not a config artifact — a full temp sweep {0: 27%, 0.6: 40%, 1.0: 27%} can't lift it; the model emits valid JSON but mis-extracts values. It's a reasoner, not an extractor (as the authors state). The sweep also confirmed temp 0.6 is the sweet spot (beats greedy and the card's 1.0).

③ Takeaways

• Frontier-level one-shot reasoning & coding at 3B — 100% GSM-Symbolic, 97% HumanEval+, 83% LCB-v6, on our rig (matches the authors' ~96% LeetCode). For its size, exceptional.
• But not a coding agent — no tool-calling → useless for IDE/agent loops. One-shot solve ≠ edit-and-run. This is the line to lead with.
• Q8 keeps it; fp8 weights break it. vLLM's fp8 weight-quant (Marlin W8A16) produced non-terminating empty output on this quant-sensitive 3B; llama.cpp's Q8_0 is near-lossless, quality intact. KV quant (q8_0/fp8) is fine either way — it's weight quant that's fragile on small models.
• Tiny footprint, full context — the whole 131K window in ~6.2 GB; light enough to co-reside with a 27B.
• Always reasons → give it a generous max_tokens (8K–40K). A small budget truncates it mid-<think> with no answer — the single most important client-side setting (and why it's incubating).

Requirements

• 1× 24 GB GPU. ~3.3 GB Q8 weights + ~2.6 GB q8_0 KV @131k ≈ 6.2 GB.
• mainline llama.cpp (ghcr.io/ggml-org/llama.cpp:server-cuda, auto-pulled).

Getting it / Run it

git clone https://github.com/noonghunna/club-3090 && cd club-3090
WEIGHTS=prithivmlmods-q8 ./scripts/setup.sh vibethinker-3b      # fetch the Q8 GGUF
./scripts/switch.sh --force llamacpp/vibethinker-3b-single      # --force: incubating status

Serves an OpenAI-compatible API on :8075 (model vibethinker-3b). It reasons inline in content (llama.cpp's parser doesn't split this model's <think>); the answer follows </think>. Set a generous max_tokens (8K+).

📖 New to the stack? The FAQ covers weights location (MODEL_DIR), switching models, and offline installs.

What'd help

Cross-rig numbers (other 3090s / 4090s — fit, TPS), and whether reasoning/coding holds at lower quants (Q4/Q5 would shrink it under ~4 GB). The portable finding worth re-testing on any small reasoning model: fp8 weight-quant can silently break a 3B where Q8 GGUF doesn't.

Credits

• The model: WeiboAI — VibeThinker-3B, the verifiable-reasoning SFT+RL fine-tune.
• The Q8 GGUF: prithivMLmods — VibeThinker-3B-GGUF.
• The base: Qwen — Qwen2.5-Coder-3B.
• The engine: llama.cpp.

[noonghunna]

📋 Update — one-shot language coverage

The HumanEval+ (97%) / LCB-v6 (83%) numbers above are Python-centric — so "great at coding" really means great at Python. To see how far the breadth goes, I ran a quick one-shot cross-language probe (merge sort → 1 2 3 5 5 6 9, compiled + executed where a toolchain exists; n=1, temp 0.6):

Language	Result	Detail
Python	✅	compiled + correct
JavaScript	✅	compiled + correct
C++	✅	compiled + correct
Go	❌	strings.Join(sorted, " ") on a []int — needs []string (algorithm correct, stdlib idiom wrong)
Java	❌	String.join(" ", int[]) won't compile + an off-by-one half-boundary
Rust	❌	hallucinates Python list-comprehensions (vec![arr[i] for i in 0..mid]) — doesn't really know Rust
TypeScript	❌	stubbed it (function mergeSort(arr): number[] { // ... }) — never implemented
Bash	⚪	rambled past the token budget (the always-reasoning trait) — no clean output

The pattern:

• Strong & clean: Python · JavaScript · C++ — the high-frequency + competitive-programming trio.
• Weak: Go & Java — both fail the same idiom (string-joining raw ints without converting). The merge-sort logic is correct; the language-specific stdlib call is wrong.
• Weakest: Rust — fundamental syntax errors (Python-isms).

It's a Qwen2.5-Coder-3B at heart: deep where training data is densest, narrow elsewhere. Practical read — a superb one-shot Python / C++ / JS solver, not a polyglot. Pair this with the disclaimer up top: great competition coder in a few languages, not a coding agent in any.

(n=1 one-shot — individual cells carry temp-0.6 variance; the tiering Python/JS/C++ ≫ Go/Java ≫ Rust/TS is the robust signal.)

[noonghunna]

Update — after reading WeiboAI's official model card more closely (the benchmark numbers are inside chart images, easy to miss): the post above underweights the math and overstates the coding. Correcting both, with the card's own numbers.

The real headline is competition math — it peers with a frontier 27B

The post led with GSM-Symbolic (100%), but that's the easy end. The card's competition-math numbers are the remarkable part:

	AIME25	AIME26	HMMT25	BRUMO25
VibeThinker-3B	91.4	94.3	89.3	93.8

For scale, that's essentially level with Qwen3.6-27B (our primary) on the same benchmarks — AIME26 94.3 vs 94.1, HMMT within ~5 pts (89.3 vs 93.8) — at 1/9 the parameters. Four independent math benchmarks agree, so it's a robust signal, not a single-benchmark fluke. This is the frontier-for-its-size claim worth leading with.

But "frontier-level coding" needs a caveat — the two coding benchmarks disagree by 42 points

LiveCodeBench v6	OJBench
80.2	38.6

LCB-v6 (80.2, matching our 83%) and LeetCode (96%) are LeetCode/contest-style — a distribution an RL-tuned model gets very good at. OJBench (38.6) is hard olympiad-tier (NOI/ICPC-class), built to be novel and contamination-resistant — and there the 3B falls off a cliff. So its coding strength is distribution-specific: near-frontier on familiar LeetCode/LCB-class problems, but the hardest novel algorithmic problems are where a 3B's ceiling shows. "Frontier one-shot coding" is true for the former, not the latter.

Net (refines, doesn't overturn, the post)

• Math — consistent across 4 benchmarks → genuinely frontier-for-its-size; trust it.
• Familiar-style coding (LeetCode / LCB / HumanEval) → strong.
• Hard novel algorithmic coding (OJBench-class) → the 3B's ceiling shows (38.6); don't assume it generalizes.
• Everything else in the post stands: no tool-calling → not an agent; Q8 keeps it, fp8 weights break it; serving numbers unchanged.

Numbers from the official card (charts under pictures/).

[tomByrer]

I'm thinking this model should be easy to fine-tune & retrain?
Weird it is bad a Bash.
Rust... needs its own sLLM ;)

llama.cpp Q8 wins on every axis

Yea I could have told you that before the tests; vLLM is more multi-user/task enterprise, llama.cpp is more single-task. ;)

Either way, thanks for the testing!

[noonghunna]

well they haven't opened the training method/data except the model and a paper. It would have been so useful had they opened it.
the best bet right now is to perhaps use it in conjunction to gemma4 12b or diffusiongemma to make the most out of it. All other models already have nearly the same level of reasoning.