🧪 Dual Carnice-V2-27B (Q8) on 2× 3090 — beellama MTP, q8_0 KV over kvarn6, base-parity quality #417
Replies: 2 comments 8 replies
-
|
This is a very useful benchmark for dual-3090 local inference. The q8_0 KV vs KVarN result is especially interesting. It is a good reminder that the best memory strategy depends heavily on the actual hardware envelope. On a tight single-card setup, compression may be worth the compute overhead, but on dual 3090s with enough VRAM headroom, higher-fidelity q8_0 KV can be both faster and cleaner. For agentic workflows, I think the most useful next layer would be measuring not only raw TPS, but workflow-level behavior:
The OpenAI-compatible serving path is also important. It makes this kind of local setup much easier to test with existing agent tools, but the real question is whether the model stays stable across planning, tool use, code edits, and long-context replay. Great work documenting the tradeoff between KV strategy, MTP, quality, and hardware fit. |
Beta Was this translation helpful? Give feedback.
-
|
Appreciate it, Richard — and your "where each is stronger" framing is the right one. Cleanest path is you running the same packs against your endpoint and sharing the JSON; we line it up against our local numbers for the identical model + packs and publish the methodology alongside, so it's reproducible and even-handed either way. Install: pip install git+https://github.com/noonghunna/benchlocal-cli.git
benchlocal-cli list # all packsFull quality suite — the 8-pack (/150) + aider-polyglot-30 — against your endpoint: for p in toolcall-15 instructfollow-15 structoutput-15 dataextract-15 \
reasonmath-15 bugfind-15 hermesagent-20 cli-40 aider-polyglot-30; do
benchlocal-cli run \
--endpoint https://<your-hub>/v1 \
--model <model-id-on-your-hub> \
--api-key <your-key> \
--pack "$p" \
--request-delay 0.5 \
--max-transient-retries 5 \
--save-json "cloud-$p.json" --progress
doneOn rate limits — yes, two controls, built for exactly this cloud case:
Two notes:
Each JSON logs per-pack token counts + per-case latencies, so cost-per-completed-run falls straight out (your price × tokens) — exactly the metric you listed. Drop the |
Beta Was this translation helpful? Give feedback.
-
|
Quick follow-up — I folded that whole cloud-run recipe into the benchlocal-cli README so it's the canonical reference rather than just this thread: Running against a cloud / managed endpoint (the cloud knobs, the rate-limit pacing + retry pair, the spend guard, provider pinning, and which packs are the clean apples-to-apples). Whenever you've got |
Beta Was this translation helpful? Give feedback.
-
|
Appreciate the detailed recipe — this is exactly the kind of reproducible comparison I had in mind. I agree that the cleanest path is to run the same packs against a managed OpenAI-compatible endpoint, keep the methodology identical, and compare the JSON outputs rather than making a vague local-vs-cloud claim. I will start with the more deterministic packs first, then decide whether to add the heavier agentic packs after checking token usage and latency behavior:
Then, if the endpoint behaves well, I can test the heavier workflow packs separately:
The rate-limit and spend-guard notes are very useful. I will use request pacing, transient retries, and a token ceiling so the comparison stays controlled instead of turning into an uncontrolled cloud spend test. For the model, I agree that matching qwen3.6-27b or the closest equivalent is the most useful path, since that makes the local vs managed comparison much more even-handed. Once I have the cloud-*.json files, I’ll share them here so the comparison can focus on the real metrics:
Thanks for also folding the cloud-run recipe into the README. That makes this much easier to reproduce cleanly. |
Beta Was this translation helpful? Give feedback.
-
I finished the first managed OpenAI-compatible endpoint run and attached the generated cloud-*.json files as a zip. Model used: Doubao-Seed-2.0-Code I kept this first run focused on the lighter deterministic packs before moving to the heavier workflow packs, so the comparison stays controlled and cost-aware. Summary:
A few early observations:
I’ll keep the heavier workflow packs separate for now, since hermesagent-20, cli-40, and aider-polyglot-30 may have very different token usage, latency, and runtime behavior. The goal here is to make the local vs managed endpoint comparison based on completed runs, latency, failure modes, token usage, retries, and cost per completed benchmark run rather than a vague local-vs-cloud claim. |
Beta Was this translation helpful? Give feedback.
-
|
Aligned on the framing — "where each stack is stronger," confounds named — and good confirmation that the latency tail tracks the thinking-on packs. The two-arm round is the one to chase next: if Doubao exposes a native reasoning toggle, run On the test key — I'll actually take you up on it, but for a slightly different purpose than the comparison, which is where I think it earns its keep: hardening benchlocal-cli's verifiers against a different model family. Our pack verifiers were developed mostly against local qwen/gemma-style output; a frontier model from another family is exactly the adversarial input that surfaces over-fit grading. Your run already has a candidate — So yes please to a small benchmark-only test key — we'll treat it strictly as a dev credential (env-only, never committed, packs only; no private/customer data, as you scoped). Easiest if you send it privately rather than in-thread. The public local-vs-cloud comparison still stays on the you-run → share-JSON → we-publish-the-method model, so that part remains vendor-neutral and reproducible by anyone. Whenever you've got the two-arm |
Beta Was this translation helpful? Give feedback.
-
|
This is great, Richard — and you ran it exactly right: deterministic-first, paced, retries on, cost-aware. Two caveats to set up an honest comparison, then the numbers. Caveat 1 — model and endpoint differ. You ran Doubao-Seed-2.0-Code (a managed frontier code model); our local baseline is qwen3.6-27b. So gaps fold "different model" + "different deployment" together — useful for "where is each stronger," but not the same-model deployment isolation (for that, a qwen3.6-27b your hub serves would be apples-to-apples). Caveat 2 — reasoning state, which we should have flagged in the recipe (our miss). benchlocal-cli sets thinking per-pack via each pack's Quality — the deterministic five (no-Docker packs):
(Our base qwen3.6-27b is benched as det/sandbox totals — det 64/75 on Three things stand out:
For the next round — run both reasoning arms. Since thinking's effect is pack- and model-specific (in our own tests it's often a wash, sometimes a slight regression on strict-format packs), the clean approach is two arms and report the delta: # all-OFF arm
benchlocal-cli run --endpoint … --model … --api-key … --pack reasonmath-15 --no-thinking --save-json off-reasonmath.json
# all-ON arm
benchlocal-cli run --endpoint … --model … --api-key … --pack reasonmath-15 --enable-thinking --thinking-max-tokens 16384 --save-json on-reasonmath.jsonCaveat 2 still applies — those flags steer a vLLM-style server cleanly, but against Doubao you'd want to set its native reasoning toggle to match each arm (otherwise the flag is a no-op and both arms are identical). Where each is stronger, on this run: Doubao — excellent tool-calling, zero hardware/maintenance, scales on demand. Local qwen3.6-27b — slight dataextract edge, reasoning state fully under your control, full privacy/control, cost is just the power cap. Different strengths, no dominator — the framing you called from the start. Then the heavy packs: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the detailed read — this is very helpful. I agree with both caveats. The Doubao run is useful as a managed-endpoint comparison, but it is not a clean same-model deployment isolation against local qwen3.6-27b. So I would treat this first pass as a “where each endpoint/model stack is stronger” comparison, not as a strict local-vs-cloud model-equivalence claim. The reasoning-state point also makes sense. I saw the same latency pattern: reasonmath-15 and bugfind-15 were the main p95 outliers, while the non-reasoning/default-off packs stayed much tighter. That does suggest the tail is likely reasoning-token generation on harder cases rather than generic endpoint variance. For the next round, I’ll first check whether this managed endpoint exposes a native reasoning / thinking control for Doubao-Seed-2.0-Code. If it does, I can run the cleaner two-arm comparison you suggested. Also, if you want to keep a managed OpenAI-compatible endpoint as a standing cloud comparison target for benchlocal, I can provide a small free test credit / test key for this purpose. The safest scope would be:
That way you can run the same packs from your side too, instead of only relying on the JSON files I upload. I think that would make the local vs managed comparison more useful: not “local or cloud is globally better,” but where each stack is stronger, weaker, and more predictable under the same benchmark workflow. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
We just shipped
beellama/carnice-v2-dual-q8-mtp🧪 — Carnice-V2-27B (@kai-os's Hermes-style agentic SFT of Qwen3.6-27B), Q8_0 weights + embedded MTP head (MTP GGUF by @stuchapin), on beellama.cpp v0.3.2 (@Anbeeld), layer-split across two 3090s. The dual / quality-max follow-through to the single-card Carnice — requested in #403. All credit for the model goes to @kai-os; @stuchapin's MTP head is what unlocks the self-speculative path — see Credits.The headline: q8_0 KV beats the kvarn6 we were asked for — +17% prefill, higher fidelity, and it still fits the full 262K on dual.
🎴 Results Card — 2× RTX 3090 PCIe (no NVLink), beellama v0.3.2, think-off
① Serving
vllm/dual-max†TTFT ~79 ms · prefill ~1197 t/s · n=5 measured, CV <1.3% ·
DRAFT_N_MAX=2= +13% decode (lossless, opt-in). Sampling temp 0.6 / top-k 20 / top-p 0.95.† Reference row — stock Qwen3.6-27B at its max-fidelity dual tier (
vllm/qwen-27b-dual-max: official FP8 weights + int8-PTH KV, vLLM TP=2). Reported as a single ~56 TPS decode figure (narr/code not separately split). This is a different engine + quant (vLLM FP8 vs beellama Q8) — not apples-to-apples on raw TPS; it's here so you can place Carnice against base. The comparable axis is capability — see the 8-pack below.② Quality — core 8-pack (/150)
vs base: stock Qwen3.6-27B at the same max-fidelity dual tier (
vllm/qwen-27b-dual-max) scores 110/150 on the same--fullharness (det 65 · sandbox 45). So the Carnice fine-tune lands within ~5–7 pts (sampling noise band) of base capability — i.e. no measurable capability cost for the agentic/Hermes character. Think-off ≈ think-on (wash). In-band with our other 27B-class composes (qwopus-coder 103). Plus: verify-full 8/8 · verify-stress 8/8 (NIAH ladder filled to 240K = 91% of n_ctx, 5.2 GB ceiling margin) · soak-continuous PASS (20×5, 0 MiB growth, 0/100 silent-empty, p50 42.2 TPS, 100% retention).③ Takeaways
carnice-v2-single). On dual's 48 GB you don't need the compression, so q8_0 is faster, higher-fidelity (q8 > kvarn6 q6-class), and reference-aligned. (-b/-ub/--no-mmapwere A/B'd flat — KV type was the only prefill lever.)vllm/dual-max's 110/150 on the 8-pack — you're trading none of the underlying Qwen capability for the Hermes/agentic tuning.-ts 0.55,0.45to offset the MTP context pinning to the main card). On a 2-GPU box it's mutually exclusive with single-card work.Why dual Q8 Carnice?
Carnice-V2 is a Hermes-style agentic/reasoning fine-tune — the use case is tool-use + reasoning with thinking on. The single-card path (
carnice-v2-single, Q5 + kvarn) trades quality for fitting 24 GB; on two cards you can run Q8 weights + high-fidelity q8_0 KV over the full 262K with no compression compromise. This is the quality-max Carnice.Getting it
The MTP GGUF is public: stuchapin/Carnice-V2-27B-MTP-GGUF (
Carnice-V2-27B-Q8_0-mtp.gguf).setup.sh/launch.shauto-fetch it, orhf download stuchapin/Carnice-V2-27B-MTP-GGUF. Runs on beellama.cpp v0.3.2 (pinned digest in the compose).Run it
bash scripts/switch.sh --owui beellama/carnice-v2-dual-q8-mtp # launches on :8070 and wires Open WebUIServes an OpenAI-compatible API on
:8070. Reasoning is on by default (it's a reasoning SFT); setREASONING=offfor fast chat,DRAFT_N_MAX=2for the +13% decode.What'd help
Cross-rig numbers (other 3090 pairs / 4090s — fit, TPS, MTP accept), and anyone who finds or trains a Carnice-matched DFlash drafter (the one thing that'd beat MTP here). The portable finding: on a dual card with headroom, standard q8_0 KV beats KVarN software-compression on prefill — KVarN's win is single-card-only.
Credits
--spec-type draft-mtpdecode.--spec-type draft-mtp+ KVarN.Shipped via PR #416. 🧪 Experimental only because beellama v0.3.2 is a rolling pre-release — fully validated otherwise. Originally requested in #403.
Beta Was this translation helpful? Give feedback.
All reactions