Repository navigation
Edge0-145B Q3_K on an RTX 3090 (24 GB, PCIe Gen 2): community bench at 24 / 16 GiB, plus why 12 GiB doesn't fit #220
Replies: 1 comment
GPQA Diamond sample on
|
| Run | Correct | 95 % interval |
|---|---|---|
max_tokens 24 000 (engine thinking budget 18 000) |
43 / 50 = 86 % | ±9.6 pp |
Same, but the 15 questions that hit the budget rerun with max_tokens 100 000 |
47 / 50 = 94 % | ±6.6 pp |
| Model page | 90.4 % |
By domain (final): physics 25/25, chemistry 20/22, biology 2/3.
The thinking budget matters
The engine sets the thinking budget to 75 % of max_tokens (18 000 at 24 000). 15 of the 50 questions hit it, and 6 of those 15 were wrong. Every question under the budget but one was right (34/35). With room to think, 4 of the 6 wrong answers became right and none went the other way. The rerun questions used 39 900 tokens on average (12.6 min each on this card), and two still hit the new 75 000 budget and stayed wrong.
So for hard reasoning, a max_tokens that looks generous still cuts about a third of GPQA-level questions short.
One capped answer was right but unscored: the forced wrap-up printed just A instead of Answer: A. I kept the strict parse, so it counts as wrong in the 86 % row.
Memorization probe
GPQA is public, so I checked for verbatim recall on the same 50 questions (thinking off, temperature 0):
- Continue the question text from its first half: the verbatim run before the first mismatch was 2–4 words for 49 of 50. One reached 12, but those 12 words are a phrase template that repeats three times in that question's own prefix, and the numbers the model filled in were wrong. Mean 5-gram overlap with the true continuation: 1.5 %.
- Reproduce the four answer options without seeing them. The distractors can't be derived, so recalling them verbatim would indicate exposure. Two questions were flagged. In one the options are literally listed in the question text. In the other the model produced the right shape with the wrong numbers.
So there's no sign of verbatim recall. That doesn't rule out softer familiarity. But the thinking length tracks difficulty, and answers flipped from wrong to right with more budget, which fits reasoning rather than recall.
Raw per-question results (question numbers, answers, token counts, no question text) are available if useful.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi! There's no RTX 3090 row on the model page yet, so here is one from a home Linux box. Below are the community bench results (
tools/strata_bench.py) at full VRAM and with the VRAM capped at 16 GiB, what happens at 12 GiB, and a small bug in the bench's agent test.Hardware and software
PCIe probe: 6.5 GB/s host->device -> pcie_frac 0.18 (default 0.55)7b60076(2026-11-14), edge0-strata 0.1.55 built as in EDGE0.md with-DCMAKE_CUDA_ARCHITECTURES=86, edge0-infercpupresetEdge0-145B-q3kas downloaded, started withedge0-infer serve; defaults except--vram-reserve-mibNothing else was running on the GPU. The only steady background load was a remote-desktop daemon at about 18 % of one core.
Results (medians of 3 runs, engine timings, tok/s)
Time to first token at 24 GiB: 6.6 s (4K), 14.9 s (16K), 28.6 s (32K). The 16 GiB numbers repeated within 1 % across two separate runs.
¹ One of three agent sessions. The other two hit the bench issue described below, not an engine fault.
The 16 GiB row uses your method (
--vram-reserve-mib 8192on a 24 GiB card). The engine then held 16 077–16 307 MiB, so the reserve does what the README says. System RAM was not capped (62 GiB), unlike your "16 GiB VRAM + 20 GiB RAM" row.Observations
pcie_fracto 0.18 accordingly. A long prompt touches nearly all experts, and the ones not cached in VRAM cross PCIe, while a decode step mostly hits the VRAM cache. So I think the Gen 2 link costs mostly prefill. That's an inference, not a controlled test. If I get the slot to Gen 3 I'll rerun and post the difference.12 GiB: does not fit with the shipped template
With
--vram-reserve-mib 12288the engine exits during load (edge0-infer serveonly shows "the engine exited before it was ready"). Startingserve/server.pydirectly with alogentry shows why:With
--max-context 32768 --kv q4_0it does start under the reserve method (292 slots), but it then uses 13 749 MiB. Allocations made after the cache is sized come out of the reserve, so the reserve method overstates what fits at this size. At 16 GiB it stayed within budget.To emulate a real 12 GB card I held VRAM with a separate process so that only 11 888 MiB were free, and used the default reserve. Then even
--max-context 32768 --kv q4_0is 447 MiB short. The MTP draft can't be dropped to make room, because a native pack needs--spec T (T >= 2). Possibly--max-context 8192plus a 300 MiB reserve plussetup --draft-vocab enwould scrape by, but I haven't tried it. So on a 12 GB card it doesn't run as shipped. 16 GiB looks like a real floor, as the requirements say.Bench issue: tool-call-only answers count as failures
The agent test sends four tools. When the model answers a turn with only a tool call,
tools/strata_bench.pyreportsFAILED: no token came back, although the engine generated tokens (26–49 in the failed turns).chat()only countsdelta.contentanddelta.reasoning_contentas a token (around line 160), and with thinking off a tool-call-only answer streams nothing butdelta.tool_calls:At 24 GiB the model happened to answer in text. At 16 GiB it sometimes chose a tool call (answers vary a little with cache size). Update: reported as #222 (confirmed as a bench bug) and fixed in #223. The same line is in upstream Strata's
tools/strata_bench.py, and the symptom showed up in an RX 9070 XT report there (Niko1221/Strata#2876).Coming in this thread
max_tokens) affects the score.edge0-infer serve.Full reports (
README.md+results.jsonfor each run):24GiB
results.json
README.md
16GiB
results.json
README.md
The engine binary had a local one-line
--helptext change while I looked at #209/#211. No code change.Edited 2026-11-16: corrected the token range of the failed turns (26–49, not 31–42), linked the issue and PR for the bench bug, and attached the reports.
All reactions