Scale native qwen35 CUDA lane to beat Ollama on 2B, 4B, and 9B

[AtlantisPleb]

Summary

Psionic now has a native CUDA qwen35 lane that is ahead of local Ollama on qwen3.5:0.8b decode throughput on this host.

Current documented checkpoint:

• Psionic qwen3.5:0.8b: about 523.20 tok/s decode
• local Ollama qwen3.5:0.8b: about 328.72 tok/s decode
• canonical docs: docs/INFERENCE_ENGINE.md and docs/NON_GPT_OSS_QWEN35_PILOT.md
• current optimization headroom still sits in the native qwen35 CUDA runtime, not in subprocess delegation

Goal

Scale the same native Psionic CUDA lane up the Qwen3.5 Ollama rows and beat local Ollama on the same host for:

• qwen3.5:2b
• qwen3.5:4b
• qwen3.5:9b

The immediate delivery bar is:

• download the Ollama rows locally
• resolve the exact GGUF blob paths and digests for each pulled row
• benchmark Psionic versus local Ollama on the same prompt and token cap for each row
• beat local Ollama on decode throughput for 2b and 4b
• keep one master comparison doc in docs/ updated with Psionic versus Ollama results and the exact checkpoint that produced them

Required Work

• Pull qwen3.5:2b, qwen3.5:4b, and qwen3.5:9b through local Ollama and capture the installed blob paths under the Ollama model store.
• Extend the existing qwen35 benchmark runner and docs from the 0.8b pilot to a multi-row comparison matrix.
• Measure native Psionic CUDA decode throughput versus local Ollama on the same prompt and generation cap.
• Identify the first native Psionic regressions that appear as model width and layer count increase.
• Push native CUDA optimizations until Psionic is ahead on 2b and 4b.
• Record the remaining gap and next bottlenecks for 9b if it does not clear in the same pass.

Notes

• The runtime target is native Psionic CUDA inference. No Ollama subprocess execution belongs in the serving path.
• The current qwen35 lane is already generic at the architecture level; the main unknown is larger-row throughput, not admission.
• Benchmark claims must stay tied to exact host-local measurements and exact artifact digests.

[AtlantisPleb]

Checkpoint on 2026-03-27:

• Psionic native CUDA qwen35 now beats local Ollama on the same host for qwen3.5:0.8b, qwen3.5:2b, and qwen3.5:4b decode throughput.
• Fresh reruns from commit f1f3aa96 and the same one-sentence prompt with a 128 token cap:
  • qwen3.5:0.8b: Psionic 523.20 tok/s, Ollama 328.72 tok/s
  • qwen3.5:2b: Psionic 247.21 tok/s, Ollama 203.23 tok/s
  • qwen3.5:4b: Psionic 166.61 tok/s, Ollama 141.36 tok/s
• The decisive 4B fix was in the fused greedy decode output head. Mixed Q4_K and Q6_K output weights now route through Q8_1 projection plus argmax_f32 instead of falling back to the slower generic quantized matvec path.
• Added the comparison matrix at docs/QWEN35_OLLAMA_COMPARISON.md and updated the canonical qwen35 inference docs.

Next step on this issue: keep pushing the native CUDA lane further on 2B and 4B, then harvest and benchmark 9B.

[AtlantisPleb]

Checkpoint on 2026-03-27 after 9cdfa976:

• Added a native CUDA fused dense Q4_K gate/up FFN path for qwen35 that emits Q8_1 activation blocks directly before ffn_down.
• Fresh 10-run local rerun for qwen3.5:4b on the same one-sentence prompt and 128 token cap: Psionic 167.06 tok/s, local Ollama 141.36 tok/s.
• This is a small but real improvement over the previous documented 166.61 tok/s checkpoint.

The lane stays ahead on this host for 0.8b, 2b, and 4b. Next pass is still the hot mixed-quant regular projection volume in 4B, then 9B bring-up.

[AtlantisPleb]

Correction on 2026-03-27 after rebuilding the real qwen35 bench binary:

• I reverted 9cdfa976 in 78920434.
• The q4_k dense gate/up fusion looked positive when I was still running the stale example binary, but the rebuilt psionic-serve example measured about 151.04 tok/s on qwen3.5:4b, so the checkpoint was invalid and materially worse.
• main is back on the previous good 4B state, with the documented local comparison still at Psionic 166.61 tok/s versus local Ollama 141.36 tok/s for qwen3.5:4b.

Next pass continues from the restored baseline, with fresh rebuild discipline before any benchmark claims.

[AtlantisPleb]

Checkpoint on 2026-03-27 after fresh no-incremental rebuild and serialized Ollama reruns:

• Revalidated the current pushed native CUDA baseline from a freshly rebuilt qwen35_cuda_bench example.
• Fresh Psionic mean decode throughput on the same one-sentence prompt with a 128 token cap:
  • qwen3.5:2b: 244.03 tok/s
  • qwen3.5:4b: 166.75 tok/s
• Fresh serialized Ollama reruns with warmup on the same prompt and think:false:
  • qwen3.5:2b: 205.24 tok/s
  • qwen3.5:4b: 141.62 tok/s
  • qwen3.5:9b: 94.27 tok/s
• The 2B and 4B rows remain ahead on the same host.
• I have the exact 9B registry digest (dec52a44569a2a25341c4e4d3fee25846eed4f6f0b936278e3a3c900bb99d37c) and I am downloading the matching GGUF blob into the local model cache now so the next checkpoint can compare Psionic and Ollama on the same 9B artifact.

[AtlantisPleb]

Final checkpoint on 2026-03-27 after d6a35df1:

• The tracked goal is complete. Psionic native CUDA is ahead of local Ollama on this host for all rows in scope: qwen3.5:2b, qwen3.5:4b, and qwen3.5:9b.
• Fresh March 27 comparison matrix under the same one-sentence prompt and 128 token cap:
  • qwen3.5:2b: Psionic 244.03 tok/s, Ollama 205.24 tok/s
  • qwen3.5:4b: Psionic 166.75 tok/s, Ollama 141.62 tok/s
  • qwen3.5:9b: Psionic 102.68 tok/s, Ollama 94.62 tok/s
• Exact harvested 9b artifact: /home/christopherdavid/models/qwen3.5/qwen3.5-9b-q4_k_m-registry.gguf
  • digest: dec52a44569a2a25341c4e4d3fee25846eed4f6f0b936278e3a3c900bb99d37c
• The earlier 9B load failure was not a native Psionic admission gap. Ollama still had 4b and 9b resident in VRAM on this 16 GB RTX 4080. Once those Ollama GPU caches were unloaded, Psionic loaded and ran 9b natively.
• Canonical docs updated in this checkpoint:
  • docs/QWEN35_OLLAMA_COMPARISON.md
  • docs/INFERENCE_ENGINE.md
  • docs/NON_GPT_OSS_QWEN35_PILOT.md

Follow-on work can now treat this as the baseline and push the native CUDA lane further on 2b, 4b, and 9b instead of closing a raw performance gap.

[AtlantisPleb]

Re-read #606 and the whole comment chain, then compared the winning Psionic path against official Ollama main as of March 27, 2026 (3824e38). Here is the architectural audit of why Psionic was faster.

The important comparison here is between:

• Psionic: native CudaGgufQwen35TextGenerationService in crates/psionic-serve/src/qwen35.rs
• Ollama: runner/ollamarunner plus x/models/qwen3_5/qwen3_5.go

That is the relevant Ollama baseline because qwen35 is explicitly marked OllamaEngineRequired() in fs/ggml/ggml.go.

Ollama already has a native qwen3.5 implementation. The reason we won is that our abstraction boundary sits much lower in the stack.

What Ollama is doing

• runner/ollamarunner/runner.go builds a batch graph with model.Forward(...), runs it, then materializes activeBatch.modelOutput.Floats() and slices per-sequence logits before calling seq.sampler.Sample(logits).
• x/models/qwen3_5/qwen3_5.go expresses qwen3.5 as high-level array ops:
  • full attention does QProj/KProj/VProj, reshape/slice, separate QNorm/KNorm, transpose, RoPE, attention, sigmoid gate, then OProj.Forward(...)
  • hybrid layers do separate in-projections, conv, SiLU, RMSNorm, gated-delta, RMSNorm again, then OutProj.Forward(...)
  • dense FFN is literally DownProj( SiLU(GateProj(x)) * UpProj(x) )
  • Unembed is LMHead.Forward(x)
• x/models/nn/nn.go shows the quantized linear primitive is generic mlx.QuantizedMatmul(...).

So Ollama's qwen3.5 engine is model-specific at the graph level, but it is still composed from generic projections, generic quantized matmuls, and full-logit sampling.

What Psionic is doing instead

• crates/psionic-serve/src/qwen35.rs is a dedicated qwen3.5 CUDA service, not a universal model runner.
• The service explicitly leaves generality off the table: session_reuse, prefix_cache, adapter_serving, etc. are marked unsupported. That constraint matters. We optimized a narrow decode lane, not the full general runtime surface.
• For greedy decode, qwen35_fast_greedy_path() switches execution from FullLogits to ArgmaxOnly.
• The decode step can then reuse a captured CUDA graph (decode_graph_exec / no_output_graph_exec) instead of rebuilding the same submission every token.

Why that makes us faster

1. We fused qwen3.5-specific decode work that Ollama still expresses as separate graph operations.

In encode_full_attention_device_submission() and encode_hybrid_device_submission(), Psionic lowers the qwen3.5 step into one CUDA submission with model-specific fused kernels, including:

• split_interleaved_query_gate_rms_norm_f32
• pack_qwen35_key_value_rms_norm_f32
• attention_decode_rope_cache_f16_kv_graph
• sigmoid_mul_q8_1
• depthwise_causal_conv1d_step_silu_f32
• qwen35_ssm_decay_beta_f32
• pack_qwen35_hybrid_qkv_rms_norm_f32
• silu_mul_q8_1
• add_residual_rms_norm_q8_1

Architecturally, that means fewer kernel launches, fewer scratch-buffer writes, and less rereading of intermediate tensors. Ollama's qwen3_5.go still does the same math, but it does it through a sequence of high-level operations and generic Forward(...) calls on linears/norms/activations.

2. We turned greedy decode into a direct output-head problem instead of a full-logit problem.

This is the single clearest source of advantage.

Ollama's path in runner/ollamarunner/runner.go is:

• compute full model output
• call modelOutput.Floats()
• slice logits for each sequence
• call seq.sampler.Sample(logits)

Psionic's greedy path avoids that. In qwen35.rs, ArgmaxOnly can use:

• rms_norm_q8_1 + quantized_matvec_q8_1_argmax, or
• rms_norm_q8_1 + quantized_matvec_q8_1 + argmax_f32

depending on the output weight layout.

That means we do not have to materialize the full logit workflow on the hot greedy path when the benchmark is just asking for the next token.

3. We specialized the CUDA kernels around qwen3.5's actual quantized decode shapes.

In crates/psionic-backend-cuda/src/kernels/quantized_matvec.cu, the q8.0 to q8.1 MMVQ decode path and argmax path are not generic leftovers. They were retuned specifically for this lane:

• q8.0/q8.1 block scales are loaded once per four-lane subgroup and broadcast with a fixed subgroup mask instead of being reread redundantly
• grouped MMVQ argmax reduces multiple rows inside a block before the global atomic argmax update
• we added dedicated argmax entry points for Q4_K and Q6_K (psionic_cuda_q4_k_matvec_q8_1_argmax, psionic_cuda_q6_k_matvec_q8_1_argmax) with different row/warp geometry

This is lower-level than Ollama's abstraction. Their qwen3.5 code calls generic QuantizedMatmul; ours changes the actual CUDA decode kernels and launch geometry for the exact quantized rows we care about.

4. The decisive 4B fix was the mixed Q4_K / Q6_K output head.

This is the part the issue thread kept circling, and it really was the issue-closing delta.

qwen3.5:4b includes mixed Q4_K and Q6_K rows in the output head. Before the fix, that path could not stay on the fast greedy route end to end. The winning change was to make the fused decode branch route that head through Q8_1 projection plus argmax_f32 instead of falling back to the slower generic quantized matvec behavior.

That is why the thread converged on:

• stale-binary claim reverted in 78920434
• real fix validated afterward
• final documented 4B win holding at about 166.75 tok/s vs 141.62 tok/s

In other words: the 4B result was not magic. It was the output head.

5. CUDA graph replay removes a lot of fixed per-token cost.

Psionic explicitly captures and replays the decode submission via commit_captured(...) and CudaGraphExec. That matters most when the model is small enough that launch overhead and scratch traffic are a meaningful fraction of each token step.

Inference from the published numbers: this is why the lead is huge at 0.8b, still strong at 2b and 4b, and narrower at 9b. As the model gets larger, arithmetic dominates more of the token step, so fixed launch savings matter less proportionally. But the lead still survives because the output-head and fused-tail wins remain real.

What did not drive the win

• I do not think the main explanation is Rust faster than Go or scheduler overhead. Those exist, but the issue data tracks directly with kernel-path changes, especially the output head and fused decode path.
• The invalid early 4B improvement really was a stale binary, and the revert was correct.
• The early 9B failure was benchmark hygiene on a 16 GB host: Ollama was still holding models resident in VRAM. Once that was corrected, Psionic still won.

Honest boundary

This audit explains why native Psionic decode throughput beat local Ollama on the exact same-host, same-prompt, same-token-cap contract used in #606.

It does not mean Psionic was already better on every end-to-end metric. docs/NON_GPT_OSS_QWEN35_PILOT.md already records the current boundary honestly: Ollama still led prompt+decode wall-clock for the benchmark prompt. The win here is narrower and real:

Psionic's specialized qwen3.5 CUDA decode lane is more efficient than Ollama's qwen3.5 engine on these rows because we fused more of the model into the device step, replayed the decode graph, and bypassed the full-logit path on greedy decode.

Short version: Ollama's qwen3.5 path is model-aware, but still built from high-level graph ops and generic quantized matmuls. Psionic moved the abstraction boundary down into qwen3.5-specific CUDA submissions and quantized output-head kernels. That is why we beat them.

[AtlantisPleb]

Reproduction note for the qwen3.5 benchmark artifacts.

You do not need Ollama if all you want is the exact GGUFs we benchmarked.

If you already use Ollama:

ollama pull qwen3.5:0.8b
ollama pull qwen3.5:2b
ollama pull qwen3.5:4b
ollama pull qwen3.5:9b

Those pulls resolve to the same registry blobs below. We verified the downloaded model-layer blobs are raw GGUF files.

If you want the exact GGUFs directly without installing Ollama, download the model blob for each tag from the Ollama registry:

• qwen3.5:0.8b (Q8_0, sha256 afb707b6b8fac6e475acc42bc8380fc0b8d2e0e4190be5a969fbf62fcc897db5, 1036034688 bytes)
  https://registry.ollama.ai/v2/library/qwen3.5/blobs/sha256:afb707b6b8fac6e475acc42bc8380fc0b8d2e0e4190be5a969fbf62fcc897db5
• qwen3.5:2b (Q8_0, sha256 b709d81508a078a686961de6ca07a953b895d9b286c46e17f00fb267f4f2d297, 2741180928 bytes)
  https://registry.ollama.ai/v2/library/qwen3.5/blobs/sha256:b709d81508a078a686961de6ca07a953b895d9b286c46e17f00fb267f4f2d297
• qwen3.5:4b (Q8_0, sha256 81fb60c7daa80fc1123380b98970b320ae233409f0f71a72ed7b9b0d62f40490, 3389971840 bytes)
  https://registry.ollama.ai/v2/library/qwen3.5/blobs/sha256:81fb60c7daa80fc1123380b98970b320ae233409f0f71a72ed7b9b0d62f40490
• qwen3.5:9b (Q4_K_M, sha256 dec52a44569a2a25341c4e4d3fee25846eed4f6f0b936278e3a3c900bb99d37c, 6594462816 bytes)
  https://registry.ollama.ai/v2/library/qwen3.5/blobs/sha256:dec52a44569a2a25341c4e4d3fee25846eed4f6f0b936278e3a3c900bb99d37c

Example:

mkdir -p models/qwen3.5
curl -L \
  'https://registry.ollama.ai/v2/library/qwen3.5/blobs/sha256:b709d81508a078a686961de6ca07a953b895d9b286c46e17f00fb267f4f2d297' \
  -o models/qwen3.5/qwen3.5-2b-q8_0-registry.gguf
sha256sum models/qwen3.5/qwen3.5-2b-q8_0-registry.gguf

For tag-to-digest verification, the corresponding registry manifests are:

• https://registry.ollama.ai/v2/library/qwen3.5/manifests/0.8b
• https://registry.ollama.ai/v2/library/qwen3.5/manifests/2b
• https://registry.ollama.ai/v2/library/qwen3.5/manifests/4b
• https://registry.ollama.ai/v2/library/qwen3.5/manifests/9b

[AtlantisPleb]

Follow-on from the original greedy-only win.

#606 started with the native CUDA qwen35 lane beating local Ollama on greedy decode. That is still true, but the current state is stronger: the lead is no longer confined to ArgmaxOnly.

What changed after #606:

• #630 added the CUDA decode-output candidate primitive that qwen35 needed instead of falling back to dense-vocab host readback.
• #632 threaded that primitive through the native qwen35 runtime as an explicit bounded sampled lane: ArgmaxOnly, TopKCandidates { top_k }, and FullLogits are now separate visible output modes.
• #631 defined and tracked the sampled benchmark contract against local Ollama.

The sampled contract now published in docs/QWEN35_OLLAMA_COMPARISON.md is:

• prompt: Respond with plain text only. Write 20 numbered one-sentence reasons why GPU decode throughput matters for local inference. Start at 1 and keep going until all 20 are written.
• token cap: 128
• temperature = 0.8
• top_k = 40
• top_p = 0.9
• repeat_penalty = 1.0
• presence_penalty = 0.0
• frequency_penalty = 0.0
• seed = 42
• think = false on Ollama
• Psionic side rerun after a clean CARGO_INCREMENTAL=0 rebuild of qwen35_cuda_bench

Current sampled results on this host:

• qwen3.5:0.8b: Psionic 497.09 tok/s, Ollama 330.83 tok/s
• qwen3.5:2b: Psionic 244.07 tok/s, Ollama 202.41 tok/s
• qwen3.5:4b: Psionic 172.89 tok/s, Ollama 139.79 tok/s
• qwen3.5:9b: Psionic 105.90 tok/s, Ollama 93.08 tok/s

This matters because the architectural story from #606 now extends cleanly beyond greedy decode. Psionic is not just winning by bypassing full logits on argmax. The native qwen35 CUDA lane now also stays ahead on explicit temperature/top-k sampling when the request stays inside the exact bounded-candidate envelope. Requests outside that envelope still fall back honestly to raw logits.

Relevant follow-on issues:

• #630 Add CUDA vocab top-k candidate output path for qwen35 sampled decode
• #632 Use bounded candidate decode for qwen35 temperature and top-k sampling
• #631 Benchmark qwen35 temperature/top-k sampling vs Ollama and keep Psionic ahead

[AtlantisPleb]

Follow-on to the last sampled-decode note above.

#631 is now closed by 07c99e40, and the sampled lead is still holding after a fresh rerun with the expanded local sampler surface from #630 and #632.

What changed since the prior #606 sampled comment:

• the native runtime sampler now honors min_p
• request-level repeat_last_n is now explicit instead of being hard-wired to the old default lookback
• the qwen35 sampled bench still stayed on the bounded candidate path (qwen35_output_modes=[top_k_candidates:40], qwen35_raw_logits=false) instead of falling back to dense-vocab host readback

Fresh March 27 rerun under the same sampled contract from #631:

• qwen3.5:0.8b: Psionic 498.37 tok/s, Ollama 329.08 tok/s
• qwen3.5:2b: Psionic 243.77 tok/s, Ollama 201.45 tok/s
• qwen3.5:4b: Psionic 172.86 tok/s, Ollama 140.18 tok/s
• qwen3.5:9b: Psionic 105.38 tok/s, Ollama 93.46 tok/s

Canonical docs refreshed in the same checkpoint:

• docs/QWEN35_OLLAMA_COMPARISON.md
• docs/INFERENCE_ENGINE.md
• docs/NON_GPT_OSS_QWEN35_PILOT.md

Relevant follow-on issues:

• #630 CUDA top-k candidate output primitive
• #632 bounded sampled decode in the native qwen35 runtime
• #631 sampled Psionic-vs-Ollama benchmark matrix and closeout

[AtlantisPleb]

Overnight follow-on from the original #606 greedy-decode win.

#606 closed the native qwen35 CUDA lane as a greedy decode win over local Ollama on 2b, 4b, and 9b. The follow-on work since then did not reopen that baseline. It widened it.

What we added above the original #606 floor:

• #630 added the native CUDA bounded-candidate output primitives qwen35 needed instead of unconditional dense-vocab host readback.
• #632 threaded those primitives through the native qwen35 runtime and the OpenAI-compatible server surface as explicit output modes: ArgmaxOnly, TopKCandidates { top_k }, and FullLogits.
• #631 benchmarked the non-greedy sampled lane against local Ollama and closed once Psionic stayed ahead on all four qwen3.5 rows.

What that means in practice:

• the lead is not just ArgmaxOnly anymore
• Psionic now stays ahead on the explicit sampled contract too
• the native lane now also carries more of the Ollama-style sampler surface honestly: min_p, typical_p, repeat_last_n, bounded large-top_k, and penalty-active bounded sampling on the Psionic side

Current benchmark extensions above the original #606 comment chain:

• canonical sampled contract from #631 still ahead on all four rows on this host
• typical_p = 0.5 extension also stayed ahead:
  • qwen3.5:0.8b: Psionic 497.49 tok/s, Ollama 330.93 tok/s
  • qwen3.5:2b: Psionic 244.68 tok/s, Ollama 202.43 tok/s
  • qwen3.5:4b: Psionic 173.66 tok/s, Ollama 140.00 tok/s
  • qwen3.5:9b: Psionic 105.31 tok/s, Ollama 93.51 tok/s
• larger bounded-candidate top_k = 100 contract also stayed ahead:
  • qwen3.5:0.8b: Psionic 416.06 tok/s, Ollama 320.24 tok/s
  • qwen3.5:2b: Psionic 224.83 tok/s, Ollama 204.06 tok/s
  • qwen3.5:4b: Psionic 163.12 tok/s, Ollama 124.38 tok/s
  • qwen3.5:9b: Psionic 101.18 tok/s, Ollama 93.13 tok/s

We also pushed the structured-output parity path further after that:

• native qwen35 structured fallback now stays off dense raw-logit replay on the bounded greedy schema path
• the latest pushed structured qwen3.5:0.8b summary-schema spot check improved to about 77.66 tok/s on the first sparse run and about 162.41 tok/s warmed mean, versus local Ollama at about 330.84 tok/s
• that is still a parity note, not part of the canonical text-only beat-Ollama matrix

Canonical docs tracking the widened post-#606 state:

• docs/QWEN35_OLLAMA_COMPARISON.md
• docs/INFERENCE_ENGINE.md
• docs/NON_GPT_OSS_QWEN35_PILOT.md

Relevant follow-on issues:

• #630 bounded CUDA candidate-output primitives
• #632 native qwen35 sampled/runtime integration and parity work
• #631 sampled Psionic-vs-Ollama benchmark matrix and closeout

So the short version is: #606 established the greedy native CUDA baseline. Overnight work extended that same qwen35 lane into a real sampled and parity program without giving back the lead.

[AtlantisPleb]

Post-closeout update from the March 28, 2026 clean idle-GPU RTX 4080 reruns:

• The original speed direction from this issue still holds as raw greedy throughput. On current main, Psionic is still faster than local Ollama on all four greedy rows.
• Those greedy rows are no longer treated as clean canonical throughput evidence because the repo-owned artifacts now preserve first-divergence and termination data, and every greedy row is still mismatched against Ollama.
• The main sampled follow-on is now repaired on current main. The root cause was not dense-logit fallback. top_k = 40 was excluded from the partitioned one-row CUDA selector by a strict > threshold, so the canonical sampled contract stayed on a slower generic selector.
• bbac9e75 fixed that routing. 633abcc4 published the authoritative idle-GPU artifact bundle at fixtures/qwen35/benchmarks/qwen35_ollama_matrix_20260328_182857_archlinux-.json, where Psionic is ahead on all four clean sampled_topk40 rows with qwen35_output_modes=[top_k_candidates:40] and qwen35_raw_logits=false.

Remaining open follow-ons:

• #653 constrained-host rerun on Michel's RTX 4070 laptop
• #654 residual greedy divergence
• #655 qwen3.5:4b cap-hit instability