Repository navigation
cnm-varexp-v0.1
MoREL runtime v0.1 - prerelease
Source: oisee/llama.cpp, cnm/release-v0.1, pinned commit 53f311f39957bf92608753ec2af214fe42ccb08e.
The branch starts at cnm/varexp / 7ed1a0a57e5621073146bca8d5c166e52cfe462b. Bias implementation c31b35f0d and its follow-up 7df19d87 cherry-picked cleanly as 1e4033199 and 53f311f39. The latter alone is not the bias implementation.
MoREL means Mixture of Ragged Experts per Layer. These GGUF models physically retain different expert counts in different layers. They require this fork's variable-expert tensor shapes, source-width router reconstruction, removed-expert masking before softmax, and original-ID to compact-slot mapping. Mainline llama.cpp and Ollama compatibility is not established. The release does not establish new model quality or benchmark results.
Models:
Download every shard to the same local directory and pass shard 1 to -m. Keep model files on the machine's local disk when running; do not mmap them across the CIFS working share. Use the model card's chat template and optional matching MTP head if serving that configuration. The short checks in MATRIX.md do not validate long-context serving, MTP, task quality, or new GPU throughput. GPU generation on i7/i5 was explicitly excluded from part 1.
Extract a machine-specific tarball and run its binaries from the extracted directory. All project shared libraries are next to the executables, with $ORIGIN runtime paths. Each archive includes llama-server, llama-cli, llama-perplexity, header-check, LICENSE, BUILD.txt, and HEADER-CHECK.txt. Check SHA256SUMS before extraction. Ubuntu 24.04 userspace and the backend's system GPU drivers/runtime are required. These builds disable HTTPS support and the embedded server Web UI. They are for our named machines, not universal Linux binaries. The i3 CPU lacks AVX2; its Vulkan package uses the local CPU configuration. The i10 CPU package explicitly enables AVX2/FMA/F16C with native tuning off. CUDA builds target i7 sm_89/86/75 and i5 sm_75 using CUDA 12.0 and GCC 12.
For the tested 32 GB i3/RX 580 smoke configuration, use --device Vulkan0 -ngl 2 --no-host --no-repack --no-op-offload --load-mode mmap --lazy-mode on -fa off. This offloads one repeating layer plus the output stage. A non-mmap profile allocated a roughly 22 GB Vulkan_Host buffer and failed command submission; the successful flags avoid that model-sized pinned-host allocation. This does not validate full GPU offload or larger contexts. See MATRIX.md and the report for the exact 512-token-context test.
Runtime controls (normally leave both unset for a physically compact MoREL model):
CNM_EXPERT_MASK=/absolute/keep.json: hard expert selection. Object keys are layer numbers, values arrays of kept expert IDs, for example{"0":[1,4,7]}. A real plan must cover every supported MoE layer, including NextN, and retain at least top-k experts. IDs are in the loaded model's expert index space; do not apply original-width plans blindly to compact weights. Grouped routing has additional safety checks.CNM_EXPERT_BIAS=/absolute/plan.json: experimental nonpositive F32 offsets before softmax/top-k, intended for uncut Qwen4Exp with ungrouped softmax routing and no existing expert-selection bias. Scalar example:{"bias":-2,"layers":{"0":[1,4,7]}}; these IDs identify cold experts. Per-expert example:{"layers":{"0":{"1":-1,"4":"-inf"}}}. Omitted entries have zero offset. Positive offsets, malformed IDs and insufficient finite experts are rejected. Cannot combine withCNM_EXPERT_MASK.CNM_ROUTING_TRACE=/absolute/trace.bin: optionalllama-perplexityrouting trace. Its evaluation callback can inhibit fusion; apply it consistently in comparisons.
The source bias documentation specifies validation and numerical limits. A hard mask and matching negative-infinity bias share a graph, but finite penalties and physically compact cuts are not asserted to be numerically identical. These release smoke checks do not repeat the bias equivalence experiments. The optional CUDA top-k determinism override remains at its default in these builds.
Runtime source is MIT (included LICENSE); model weights have their own Qwen community license and quantizer attribution in their HF repositories.
This prerelease publishes the four machine-specific packages verified in part 1. Generic GitHub Actions builds are listed separately when available; they are untested by us.
Release provenance
Tag cnm-varexp-v0.1 points to 45d2652d955542b1666c6fdcb5ed45e87d811637. The runtime source is unchanged from 53f311f39957bf92608753ec2af214fe42ccb08e; subsequent release commits only adapt GitHub workflows for this fork.
Machine package SHA-256
8c40381780f37427c52d7a5b30efe07bf0e0448ad0880f1a7d025ebf2ad0876e llama-cnm-varexp-v0.1-i10-cpu-linux-x86_64.tar.gz
5420ec8769075c1affe3b011b5a0d7dc92fa3cd688accfb7b8ee4b6e5b07f345 llama-cnm-varexp-v0.1-i3-vulkan-linux-x86_64.tar.gz
7c7390f8193197fb38b8e3aa9ddd5507afa6caa21aaed19672e4c9857a8c502c llama-cnm-varexp-v0.1-i5-cuda-linux-x86_64.tar.gz
567d67ff6ca26c3046c39247b8efa704f897d981f2fa2671a6edd7ebe8a7e16b llama-cnm-varexp-v0.1-i7-cuda-linux-x86_64.tar.gz
Test matrix from part 1
v0.1 build and verification matrix
Source commit: 53f311f39957bf92608753ec2af214fe42ccb08e on cnm/release-v0.1. All builds use Release mode and shared project libraries. Detailed flags, compilers, date and CPU information are in each archive's BUILD.txt.
| Host / actual hostname | Actual CPU and backend hardware | Build | Tested on that machine | Not tested |
|---|---|---|---|---|
| i7 / u7 | Intel Core i7-10700K; 2 x RTX 4060 Ti 16 GB | CUDA 12.0, GCC 12, fat sm_89/86/75; PASS | All three programs --version; PB3-208 metadata/tensor-shape load with zero weight buffers; extracted-archive relocation/ldd; invalid GGUF rejected; actual library contains all three architectures |
GPU generation intentionally excluded; sm_86/sm_75 execution; long context, MTP, serving |
| i5 / u5 | AMD Ryzen 7 5800X; RTX 2070 8 GB | CUDA 12.0, GCC 12, sm_75; PASS | All three programs --version; PB3-208 metadata/tensor-shape load with zero weight buffers; extracted-archive relocation/ldd; invalid GGUF rejected; actual library contains sm_75 code |
GPU generation intentionally excluded; long context, MTP, serving |
| i3 / u3 | Intel Core i5-3570K; Radeon RX 580 8 GB, RADV Mesa 25.2.8 | Vulkan; local CPU tuning (AVX, no AVX2); GCC 13; PASS | vulkaninfo; shipped CLI lists Radeon Vulkan0; all three programs --version; PB3-208 header load; extracted-archive relocation/ldd; invalid GGUF rejected; 16-token Vulkan completion PASS: exit 0, exact 16-token count, one repeating layer plus output stage offloaded |
Full GPU offload, other Vulkan devices, long context, MTP, serving |
| i10 / truenas | AMD Ryzen 5 5600G, 6 cores / 12 threads | CPU AVX2/FMA/F16C, native OFF, GCC 13; Ubuntu 24.04 Docker; PASS | All three programs --version; PB3-208 header load; extracted-archive relocation/ldd; invalid GGUF rejected; successful -n 16 CPU completion from local ZFS model files, exit 0 |
GPU backends, long context, MTP, serving |
Header checks use the packaged header-check helper linked to that package's libllama, with no_alloc=true, load_mode=NONE, lazy_mode=OFF, and zero GPU layers. They load both split headers, validate tensor shapes and report HEADER_OK layers=48 embedding=2560 with zero-sized weight buffers. This is stronger than a vocabulary-only load, but does not execute the model or check every weight byte. GPU visibility is disabled during CUDA package checks.
The completion model is PB3-208, first shard Qwen3.8-Flash-Next-GSQ-RCO-MoREL-PB3-208-00001-of-00002.gguf, with both shards present. The CPU test runs in the Ubuntu container with a read-only bind of the host's local ZFS dataset. The Vulkan test uses a separate copy on i3's ext4 filesystem and verifies both hashes against the public card. Neither completion reads weights through CIFS. The final Vulkan command uses -ngl 2 (one repeating layer plus the output stage), --no-host --no-repack --no-op-offload --load-mode mmap --lazy-mode on to avoid a model-sized Vulkan pinned-host buffer. Both request 16 output tokens with --ignore-eos, fixed seed 4242, temperature 0, context 512, --no-warmup, and a short arithmetic prompt. Output can consume the budget in reasoning; this is an execution smoke test, not an answer-quality test. Commands and raw output are in the run record.
Every build uses nice 19; i7/i5 compilation uses two parallel jobs. i5 node-check ran before build work. No server or LOCK was stopped or changed by this task. i10 installs its toolchain only inside a new Ubuntu container image; i5 uses a private copied CMake 3.28 runtime under its isolated build directory, with existing GCC/CUDA packages.
Each package contains llama-server, llama-cli, llama-perplexity, project shared libraries, LICENSE, BUILD.txt, HEADER-CHECK.txt, and the header-check helper. System C/C++ libraries and GPU drivers/CUDA system libraries are external dependencies, recorded in ldd logs. HTTPS and the embedded server UI are disabled in these local builds. The tarballs target these machines and Ubuntu 24.04, not arbitrary Linux systems. Version build counts can differ between the full local checkout and shallow remote clones; the source SHA is identical.
Windows CUDA, macOS arm64/Metal and generic Linux CUDA/Vulkan Actions builds are part 2, not built or tested here. PB3-304 and K192p generation are not tested by this release run. Existing model-card benchmark claims are not revalidated by packaging.
Known i3 startup failures are retained in the report: output-only -ngl 1 was aborted, non-mmap Vulkan_Host loading exhausted command-submission memory, and a non-mmap/no-host load reached its short cap. Only the final mmap/no-host profile is covered by the successful completion.
Generic GitHub Actions builds: unavailable
The fork reports Actions enabled with all actions allowed, but its workflows and runs APIs both return total_count=0. Dispatching release.yml on cnm/release-v0.1 returns HTTP 404, including after the branch-only workflow adaptation and a matching branch push. No build job ran and no Actions artifacts exist. Windows x64 CPU/CUDA, macOS arm64 and generic Linux x64 CPU/CUDA/Vulkan packages are therefore omitted from this prerelease. There are no flaky jobs to rerun. The four attached packages are the machine builds described above, not GitHub Actions builds.
The adapted workflow removes the upstream deploy-key tag push, uses the fixed prerelease tag for artifact names, and retains only the requested platform matrices (CUDA 12.4 on Windows and CUDA 12.8 on Linux). Windows CUDA backend output requires merging the corresponding Windows CPU executables before publication; Linux and Windows CUDA runtime companion packages would also need publishing. No such unbuilt asset is advertised here.