Colibri Benchmarking and Optimization Report

[mkaouri]

1. Overview & Goal

This report chronicles a successful end-to-end deployment, automated calibration, and physical disk-bound optimization of the Qwen3.6-35B-A3B Mixture-of-Experts architecture using Colibri. The objective was to run this massive model locally on a standard, pure-CPU Ubuntu server without discrete GPU hardware, and maximize the decoding speed solely through software tuning and environment adjustments.

---

2. Environment & System Topology

Before starting the optimization loops, we audited the physical layer of the host system:

• Operating System: Ubuntu Linux (executing inside a (base) active environment).
• CPU Architecture: 12 Logical Cores (Cores 0-11).
• Memory Topology (NUMA): NUMA node(s): 1 (Unified node0 mapping CPU cores 0-11). This confirms that cross-socket memory access latency does not impact this machine; hence, interleaved allocations via COLI_NUMA=1 have a neutral effect.
• Target Model: Qwen3.6-35B-A3B (23 GB footprint on disk; 256 total structural experts, routing 8 active experts per layer via a 30-layer DeltaNet linear attention architecture).

---

3. Installation & Out-of-the-Box Baseline

The runtime engine was built from source inside the root profile:

# Clone the codebase and enter root workspace
git clone https://github.com/JustVugg/colibri
cd colibri

# Kick off hardware mapping and model provisioning
./start-here.sh

The automated script compiled the optimized C binary for our CPU architecture and downloaded the model weights directly to the default catalog path: /root/colibri-models/qwen36-35b.

Initial Baseline Test

We initiated an interactive CLI chat session against unoptimized file-system buffering structures:

c/coli chat --model /root/colibri-models/qwen36-35b

• Prompt Passed: "Hello! Can you help me write a quick Python script?"
• Baseline Benchmark: 65 tokens generated in 71 seconds ── 0.91 tok/s.
• Diagnostic: The CPU spent precious cycles idling, heavily throttled while waiting for standard Linux OS page buffers to cache and deliver structural expert weights token-by-token.

---

4. Hardware Profiling & Tuning Sweeps

To extract better thread efficiency, we executed Colibri's internal automated parameter sweeper:

c/coli tune --model /root/colibri-models/qwen36-35b

Telemetry Logs & Analysis:

1. VRAM / RAM Freed: On load, 291 dense matrices were dynamically quantized, freeing up 7.2 GB of f32 memory overhead.
2. Persistent Footprint: Core embeddings and attention heads settled at 4.83 GB of Resident Set Size (RSS) RAM.
3. Tuning Sweep: The engine stress-tested different configurations, tracking latency per forward pass across thread thresholds (omp-3) and shifting sliding expert caches (cache-75, cache-50):
   • Turn 1 Evaluation: T(1) 231.9 ms down to T(1) 251.0 ms across cache variations.
   • Turn 2 Evaluation: Clear acceleration showing T(1) 111.3 ms to 127.3 ms.
4. Tuning Verdict: The tool verified that the native base configuration was already ideal for our 12-core topology: ✓ baseline retained: no candidate cleared the 3.0% gain and safety gates. It permanently locked the hardware profile signature at /root/.config/colibri/tuning/76677b91b02463e62dca.json.

---

5. Bypassing Page Cache with Direct I/O

Because Mixture-of-Experts models rely on streaming matrix shards continuously from storage layers, system page cache buffering creates an inherent bottleneck on CPU setups. We injected the DIRECT=1 runtime flag to enforce O_DIRECT, forcing the binary to read weights straight from the disk controllers:

DIRECT=1 c/coli chat --model /root/colibri-models/qwen36-35b

The Optimization Matrix

Strategy	Execution Environment Flag	Target Layer	Performance	Evaluation
Out-of-Box Baseline	c/coli chat --model [...]	Standard OS Page Cache	0.91 tok/s	Throttled by standard OS disk-read latencies.
NUMA Node Optimization	COLI_NUMA=1 c/coli chat [...]	Remote Socket Interleaving	0.91 tok/s	Neutral performance impact due to single physical NUMA node topology.
Hardware-Aware Tune	c/coli tune --model [...]	Core Thread/Cache Sweeping	0.91 tok/s	Confirmed 12-thread capacity limits and cached local signature.
Direct Hardware I/O	DIRECT=1 c/coli chat [...]	Direct Disk Reads (O_DIRECT)	1.00 tok/s	~10% overall decoding speedup by completely bypassing kernel cache delays.

---

6. Engineering Conclusions

By adjusting runtime flags and using Direct I/O (DIRECT=1), generation speeds increased from 0.91 tok/s to 1.00 tok/s.

On pure CPU architectures, once software bottlenecks are eliminated, performance scales strictly with physical hardware bandwidth. Since MoE layers demand heavy disk I/O on every sequential token pass, achieving higher token velocities (e.g., matching the 6.0 tok/s to 9.9 tok/s thresholds seen in project benchmarks) requires migrating the underlying storage blocks to high-performance PCIe NVMe M.2 Solid State Drives.

---

Big thanks to the contributors and maintainers of Colibri for keeping this engine dependency-free and making local inference on budget hardware highly accessible!