[RFC] Model-agnostic Colibrì without a generic forward: shared runtime, specialized family engines

[terrizoaguimor]

Summary

Colibrì now supports five very different model families, but it still pays for many common capabilities once per engine. That is producing a recurring class of drift bugs: one family learns a fix or mechanism, while the siblings retain older behavior.

I would like to propose a specific meaning of model-agnostic Colibrì:

Keep architecture math and hot paths specialized per family, while making the control plane, runtime contracts, and reusable mechanisms model-agnostic.

This is not a proposal for a generic graph interpreter, a universal config.json loader, a Rust rewrite, or an immediate replacement of the one-C-file-per-family model. It is an incremental route to make a new family plug into one stable host contract instead of copying the launcher, serving, persistence, telemetry, storage, and cache plumbing again.

The key distinction is:

• Specialized below: attention, recurrent/KV state, residual topology, router semantics, expert activation, speculative decoding, prompt/tokenization details, and fused kernels.
• Shared above and around it: family identity/capabilities, Engine/Session lifecycle, serving frames, storage/catalog, ExpertStore leases, sampling policy, route-history lifecycle, telemetry, backend capability checks, planning, build/install/release conformance.

This keeps the performance identity of Colibrì while reducing the maintenance cost of adding and operating model families.

---

Why discuss this now

The project has already evolved from one GLM engine into a family roster:

• GLM-5.2: c/colibri.c
• Inkling: c/inkling.c
• Kimi K3: c/kimi_k3.c
• OLMoE: c/olmoe.c
• DeepSeek V4: c/deepseek_v4.c

More families are proposed or under review, including Qwen3.6, MiniMax-M3, Hy3, Mistral Large 3, Dots3 and others.

The separate-engine policy has real benefits:

• isolation;
• family-specific optimization;
• simpler numerical reasoning;
• smaller regression surface;
• one tiny/oracle contract per family;
• no giant union of every architecture state.

I do not think those benefits should be discarded.

The problem is that the isolation boundary currently includes much more than model math. Every engine has grown its own version of pieces that should have one contract. Across the five engines there are roughly 27K lines of C, and the most dangerous duplication is not the attention arithmetic. It is at the edges.

---

Evidence: drift has become a recurring defect class

Several issues were not isolated coding mistakes; they were consequences of a mechanism living in one engine or one dispatch table instead of a shared contract.

• The learning cache is engine-specific: .coli_usage has two incompatible writers and two engines that cannot produce it at all #700: .coli_usage had incompatible producers/readers and incomplete engine coverage. The successful fix was route_trace.h: one format, reader and writer, adopted incrementally.
• coli launcher can't run OLMoE: missing model_arch() branch + CHAT=1 never set #879: OLMoE was present in the roster/package but launcher dispatch still sent it to GLM.
• [Bug]: coli tune banners DeepSeek V4 but calibrates with the GLM engine and fails #898: coli tune announced DeepSeek V4 but invoked the GLM engine.
• deepseek_v4 accepts VK=1 and COLI_VULKAN=* and silently ignores them #894: V4 accepted Vulkan-related settings while no Vulkan path existed.
• The learned partial mirror is GLM-only, and Kimi K3 is the model that needs it most #812: mirror primitives exist in st.h, but read-splitting policy remained GLM-only and the planner did not understand K3 names.
• [Bug]: USAGE_SAVE=0 is silently ignored on the GLM engine - documented as shared, implemented in only 2 of 4 engines #1039: USAGE_SAVE=0 is presented as a shared control but is not honored uniformly.
• [Bug]: Loading Kimi K3 layers is slow, and then Colibri stops doing anything. #748 and related Windows fixes: protocol/binary-mode behavior had to be corrected across siblings.
• [Bug]: Web profiling tab on DeepSeek V4 is 100 percent on "other" #890: the common dashboard protocol received incomplete V4 phase telemetry.
• OpenMP sizing, stop behavior, RAM budgeting, command dispatch and backend notices have each needed follow-up work because a fix reached one execution path but not every family.

The pattern is important: a generic forward would not have prevented most of these bugs. A shared family/runtime contract would have.

---

What model-agnostic should mean

1. One family registry and capability contract

Today family detection and metadata are repeated across the launcher, gateway, doctor, planner, autotune, build targets, installation, release packaging and tests.

A family descriptor should answer, from one source of truth:

family id
accepted model_type values / detection rule
engine binary or linked plugin
build target
platform support
default model id
prompt renderer/tokenizer adapter
context/default limits
tools / grammar / audio / thinking capabilities
CUDA / HIP / Metal / Vulkan capabilities
tiny oracle command
install/release artifact expectations

Unknown architectures should be refused explicitly, not silently treated as GLM.

The first version does not need to be a C plugin registry. A shared control-plane registry consumed by coli, openai_server.py, doctor and tests would already remove a large amount of drift.

2. A common Engine/Session lifecycle

DeepSeek V4 already demonstrates a useful shape in deepseek_v4.h:

• Engine owns config, weights, storage, expert cache and backend state.
• Session borrows Engine and owns sequence/recurrent state.
• Sessions must be destroyed before Engine.
• Generation reports structured stats and emits tokens through a callback.

A neutral API could eventually look like:

typedef struct ColiEngine ColiEngine;
typedef struct ColiSession ColiSession;

int coli_engine_open(...);
void coli_engine_destroy(...);

int coli_session_create(...);
int coli_session_generate(...);
int coli_session_reset(...);
void coli_session_destroy(...);

The family implementation remains responsible for its private state and math. The common host only needs lifecycle, capabilities, errors, memory summary, request options and token events.

This should first be implemented as adapters around existing engines, not as a rewrite of their forwards.

3. Opaque sequence state, not a universal KV tensor

A universal rectangular KVCache would be the wrong abstraction:

• GLM uses compressed MLA state and DSA information.
• Inkling combines K/V with multiple convolution states.
• Kimi K3 includes KDA recurrent matrices, convolution windows and MLA state.
• OLMoE uses conventional K/V.
• V4 uses window rings plus recurrent compressor/indexer state.

The common contract should be operational:

reset
append/prefill/decode
reuse or truncate a token prefix
snapshot
restore
report memory

The bytes and layout stay private to the family.

4. Shared serving codec, family-owned generation

The READY/SUBMIT/ACCEPT/DATA/ERROR/DONE protocol is implemented separately by every engine. The framing, limits and validation can be shared without sharing the scheduler or forward pass.

A common codec should own:

• parsing and validating frame headers;
• exact byte-counted payload reads;
• finite/range checks;
• STOP and CANCEL framing;
• canonical errors;
• READY, ACCEPT, DATA, ERROR, DONE emitters;
• Windows binary mode;
• optional extension payloads for audio/grammar.

Each engine would still own:

• request admission policy;
• KV/recurrent state;
• prefill and decode;
• token filtering;
• scheduling and batching.

This avoids putting callbacks inside hot arithmetic while removing five copies of protocol security and correctness logic.

5. Tensor and storage contracts

The project already has useful foundations:

• st.h: safetensors indexing and reads;
• tensor.h: non-owning semantic tensor views;
• quant.h: shared quantized CPU kernels;
• backend APIs for CUDA/HIP, Metal and Vulkan.

The next abstraction should be an inventory/catalog of named tensor metadata and validated views, not an interpreted graph of operations.

A family adapter should map checkpoint names and shapes into its own layer plan. The runtime should provide:

• bounded config/file reading;
• index ownership and errors that return rather than exit;
• validated tensor views;
• direct/buffered/mirror-aware read policy;
• backend capability queries;
• shared accounting.

This preserves architecture-specific naming and layout, including fused expert tensors and native MXFP4/FP4 formats.

6. ExpertStore as a strangler seam

expert_store.h already defines the most promising runtime vtable:

• (layer, expert) key;
• lookup/release lease contract;
• prefetch;
• stats;
• destroy.

An indirect call here is negligible relative to disk I/O or expert matmul. The migration should not replace every cache at once.

Suggested sequence:

1. Wrap each existing cache behind an adapter.
2. Preserve its current eviction, pinning and GPU placement behavior.
3. Normalize only key/view/lease/stats.
4. Add lookup_many or batch leases only after a real second consumer proves the requirement.
5. Consolidate policies later, one measured behavior at a time.

7. Shared usage-history lifecycle

route_trace.h unified the file format, but path selection, save controls and save cadence remain engine-specific.

A common policy should define:

• default <model>/.coli_usage;
• COLI_USAGE override;
• USAGE_SAVE behavior;
• per-turn and shutdown persistence;
• explicit stats export independent from disabling learned-history writes;
• backward-compatible aliases for engine-private variables during migration.

This is a small, low-risk first runtime extraction and would directly address #1039 and the remaining lifecycle inconsistencies.

8. Shared telemetry core

The dashboard protocol should have one implementation for:

• hardware probe;
• line framing/flush;
• STAT, HWINFO, common profile fields;
• tier/heat bitmap encoding.

Each family should expose an adapter that enumerates its sparse rows, tiers, heat and phase counters. This avoids claiming every model has the same layers or expert geometry while ensuring every shipped engine satisfies the same wire contract.

9. Backend capabilities before backend unification

A generic backend should not begin by lowering every family to matmul() calls. Colibrì benefits from family-specific fused paths and device-resident pipelines.

The safe first step is a capability contract:

backend compiled?
backend supported by this family?
formats/layouts accepted?
unified or discrete memory?
expert tier, dense path or attention path?
requested setting active, rejected or ignored?

Primitive operations can be shared. Fused family-specific extensions must remain possible.

---

What must remain specialized

The following are not good candidates for a generic interpreted IR:

• attention implementation;
• residual topology;
• router scoring, bias, tie-breaking and normalization;
• expert activation and latent projections;
• recurrent/KV representation;
• position encoding;
• speculative decoding and rollback;
• model-native prompt/token segmentation;
• fused CPU/GPU kernels;
• exact accumulation and rounding order.

Examples make the point:

• GLM has MLA + DSA + MTP.
• Inkling has global/sliding GQA, relative bias and short convolutions.
• Kimi K3 has KDA, gated NoPE MLA, AttnRes and LatentMoE.
• OLMoE uses conventional attention and softmax routing.
• V4 has compressed window attention, compressor/indexer recurrence, mHC and DSpark.

Trying to describe all of these as a dynamic list of primitive tensor operations would likely create intermediate tensors, barriers, lost fusion, device transfers and numerical drift. A coarse family plugin avoids that.

---

Performance model

A vtable call is not the relevant risk when it occurs:

• once per model;
• once per session;
• once per request;
• once per layer;
• once per expert lease;
• once per token sampling event.

Those costs are negligible relative to storage reads and matrix multiplication.

Callbacks should not occur inside:

• quant blocks;
• output rows;
• attention positions;
• recurrent heads;
• expert accumulation loops.

The rule should be: abstraction at operation/lifecycle boundaries; static specialized code inside hot loops.

---

Proposed architecture

OpenAI / Anthropic / TUI / CLI
               |
      Family registry + capabilities
               |
       Common engine host/runtime
               |
  +------------+------------+-------------+
  |            |            |             |
 GLM         Inkling      Kimi K3         V4       ...
 plugin       plugin        plugin         plugin
  |            |            |             |
  +------------+------------+-------------+
               |
 shared mechanisms:
 config I/O · tensor catalog · storage policy
 serve codec · usage lifecycle · telemetry
 sampling/stops · ExpertStore · backend capabilities

Initially, “plugin” may simply mean an adapter around a separate sibling binary. A stable in-process ABI should be considered only after multiple families implement the same lifecycle contract without losing correctness or speed.

---

Incremental roadmap

Phase 0: freeze an executable family contract

Before refactoring, add a conformance matrix for every shipped family.

For each family, test:

• model detection;
• actual executable selected for run, chat, serve, web, tune, doctor, build;
• prompt renderer or explicit unsupported response;
• capability matrix;
• build target;
• install artifact;
• release artifact by platform;
• tiny/oracle gate;
• planner support or explicit refusal;
• telemetry handshake.

This should be a test/check first, not a runtime rewrite. It would have caught several historical dispatch and packaging bugs.

Phase 1: one control-plane family registry

Move duplicated family metadata into one source of truth consumed by the launcher and gateway. Extend doctor/planner/build/release checks from that registry.

No C forward changes.

Phase 2: common usage-history policy

Normalize COLI_USAGE, USAGE_SAVE, default path and save cadence. Migrate one engine per PR. Keep explicit stats export independent.

This is a good first shared runtime mechanism because it cannot alter logits or token order.

Phase 3: common serve codec

Build transcript/golden tests first. Migrate framing only, in this order:

1. OLMoE
2. Kimi K3
3. V4
4. Inkling, including audio extensions
5. GLM multiplexed mode last

Do not move scheduling or KV ownership in this phase.

Phase 4: telemetry and config transport

Share:

• hardware probes;
• emitters/encoding;
• bounded file loading;
• typed JSON helpers.

Keep each family config struct and semantic validation private.

Phase 5: neutral Engine/Session API

Generalize the proven V4 lifecycle shape. First wrap a simpler sibling, probably OLMoE, without changing its forward.

Demonstrate two real consumers before expanding the API.

Phase 6: storage policy and ExpertStore adapters

Introduce:

• shared tensor catalog ownership;
• replica/mirror-aware read policy;
• adapters around existing caches;
• normalized leases/stats.

Do not change eviction, pinning, direct I/O and GPU placement simultaneously.

Phase 7: optional common host executable

Only after several families pass the same ABI and oracle gates:

• link them into one host experimentally;
• keep sibling binaries available;
• measure binary size, startup, RSS and throughput;
• decide from evidence whether one host is operationally better.

A single executable is an optional result, not the first goal.

---

Acceptance criteria for every migration

I suggest requiring all of the following:

• token-exact tiny oracle before/after;
• unchanged persisted formats unless migration is explicit and backward-compatible;
• no silent default changes;
• no callback in inner arithmetic loops;
• no new materialized intermediate tensors;
• ASan/UBSan clean;
• one family migrated per PR;
• unknown/unsupported combinations fail explicitly;
• local make check and all-engine CI remain green;
• performance within ±1% unless an improvement is separately measured;
• abstraction expanded only after a second real consumer requires it.

For model-specific additions, preserve the established bar:

• isolated family implementation;
• tiny/oracle fixture;
• architecture reference;
• real-checkpoint validation where possible;
• build/install/release integration;
• named ownership or hardware access.

---

Relationship to existing work

This proposal is intended to complement, not invalidate, existing directions:

• PR feat(core): model-architecture seam, chat templates, text antiprompt #667 correctly identifies the need for architecture identity and early refusal, but exposes a large GLM-shaped Model/Cfg seam before proving a second consumer. A smaller capability/lifecycle contract would be easier to validate.
• PR v4: pluggable expert-store backend registry #964 explores an ExpertStore registry for V4. The underlying ColiExpertStore interface is promising; a cross-family factory should eventually avoid V4-specific option types.
• PR Add a registry for the 214 environment variables, and check the environment against it #800 demonstrates the value of a checked environment registry. The family registry should provide extensible capability/prefix ownership rather than fixed engine bits forever.
• PR MiniMax-M3 support — GQA + MSA block-sparse attention, converter (follow-up to #418) #601 may provide a real test of two families in one core file, but it also shows why every shared hot-path change requires independent oracles.
• PR feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE #712 demonstrates the opposite strength: excellent isolation and almost zero regression surface. The runtime-contract proposal should preserve that property.
• [Feature]: Project Complexity Reduction #660 raises the legitimate complexity concern. This RFC keeps the dependency-free C engines and subprocess boundary; it aims to reduce duplicated engine plumbing, not pull ecosystem code into hot paths.
• Discussion Model agnostic? Or just GLM 5.2? #406 asked whether Colibrì is model-agnostic. The project now supports multiple families; this RFC asks how to make that support systematic.

The strongest positive precedent is #700: a duplicated cross-engine mechanism became one small header with primitive inputs, then each engine adopted it separately. That is the migration style proposed here.

---

Non-goals

To avoid ambiguity, this RFC does not propose:

• replacing C;
• adopting a third-party graph runtime;
• importing a dependency-heavy framework;
• supporting dense models automatically;
• loading any arbitrary Hugging Face architecture from configuration alone;
• making every backend work for every family immediately;
• combining all engines into one PR;
• removing sibling binaries before an in-process host is proven;
• trading token exactness for abstraction;
• slowing hot loops to make code look uniform.

---

Suggested first deliverable

The most useful first PR would be:

test/integration: define and enforce the sibling-family contract

It should contain tests and only the minimal mechanical fixes they expose. It should not introduce the Engine ABI yet.

A second PR could then introduce the shared family registry. A third could normalize usage-history policy. That sequence gives us measured, reviewable steps and lets maintainers stop or redirect the architecture before any risky runtime migration.

---

Questions for maintainers and contributors

1. Is this definition of model-agnostic compatible with the one-C-file-per-family identity?
2. Should the stable boundary remain subprocess-based indefinitely, or is an optional in-process Engine/Session ABI desirable after two adapters exist?
3. Would the family-conformance test be an acceptable first PR?
4. Which mechanism should follow the registry: usage lifecycle, serve framing, or telemetry?
5. Is expert_store.h intended to become cross-family, or remain V4-local for now?
6. Should a family registry cover build/install/release expectations, or only runtime dispatch?
7. What performance tolerance should block an abstraction migration: ±1%, tighter, or component-specific?
8. Which existing sibling should be the first Engine/Session adapter after V4: OLMoE, Inkling, or Kimi K3?
9. Should we formalize “two real consumers before generalization” as a project rule?
10. Are there architecture directions in feat(core): model-architecture seam, chat templates, text antiprompt #667, MiniMax-M3 support — GQA + MSA block-sparse attention, converter (follow-up to #418) #601, feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE #712, Add a registry for the 214 environment variables, and check the environment against it #800 or v4: pluggable expert-store backend registry #964 that this plan should explicitly preserve or avoid?

If the direction is welcome, I am willing to start with the conformance test/registry work and keep every step narrowly scoped, oracle-backed and independently revertible.

[terrizoaguimor]

@JustVugg Take a look at this, I think is a good roadmap.

[JustVugg]

This is the right diagnosis, and I can give it stronger evidence than the issue list: the drift class you describe produced the same bug twice today, in two unrelated PRs, found by two independent reviews.

Today's proof

I spent today reviewing the two model-family PRs currently in flight — #712 (Qwen3.6) and #601 (MiniMax-M3). They share no code, no author, and no design: Qwen is a standalone qwen36.c, MiniMax is ARCH_M3 inside colibri.c. Two different people, two different shapes of contribution.

Both fail the planner in exactly the same way:

resource_plan.py reads kv_lora_rank, qk_rope_head_dim, qk_nope_head_dim, v_head_dim, n_routed_experts. Neither family's config carries those keys. Measured by running build_plan() on a converted container of each: kv_bytes = 0, configured_experts = 0.

So coli plan, coli doctor and --auto-tier under-report the KV pool to zero for both — roughly a GB at 60 layers × 4096 context on the MiniMax side. That reaches every number a user sizes their machine from. Neither author did anything wrong; the planner simply encodes MLA-shaped assumptions with no contract saying what a family must declare.

The rest of the Qwen review reads like a checklist generated from your RFC:

Finding	Your section
Missing from cmd_build's dict → KeyError traceback on the command coli info tells the user to run	§1 registry
Missing from _BANNER_MODELS → zero dispatch test coverage, because test_launcher_dispatch.py and release.yml both iterate that table	§1 registry
Not in ci.yml ENGINES or release.yml SIBLINGS → a released archive dispatches to a binary it does not contain	§1 registry (install/release artifact expectations)
"qwen" in model_type claims the entire Qwen namespace, capturing models that previously fell through to GLM	§1 (detection rule; "unknown architectures should be refused explicitly")
Missing from the response_format reject list → a grammar request leaves unread bytes in stdin and permanently desyncs the engine	§4 serving codec
coli stop's process-name list has no entry → a multi-GB engine survives	§1 registry
Engine caps context at 8192 while the launcher sends --max-tokens 16384 → HTTP 400 on every chat message	§1 (context/default limits)

And MiniMax's headline blocker is §4 verbatim: the EOS fallback was added to run_serve but not to run_serve_mux — which is the entry point openai_server.py actually uses, since it always launches with SERVE_BATCH=1. One of three serve paths learned the fix. That is the drift mechanism in a single sentence.

What that changes about the proposal

It moves this from "reasonable refactor" to "we are currently paying this cost per PR, in review time." Every new family costs a full manual integration audit precisely because there is no contract to check against — and the audit is unreliable, since both of today's blockers were found by reading rather than by any test.

Your §1 is also, by some distance, the cheapest item on the list. A family registry alone would have caught six of the eight rows above, and it requires touching no forward pass, no kernel and no engine .c — it is a descriptor table consumed by coli, openai_server.py, doctor.py, resource_plan.py and the tests. That asymmetry is worth stating explicitly: the highest-value piece of this RFC is also the one with the least risk attached.

The precedent is already in the tree, and it is yours: #969 consolidated expert history into route_trace.h and came out net −35 lines while doing more. That is the shape — one format, one reader, one writer, adopted incrementally — and it has now been proven once in production.

Where I would push back

A half-finished abstraction is worse than none. If the registry lands but stays optional, family six routes around it and we have added a layer while keeping the drift. The registry has to become the only way a family is declared, with a test that fails when a family exists outside it — the same trick test_launcher_dispatch.py already plays with _BANNER_MODELS, which is exactly why we noticed Qwen's gap.

§3 is the one I would guard most carefully. You are right that a universal rectangular KV tensor is the wrong abstraction, and the reason is visible in today's reviews: Qwen's DeltaNet recurrent state cannot be prefix-reused the way GLM's KV can, so "reuse a token prefix" is not a uniform operation — for some families it is correctness-bearing rather than an optimization. The operational contract needs to let a family answer "I cannot do that" without the host assuming it can.

Sequencing. I would take it strictly in this order, each landing separately with its own test: registry (§1) → serving codec (§4) → ExpertStore seam (§6) → Engine/Session adapters (§2). §5 and §3 are the ones to design last, once three families have gone through the first four and the contract has been falsified by real use.

Concretely

If you want to take §1, I will review it as a priority, and I would suggest scoping the first PR to: the descriptor table, the four Python consumers, and a test that fails when a known family is missing from it — with Qwen and MiniMax used as the fixtures, since we now have two documented cases of exactly what it must catch.

Thank you for writing this up properly, with the bug list rather than the aesthetics. The argument that a generic forward would not have prevented these bugs, but a shared runtime contract would, is the correct framing and it is now supported by two same-day measurements.

[terrizoaguimor]

Roadmap update after #1063:

• Registry contract: merged in refactor(core): make model families registry-owned #1063.
• Registry drift completion: refactor(core): finish registry-owned family dispatch #1068 is now open. It follows the maintainer-requested shape exactly: all FamilyDescriptor sites use keyword arguments, new fields are defaulted, the four launcher adapter allow-lists are gone, and tune replay templates are registry-owned. It deliberately does not move behavior-specific openai_server.py branches or add speculative capability flags.
• Per-family planner measurements: remain separate in Planner: per-family adapters needed — Kimi K3, Inkling, OLMoE and V4 now refuse instead of reporting zero-byte KV #1066 and can proceed in parallel.
• Next runtime step after refactor(core): finish registry-owned family dispatch #1068: shared serving codec, beginning with transcript/golden tests and OLMoE framing only; then the ExpertStore seam; then Engine/Session adapters, preserving the sequence agreed above.

Validation on #1068: full make check passes (528 Python tests, 60 skipped, complete C suite).

PR: #1068