[RFC] Shared accelerator operation contract

[Kenneth-Javier]

Summary

This is meant as the compute-side counterpart to #1057. That RFC established
how Colibri becomes model-agnostic without a generic forward: specialized family
engines, shared runtime mechanisms, migrated one consumer at a time behind a
contract that is the only way. This RFC proposes the same treatment for the
device side.

The starting point is not a design. It is an operation that four backends
already implement with the same meaning and nearly the same arguments, the
quantized matmul, and that each engine currently reaches in its own way. The
proposal is to make that existing contract explicit, mandatory and tested, and
then to use it as the template for the next operation.

No generic forward, no new runtime dependency, and no change of placement or
numerics in the first step.

Evidence (dev 4e28e398, read-only)

Four backends already agree on the operation. Same arguments, same order,
same "1 = handled, 0 = not mine" return:

backend	entry point
CUDA / HIP	coli_cuda_matmul(ColiCudaTensor **t, y, x, weights, scales, fmt, S, I, O, device, gs)
Metal	coli_metal_matmul(ColiMetalTensor **t, y, x, weights, scales, fmt, S, I, O, gs)
Vulkan	coli_vk_matmul(ColiVkTensor **t, y, x, weights, scales, fmt, S, I, O, gs)
XDNA2	coli_xdna_try_matmul(family, ColiXdnaPrepared **slot, fmt, q4, scale, I, O, gs, planar, y, x, S)

Two more entry points are the same operation in another form: coli_metal_gemm
(fmt 1/2/4, the same parameters, but no per-tensor handle: a synchronous GEMM
over unregistered weights) and coli_vk_matmul_pair (two quantized matmuls that
share one input, fused into one submit). The contract has to cover both: a
provider may keep no per-tensor state, and a call may carry two outputs over one
input.

The semantic common core is y, x, weights, scales, fmt, S, I, O, gs.
Provider-local prepared state (ColiCudaTensor, ColiVkTensor,
ColiMetalTensor, ColiXdnaPrepared) may exist, but it belongs to the
provider's resource plane, not to the operation. XDNA2 adds
the two things a GPU-only design never had to spell out: a semantic tag
(family, which qualification is bound to, never inferred from shape) and the
weight layout (planar).

The engines reach it in different ways. 18 direct calls in four engines:
colibri.c 6, kimi_k3.c 7, qwen36_tier.c 3, glm53.c 2. Inside colibri.c
alone, Metal and CUDA/HIP are tried inside matmul_qt_ex (:1236), while Vulkan
and XDNA2 are called at individual sites.

Placement is decided three different ways:

backend	what decides whether a tensor runs there
CUDA / HIP	data: QT.cuda_eligible, set at load
Vulkan	code location: 7 dense call sites, plus VK_FMT_OK, Vulkan's format rule copied into the engine in 7 places
XDNA2	code location + semantic tag + artifact registry

That is also why the obvious refactor is the wrong one. Folding Vulkan and XDNA2
into matmul_qt_ex would change which tensors may try them. The request has to
be separated from placement before dispatch can be centralised.

What is not a compute contract: ColiEdgeAdapter and the Segment runtime are
engine-level ABIs for distributed inference. They are the idiom to follow
(struct size, ABI version, explicit registration, opaque implementation), but
Colibri has no device-level operation contract today.

Invariants

1. No generic forward. Engines stay specialized and keep their math. They
   consume operations; they stop owning accelerator implementations.
2. Operation, placement and implementation are separate. The request says
   what to compute. Policy says which implementations may try. A provider
   says how.
3. No vendor, runtime or device in the shared surface. No cuda_device,
   XRT context or AIE tile in a request; those stay inside providers.
4. Exactness is first-class. README.md:36 forbids automatic BF16
   substitution, so a reduced-precision provider is eligible only on an explicit
   request, and measured placement compares only providers in the same
   exactness class.
5. Qualification follows meaning, not shape. Two matmuls with the same M/K/N
   need not have the same right to use the same accelerator artifact.
6. Mandatory, not optional. Migrated engine code may invoke an operation
   only through the shared contract, and a test fails if it does not. This is
   the [RFC] Model-agnostic Colibrì without a generic forward: shared runtime, specialized family engines #1057 rule: a half-finished abstraction is worse than none. How the test
   detects a bypass (today, a scan for backend entry points) is an
   implementation detail; the invariant outlives any renaming of backend
   functions.
7. Backend ABI evolution is explicit and fail-closed. Migrating every in-tree
   caller does not mean reusing a symbol with a new signature across the
   runtime-loaded DLL boundary (coli_cuda.dll, coli_hip.dll). A host must
   never call an old backend with a new calling convention.
8. CPU stays the reference and the fallback. Turning it into a provider is a
   later step, not part of the first migration.
9. Testable without accelerator hardware. CI already runs Vulkan on Lavapipe
   and XDNA2 against synthetic helpers; every phase keeps that property.

Design sketch

Split what lives as long as the weights from what changes per call:

/* Created once, when the tensor is loaded or registered. */
typedef struct {
    uint32_t          struct_size;
    uint32_t          abi_version;
    int               fmt, I, O, gs;
    ColiWeightLayout  layout;
    ColiSemanticOp    semantic;    /* e.g. COLI_SEM_MOE_SHARED_GATE_UP */
    ColiExactness     exactness;   /* REQUIRED / REDUCED_ALLOWED */
} ColiQMatmulDesc;

/* Per call. A pair call (two outputs over one input) carries two
 * descriptors and two outputs; the form is fixed in the B2 PR. */
typedef struct {
    uint32_t     struct_size;
    uint32_t     abi_version;
    float       *y;
    const float *x;
    int          S;
} ColiQMatmulCall;

Providers keep their existing per-tensor state (ColiCudaTensor, ColiVkTensor,
ColiXdnaPrepared), or none, as Metal's GEMM does today. The first migration
does not move tensor ownership.

XDNA2's COLI_XDNA_FAMILY_* becomes a shared ColiSemanticOp, because it was
never an XDNA concept: it is the qualification key for what an operation means.

Capability moves, placement does not. Today the engine decides both whether
a tensor may go to Vulkan and whether Vulkan can handle its format. After the
first step the engine still makes the first decision; the provider answers the
second (VK_FMT_OK becomes a Vulkan-side supports(desc)).

The DLL boundary. The loader already binds each coli_cuda_* symbol by name
and treats a missing mandatory symbol as fatal for the backend, with a
diagnostic naming it (backend_loader.c, RESOLVE). A new, versioned entry
point, e.g. coli_cuda_qmatmul_v1(ColiCudaTensor **, const ColiQMatmulDesc *, const ColiQMatmulCall *), resolved as mandatory, means an older DLL is refused
cleanly rather than called with the wrong arguments. The two mechanisms cover
different things: the versioned symbol governs the calling convention, and
struct_size / abi_version govern how the structures behind it evolve. A
get_api(requested_version) table can replace per-symbol evolution later, if
the surface grows.

Phased migration

Each phase follows the #1057 discipline: coverage lands first, one mechanism
per PR, and nothing optional is left behind.

B2: the quantized-matmul contract

• Coverage first. Bit-exact validation against the CPU reference for exact
  providers CI can execute, currently Vulkan/Lavapipe. For reduced-precision providers, explicit expected-result or
  tolerance checks plus eligibility checks; the synthetic XDNA2 helpers validate
  dispatch and failure semantics, including the existing guarantee that every
  failure falls back to a bit-identical matmul_qt result. And compile coverage for every
  caller being migrated, which is not there today: CI builds colibri with
  VK=1 but never glm53 or kimi_k3, and its METAL=1 loop covers
  colibri, inkling and kimi_k3 but not glm53. Five of the 18 calls
  (glm53 2, kimi_k3 3) would change signature with nothing compiling them. That gap exists
  regardless of this RFC; the first commit closes it.
• ColiQMatmulDesc / ColiQMatmulCall, including the pair form; every engine
  call site in scope and the tests migrated in the same PR. The counts in this
  RFC are baseline facts for 4e28e398: implementation starts by re-auditing
  current dev and migrates every caller in scope at that point.
• The invariant-6 test. Its first implementation scans engine sources for the
  backend entry points of this operation (coli_cuda_matmul,
  coli_metal_matmul, coli_metal_gemm, coli_vk_matmul,
  coli_vk_matmul_pair, coli_xdna_try_matmul), with the provider files
  exempt, derived from the sources rather than from a hand list. Each name was
  checked to be this operation and nothing else, so the scan forbids no
  unrelated backend code.
• VK_FMT_OK leaves the engine.
• The CUDA/HIP DLL gets a new, versioned mandatory symbol; the old one is no
  longer resolved.
• Unchanged: which tensors try which backend, numerical paths, tensor
  ownership, dependencies.

B3: provider chain and policy. Placement becomes explicit alongside the
request, as policy / eligibility metadata rather than code location, never
inside the compute request itself. Dispatch becomes one provider chain, and measured selection within
an exactness class builds on the startup GEMV timing that v1.12.1 introduced for
Qwen3.6.

B4: a second operation, the expert group. CUDA and Vulkan already agree on
its signature (coli_*_expert_group); Metal's moe_block needs an adapter.
This is the phase that shows whether the contract generalises or merely fitted
matmul.

Non-goals

• A generic forward or a graph interpreter.
• A runtime compiler dependency. B2 introduces none. Triton-XDNA and MLIR-AIR
  are treated as artifact producers for an existing provider; runtime
  compilation is outside this RFC.
• New placement behaviour in B2.
• Turning the CPU path into a provider in B2.
• Inkling's dense BF16 GEMM (ink_cuda_matmul_bf16): a different operation with
  its own backend, left for a later phase rather than half-migrated now.

Questions for the maintainer

1. Is this a direction you want, and in this order?
2. Naming: ColiSemanticOp / COLI_SEM_* for what XDNA2 calls a family?
3. Refusing an older coli_cuda.dll / coli_hip.dll outright, with the existing
   missing-symbol diagnostic: acceptable, given that host and DLL ship together?
4. Where should the bypass test live: beside test_makefile_deps.py, or in a new
   contract test?

Related

#1057 (shared runtime, specialized engines), #1261 (the XDNA2 lane that surfaced
the semantic tag and the layout dimension), #1590 (Linux portability, which a
provider boundary makes a runtime question rather than an engine change), #959.