RFC: Laya routes by request class, not by asking for a device

[bong-water-water-bong]

Problem

Lemonade's onebit recipe starts 1bit serve --device auto by default, and --device auto still means "Vulkan" for every .gguf. The Laya router (step 4: #18, #90, #91, #93) only runs when someone passes --laya-model DIR by hand, and nothing in the packages or the recipe does that. So Laya is built and gated, but no real request goes through it.

Asking Laya the routing question directly ("which device should run this request?") is not usable as a default. Measured 2026-09-28 on Strix Halo, pinned root checkpoint, 1bit route, eight requests (code, "hi", a translation, a contract summary, a B-tree explanation, arithmetic, a large refactor, a haiku):

Candidates	Picks
vulkan,hrx,zinc	zinc 8 of 8
npu,hrx,vulkan,zinc	npu 4, zinc 4

The picks do not follow the request, and they can't: the model has no knowledge of this hardware. ZINC runs 64 architectures and is not the fastest device on this box for most of them. Each decision also costs 0.38 s before the first token.

Design

Laya answers questions about the request, which it is trained for. A measured policy picks the device and settings. Proposed:

1. Classify. laya/route asks typed questions whose answers are properties of the request: task kind (code, prose chat, short answer, long-document work) and whether it needs a long context. The answers come with Laya's calibrated confidence; below a threshold the request falls back to the default route.
2. Policy table. config/route-policy.json maps (task kind, model family) to a device and serve settings, filled only from measured results in the docs. For example, Qwen3.8-27B: code and prose go to Vulkan with the DFlash2 drafter (45.7 / 28.3 tok/s against 31.7 / 25.8 with MTP, docs/serve.md), and short answers go to Vulkan with MTP (28.2 against 17.8). A model with no rows keeps today's route. Embedding and reranking models never go through Laya.
3. Once per conversation. serve caches the class per conversation (the leading messages' hash), so the decision cost is paid once, not on every turn.
4. On by default when installed. --device auto uses Laya when a checkpoint is found (--laya-model, else $ONEBIT_LAYA_MODEL, else ~/.local/share/1bit/laya), and Vulkan otherwise, as today. --no-laya turns it off. scripts/fetch-laya.sh installs the pinned checkpoint to that path.

Files: laya/route.{h,cpp}, app/serve.cpp, config/route-policy.json, docs/laya.md, docs/serve.md, scripts/fetch-laya.sh, packaging.

Alternatives

• Keep it opt-in. Costs nothing, but Laya never runs for real users.
• Ask Laya for the device directly (today's --laya-model behaviour). The table above shows why not.
• Fine-tune a routing head on our own measurements (device features in the state, measured tok/s as the target). It could learn devices, but it needs a labelled dataset we don't have yet. The policy table would produce that data.
• Heuristics without Laya (keywords, prompt length). Cheaper, but brittle across languages and phrasing, which is what Laya is for.

How it will be checked

• Classifier accuracy: a fixed set of 200 labelled requests (code / prose / short / long-document) in tests/laya_route_cases.json. Report accuracy and confidence calibration of the typed answers against the labels, on the pinned checkpoint.
• End to end: tests/laya_route_e2e.sh extended, with requests of each class reaching the backend the policy names (fake backends).
• Speed: time to first token with and without Laya (the first turn pays the decision, later turns must not), and decode tok/s per class against the plain Vulkan route, on Strix Halo with Qwen3.8-27B and Qwen3-Coder-30B-A3B.
• Fallbacks: no checkpoint means today's behaviour, byte for byte on serve_e2e; low confidence means the default route.

Dependencies, licences and security

No new dependency: Laya (Apache-2.0) and its checkpoint pin are already in NOTICE and config/laya.json. No network exposure, secrets or workflow changes. The checkpoint (842 MB for the root checkpoint) is installed by fetch-laya.sh, not bundled in the package.

Open questions

• Which typed questions and labels to use, and whether the typed-decisions/ checkpoint answers them better than the root checkpoint.
• The confidence threshold for falling back.
• Whether the default-on part should wait for a release cycle as opt-in (--laya auto) first.

[bong-water-water-bong]

Accepted by the maintainer (2026-09-28). Building it; the PR will link here.

[bong-water-water-bong]

Built and merged in #187. Answers to the open questions, measured: the typed-decisions checkpoint classifies best (95.5% vs 92.0% root, with long_doc decided by size); the confidence threshold is 0.16 (the least confident quarter is 84% right, the rest 98-100%); and it ships opt-in (--laya), since llama-server ignores per-request speculative settings and no class has a measured faster device yet. Details in docs/laya.md.