Skip to content

feat(cli): larql run <vindex3-container> — the container speaks for itself - #382

Merged
chrishayuk merged 1 commit into
mainfrom
run-vindex3
Sep 1, 2026
Merged

feat(cli): larql run <vindex3-container> — the container speaks for itself#382
chrishayuk merged 1 commit into
mainfrom
run-vindex3

Conversation

@chrishayuk

Copy link
Copy Markdown
Owner

What

larql run <vindex3-container> [prompt] — text in, text out, from the container's own program. run() detects a VINDEX3 container from its index.json generation before any VINDEX2-only reader opens it and routes to run_cmd_vindex3: the container's tokenizer, greedy decode through the same interpreter larql vindex3 exec reports on, text streamed as it is produced, EOS from the container's generation_config.json. Weights load once; every prompt (and every chat line) gets a fresh continuation state.

  • --metal selects the Metal realisation (macOS, gpu feature); otherwise the larql-compute CPU kernels.
  • Dense-engine flags (--top, --kv-cache, --engine, --ffn, --routed-from, --moe-*, --experts*, --ops, --constrained, --image, --mm-weights) are refused by name, in one message, never ignored.
  • --emit-ids prints prompt and generated ids to stderr, so a run doubles as an id-level oracle for vindex3 exec --tokens.
  • A container encoded without tokenizer.json is refused with a pointer to vindex3 exec --tokens.

Structure

The setup run_exec owned (inspect → plan → operand store → backend choice) moves to vindex3_cmd/prepare.rs behind a BackendVisitor (backends are distinct concrete types, so the choice is made once and the visitor sees the type). The greedy loop moves to vindex3_cmd/decode.rs with a token sink that can halt the turn; run_generate is now the exec verb's report wrapper and returns the ids. Note: prepare is a second opener beside larql_inference::vindex3::runtime::open_component (#381 made that one policy-aware); the stacked PR deletes this duplicate and delegates — that is the step that makes the opener the single authority.

Witness (release build, "The capital of France is", 12 tokens)

container backend output tok/s ids vs vindex3 exec --tokens
oute-mamba2attn-250m production (CPU) " the city of Paris." 82 identical
qwen3-4b --metal " Paris. The capital of Germany is Berlin." 17.4 identical

Gates

21 new tests (loop, dispatch, detection, refusals, EOS by id and by stop string, chat-turn isolation, Metal on the fixture). cargo fmt --all --check; clippy --bins --tests --no-deps -- -D warnings under default features and --no-default-features; cargo test -p larql-cli in both shapes (805 CPU-only); registry check; doc link/reference gates. Coverage in the CI shape: decode.rs 100%, run arm 91.5%+, prepare.rs 82% (the untested arm is the pre-existing "plan does not close" refusal, which needs binary-segment surgery to reach).

Docs

docs/cli.md (run section + exec table), docs/vindex3-registry-design.md (the recorded gap is closed), README (one sentence). The stale "no KV cache" statement on exec --generate is corrected.

… itself

run() detects a VINDEX3 container from its index generation and routes
to run_cmd_vindex3: tokenizer from the container, greedy decode through
the same interpreter `vindex3 exec` reports on, text streamed, EOS from
generation_config.json. The setup run_exec owned (inspect, plan, operand
store, backend choice) moves to vindex3_cmd::prepare so both verbs share
one path; the greedy loop moves to vindex3_cmd::decode with a token
sink, and run_generate returns the ids. Dense-engine flags are refused by
name. --emit-ids makes a run an id-level oracle: identical to exec on
oute-mamba2attn-250m (CPU) and qwen3-4b (Metal).
@chrishayuk
chrishayuk merged commit b898c8c into main Sep 1, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant