Skip to content

CMF 0.6.6 — bounded O(1) and faster GPU readout

Latest

Choose a tag to compare

@infosave2007 infosave2007 released this 08 Sep 20:40
12f5852

CMF 0.6.6 improves the experimental O(1) attention path used with Qwen3.8-27B. Short prompts can retain their exact startup prefix until the bounded handoff is ready; generation and batch transitions now preserve state and propagate failures consistently.

The GPU readout computes independent landmark columns in parallel while preserving the summation order within each column. On one RTX PRO 4000 Blackwell 24 GB, three alternating pairs at 8,192 input / 128 output tokens reduced median time to first token from 73.82 to 68.31 seconds (7.46%) and increased median decode throughput from 25.77 to 26.17 tokens/s (1.56%). These compare two CMF O(1) implementations with the same profile, not CMF against another engine.

The measured profile uses 32 landmarks, a 128-token near window, 4 sink tokens and a 256-token startup prefix. Its sealed GPU attention state is 46,236,672 bytes (44.09 MiB). Model weights, recurrent state, host copies and working buffers are additional. O(1) describes state growth with context length for fixed parameters; it does not imply exact unlimited recall or constant total prompt-processing time.

The wider near-window path supports experimentation up to 2,048 tokens. The validated default remains 128. This release does not claim improved model quality or results from native training.

CPU/GPU parity coverage includes the wider window and empty distant-state branch. The real-tape replay test remains explicitly opt-in because it needs an external retained tape.

Packages: cortiq-core, cortiq-engine, cortiq-net, cortiq-server, cortiq-cli, cortiq-ffi 0.6.6. Platform artifacts include the CLI for Linux, macOS, Windows and Android, plus Android/iOS FFI libraries.

The public Rust lifecycle additions are Pipeline::o1_begin_with_prefix and Pipeline::o1_seal_checked. Existing entry points remain; deferred transition failures propagate through the existing forward/scoring Result paths.

Validation before tagging: complete Ubuntu/macOS CI and clippy, 3/3, and all nine platform builds, 9/9. The merged release tree is identical to the tested PR tree. See PR #9 and the changelog for details.