Compact Causal Masks

Compact causal masks

An ordinary causal attention mask stores a value for each query/key relationship. For a strictly ordered causal prefix, the same visibility rule can be represented by one boundary index per query row. This feature builds and consumes that compact representation automatically when the graph proves it is equivalent.

Eligibility and fallback

There is no user-facing flag. The path requires Flash Attention, a simple ordered causal prefix, supported cache ordering, and backend support for the compact I64 boundary representation. Multiple active streams, gaps or reordered cells, sliding-window attention, ALiBi, model-specific mask edits, and other richer patterns keep the normal dense mask.

The fallback is intentional and preserves established arithmetic. Compact masks reduce mask allocation and host/device mask traffic; they do not reduce persistent KV and are not guaranteed to improve every model.

Note

Backend scope: CPU and CUDA implement the compact representation at the audited tips. Other accelerator backends use the dense mask. This is not a generic claim that every Flash Attention backend supports compact masks.

Branch implementations

Branch	Graph selection	Backend consumers
llama/dev	src/llama-graph.cpp	CPU, CUDA
moe-cache	src/llama-graph.cpp	CPU, CUDA

beellama/main does not contain this feature at the audited published tip. Its proposed port remains PR 7, so this wiki does not claim it as available there.

Design history for the shipped implementation: PR 53 and PR 54.

Feature index