mlops-vocab 0.1.0

mlops-vocab 0.1.0

Part of mlops: CUDA operations for NVIDIA GPUs sm_50 … sm_90.

Install

Nothing here downloads anything. Install first:

• PyTorch (CUDA build);

Then, in the directory with the downloaded wheel (cpXY = your Python, torchA.B / cuNNN = your PyTorch and its CUDA):

pip install --no-index -f . mlops-vocab

--no-index takes mlops-vocab from this directory and its dependencies from what is installed, never from PyPI.
Other combinations: ./setup.sh vocab from the source archive.

C / C++: unpack mlops-vocab-0.1.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libmlops_vocab.a (see its README).

Every artifact carries SASS for sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

• mlops-vocab-0.1.0-linux-x86_64-cuda12.tar.xz (15M)
• mlops-vocab-0.1.0-src.tar.gz (80K)
• mlops_vocab-0.1.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (35M)

Changes

First release. Everything along the vocabulary axis, sm_50 .. sm_90.

• mlops_embed. The rows of device-side token ids (table in VRAM or in mapped host memory) widened to FP32, with
  bf16(x * w) and the sums of squares of 256 elements: the inputs of the first RMSNorm. An id outside the vocabulary
  gives a zero row and sets a flag.
• mlops_lm_head. The BF16 head for 1 .. 8 tokens, cut along hidden: the S warps of a CTA take disjoint ranges
  of the columns and their partial sums add in warp order in shared memory, so the logits do not depend on how many
  elements the rows are split over. CUDA cores on every architecture; tensor cores from Volta (m8n8k4, sm_70 / 72),
  Turing (m16n8k8, FP16 converted from BF16 in registers) and Ampere on (m16n8k16, BF16). The epilogue applies the
  deferred RMSNorm scalar and logit_scale and can write each element's maximum, lowest argmax and key-byte counts
  for the sampler. The instances of a shape are listed as data (mlops_lm_head_variants); the call and the items run
  the one the caller names. On an L40S it reads Qwen3-8B's 151936 x 4096 matrix at 750 GB/s (98 % of the DRAM peak);
  on a GTX 1080 at 250 - 255 GB/s, the speed of simple streaming reads there.
• mlops_sample. Greedy, temperature, top-k, top-p and a repetition penalty without sorting the vocabulary. The
  masses are counted in fixed point, so a token depends on (logits, configuration, seed, step) alone, not on the
  grid or the device; the seed and the step live on the device. One call is 2 - 66 us on a GTX 1080 and 3 - 39 us on
  an L40S at 151936 tokens; with the head's statistics greedy needs no pass over the logits.
• Items (mlops_vocab_items.h, .cuh): the work of each call as CTA-sized items with their entries for NVRTC and
  the ranges they read and write.
• Python mlops.vocab: embed, lm_head, Sampler, embed_variants, lm_head_variants.
• The Volta and Turing instances have only been run on an L40S (where their layouts are checked), not on their own
  GPUs.
• Tested: GTX 1080: the C tests (91 checks: bit for bit against host emulations; the tensor-core instances against the
  exact sum within a bound), pytest (14), the samples, the wheel in a fresh venv, the C archive from plain C.
  sm_89 (48 GB): the C tests, including every tensor-core instance, and pytest.

mlops-norm 0.1.0

mlops-norm 0.1.0

Part of mlops: CUDA operations for NVIDIA GPUs sm_50 … sm_90.

Install

Nothing here downloads anything. Install first:

• PyTorch (CUDA build);

Then, in the directory with the downloaded wheel (cpXY = your Python, torchA.B / cuNNN = your PyTorch and its CUDA):

pip install --no-index -f . mlops-norm

--no-index takes mlops-norm from this directory and its dependencies from what is installed, never from PyPI.
Other combinations: ./setup.sh norm from the source archive.

C / C++: unpack mlops-norm-0.1.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libmlops_norm.a (see its README).

Every artifact carries SASS for sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0), built against NVFP4EffMA 0.6.0.

Assets

• mlops-norm-0.1.0-linux-x86_64-cuda12.tar.xz (1,4M)
• mlops-norm-0.1.0-src.tar.gz (60K)
• mlops_norm-0.1.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (12M)

Changes

First release. RMSNorm and LayerNorm for sm_50 .. sm_90, with a residual add before and an INT8 or NVFP4 output after.

• Flavors. RMSNorm and LayerNorm; the weight, 1 + weight (Gemma, Qwen3-Next, LayerNorm1P) or none; a bias; Hugging
  Face's order of roundings (round(round(v r) w), Llama / Qwen) or a single rounding (Gemma, LayerNorm, biases,
  gates); silu gates before or after the norm (Mamba-2, Nemotron-H, Qwen3-Next); groups; x in BF16, FP16 or FP32 with
  weights in BF16 / FP16 / FP32. 105 kernels, none with spills on sm_61 .. sm_89.
• Quantized outputs. INT8 (amax / 127 per 16 values) and NVFP4 (E2M1 with E4M3 scales and a decode-factor
  global): the bytes nvfp4a_quantize of NVFP4EffMA makes of the same y, bit for bit, without a pass over y. The E4M3
  and E2M1 primitives are NVFP4EffMA's (its headers are a build dependency, nothing at run time); the E2M1 conversion
  is rewritten in fewer instructions and checked against NVFP4EffMA's for all 2^32 floats.
• Reproducible sums. The squares are summed in units of 256 elements in a fixed order (the same as mlops_embed and
  mlops_lm_head use), so the result does not depend on the variant, the element count or the GPU (the gates' exp
  aside). From sm_80 a BF16 pair is rounded by cvt.rn.bf16x2.f32, below it by integer instructions with the same
  results.
• Variants and items. mlops_norm_variants lists the instances (1, 2, 4, 8 warps) as data; mlops_norm_items.h
  describes a call as items for an executor, with their NVRTC entries and access ranges. A group is a multiple of 256
  values; per-head norms of 64 or 128 belong to mlops_fbla_qkv_at.
• Python mlops.norm: norm, rms_norm, layer_norm, RMSNorm, LayerNorm, norm_variants. order=None picks
  Hugging Face's order for an RMSNorm with a plain weight and no bias or gate, a single rounding otherwise.
• 2048 x 4096 BF16 RMSNorm: 160 us on a GTX 1080 (127 - 130 us with an INT8 or NVFP4 output), 20 us on an L40S (18 - 19 us).
• Tested: GTX 1080: the C tests (75 checks: bit for bit against a host emulation, the quantized bytes against
  nvfp4a_quantize, items through NVRTC), pytest (26), the samples, the wheel in a fresh venv, the C archive from plain
  C. sm_89 (48 GB): the C tests and pytest.

mlops 0.7.0 (meta package)

mlops 0.7.0

The meta package: installs every mlops package (mlops-optimizers~=0.2.0, mlops-positional~=0.1.0, mlops-attention~=0.4.0, mlops-vocab~=0.1.0, mlops-norm~=0.1.0). It has no code of its own.

Install

Install PyTorch and nvfp4effma first, put this wheel and the wheels of the packages (their releases) in one
directory, then:

pip install --no-index -f . mlops

From sources: ./setup.sh all.

Assets

• mlops-0.7.0-py3-none-any.whl (24K)

Changes

• Two new packages: mlops-vocab 0.1.0 (embedding rows, the BF16 LM head, token sampling) and mlops-norm 0.1.0 (RMSNorm
  and LayerNorm with INT8 / NVFP4 outputs). mlops-norm builds on NVFP4EffMA's headers (nvfp4effma 0.6.x); nothing is
  needed from it at run time.
• mlops-attention 0.4.0 (requirement ~=0.4.0): work items for an executor and mlops_fbla_prefill_at in the C API.
  mlops-optimizers and mlops-positional are unchanged.

mlops-attention 0.4.0

mlops-attention 0.4.0

Part of mlops: CUDA operations for NVIDIA GPUs sm_61 … sm_90.

Install

Nothing here downloads anything. Install first:

• PyTorch (CUDA build);

Then, in the directory with the downloaded wheel (cpXY = your Python, torchA.B / cuNNN = your PyTorch and its CUDA):

pip install --no-index -f . mlops-attention

--no-index takes mlops-attention from this directory and its dependencies from what is installed, never from PyPI.
Other combinations: ./setup.sh attention from the source archive.

C / C++: unpack mlops-attention-0.4.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libmlops_attention.a (see its README).

Every artifact carries SASS for sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

• mlops-attention-0.4.0-linux-x86_64-cuda12.tar.xz (1,8M)
• mlops-attention-0.4.0-src.tar.gz (188K)
• mlops_attention-0.4.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (13M)

Changes

Additions to the C API; the cache format, mlops_fbla_cache and the existing calls are unchanged, so programs built
against 0.3.x headers keep working. The Python package is unchanged.

• Work items (mlops_fbla_items.h, mlops_fbla_items.cuh). mlops_fbla_prefill_at_items, _append_at_items,
  _decode_at_items and _qkv_at_items describe a call as passes of CTA-sized items: the device entry of each,
  its argument blob, grid, threads, shared storage and registers, and the byte ranges it reads and writes
  (mlops_fbla_item_access). An executor that runs the entries from kernels of its own, compiled with NVRTC for the
  plan's compute_<arch>, computes what the call computes, bit for bit. make install puts the device-header tree
  under include/mlops/ so installed consumers need no source checkout. The C archive ships the host header only;
  the device entries need the source tree or make install.
• mlops_fbla_prefill_at. The prefill with the chunk's start and count in device memory (max_count is the
  ceiling that sizes the launches), as decode_at does for the length: a CUDA graph or a program queued ahead of the
  host replays it as the cache grows.
• Decode kernels. exact_block tiles the queries by two, so a group's V is read once per tile (same FMA order,
  same results); the refine item sets up its query pairs once. A 16-KV-head row is added to bench_fbla.
• Positional. mlops_rope.h and rope.cuh compile under NVRTC (they are included by the item entries); no
  change for host builds.
• Tests. test_items compiles every entry with NVRTC and runs it on its described CTA against the call, bit
  for bit, for both formats, GQA 1 / 2 / 4 / 5 / 8, strided caches, chunk ceilings, TC split / merge and multi-slice
  decode; it checks each range against its allocation and the stores of every item run alone against the declared
  ranges. --bad-count and --omit-count-range are negative controls and must fail.
• Tested: GTX 1080 (sm_61, CUDA 12.0): this release's static library and wheel, the C++ tests and test_items
  (152 checks). L40S (sm_89, CUDA 13.4): the C++ tests and test_items (266 checks).
• benches/1080.md is re-measured (its header still reads 0.3.0: it was run before the version was raised).

mlops 0.6.0 (meta package)

mlops 0.6.0

The meta package: installs every mlops package (mlops-optimizers~=0.2.0, mlops-positional~=0.1.0, mlops-attention~=0.3.0). It has no code of its own.

Install

Install PyTorch and nvfp4effma first, put this wheel and the wheels of the packages (their releases) in one
directory, then:

pip install --no-index -f . mlops

From sources: ./setup.sh all.

Assets

Changes

• mlops-attention 0.3.0 (requirement ~=0.3.0): the FBLA prefill on tensor cores (sm_75 .. sm_89), persistent
  images for chunked prefill, a context-independent decode grid, quality and speed measured on WikiText-2 and
  against flash-attn 2. See its release. mlops-optimizers and mlops-positional are unchanged.

mlops-attention 0.3.0

mlops-attention 0.3.0

Part of mlops: CUDA operations for NVIDIA GPUs sm_61 … sm_90.

Install

Nothing here downloads anything. Install first:

• PyTorch (CUDA build);

Then, in the directory with the downloaded wheel (cpXY = your Python, torchA.B / cuNNN = your PyTorch and its CUDA):

pip install --no-index -f . mlops-attention

--no-index takes mlops-attention from this directory and its dependencies from what is installed, never from PyPI.
Other combinations: ./setup.sh attention from the source archive.

C / C++: unpack mlops-attention-0.3.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libmlops_attention.a (see its README).

Every artifact carries SASS for sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

• mlops-attention-0.3.0-linux-x86_64-cuda12.tar.xz (1,2M)
• mlops-attention-0.3.0-src.tar.gz (152K)
• mlops_attention-0.3.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (6,7M)

Changes

The cache format and the decode / prefill API are unchanged. One ABI change: mlops_fbla_cache has a new trailing
field, tc_images (null: none), so programs built against 0.2.x headers must be recompiled. Python:
FBLACache.images, set by empty(); prefill_p128 in mlops.attention.fbla_reference.

• Prefill on tensor cores (sm_75 .. sm_89). The closed groups are attended through int8 tensor-core products of
  the group's q-hat and the 6-bit codes (one softmax block of 128 tokens a group; the sink and the row's own group
  in BF16 with BF16 P, blocks of 64), any q_heads / kv_heads ratio. sm_61 .. sm_72 and sm_90 stay on the DP4A
  kernel. Against flash-attn 2.8.3 (BF16) on an L40S the prefill call (close, transcode, attention, finish) is
  1.3 .. 1.9× faster, except an empty cache (0.85 .. 1.04×). The output differs from decode's (blocks of 16)
  within the BF16 rounding of the output; fbla/tests/test_fbla.cu and prefill_p128 model it in float64.
• Persistent images (mlops_fbla_cache.tc_images, mlops_fbla_images_bytes, empty() allocates them on
  tensor-core GPUs: 34 KB per group and KV head, only with a nonzero capacity). A chunked prefill transcodes each
  closed group once instead of every closed group at every call: 256K context, chunks of 2048: 1.37 -> 0.024 ms.
• Decode: the grid no longer depends on the context. The CTAs of the coarse and refine passes are the device's
  resident CTAs split over the heads, and the partials are combined by a separate kernel that spreads a row over
  SMs by channel. KV 1 / GQA 4 at 16K: 92 -> 31 us (flash-attn 2: 43 us); KV 8 is unchanged. The decode scratch
  needs no initial contents (the CTA counters are gone) and is smaller by one counter per KV head.
• Quality and benchmarks on WikiText-2. bench/fbla_lm.py, fbla_token_metrics.py and capture_qkv.py use the
  WikiText-2 test split by default (the tables measured on the repository's READMEs changed with them), and
  measure the prefill kernels. bench/fbla_vs_sdpa.py times decode and prefill against flash-attn 2 (else
  PyTorch's flash / memory-efficient SDPA). Qwen3-0.6B, 11 windows of 16K: FBLA 3+3 / 4+2 decode +0.0013 / +0.0014,
  tensor-core prefill -0.0013 against BF16 SDPA (loss 2.784; windows differ by ±0.005), llama.cpp q8_0 +0.0011,
  KIVI-4 +0.0010, q4_0 +0.20 (docs/overview.md).
• Wheel and distribution. The wheel (linux_x86_64, not manylinux) is built on Ubuntu 24.04 (GCC 13) and needs
  GLIBCXX_3.4.32; on older systems (Ubuntu 22.04 has 3.4.30) use ./setup.sh attention from the source archive.
• Builds with nvcc 12.0 .. 13.4: the sm_75 tensor-core path avoids an early-clobber asm output nvcc 12.0 rejects.
• Tested on a GTX 1080 (sm_61, CUDA 12.0; this release's static library and wheel): the C++ tests (64), pytest (16),
  samples 01 and 02. On an L40S (sm_89, CUDA 13.4): this release's static library passes the C++ tests (88, the
  tensor-core prefill with and without persistent images), compute-sanitizer memcheck, synccheck and racecheck report no errors, hazards or warnings;
  the sm_75 specialization was run as compute_75 PTX on the L40S (JIT; no sm_75 GPU was available), and the
  Python wheel only on the GTX 1080 (the DP4A prefill).

mlops 0.5.0 (meta package)

mlops 0.5.0

The meta package: installs every mlops package (mlops-optimizers~=0.2.0, mlops-positional~=0.1.0, mlops-attention~=0.2.0). It has no code of its own.

Install

Install PyTorch and nvfp4effma first, put this wheel and the wheels of the packages (their releases) in one
directory, then:

pip install --no-index -f . mlops

From sources: ./setup.sh all.

Assets

• mlops-0.5.0-py3-none-any.whl (20K)

Changes

• mlops-attention 0.2.0: FBLA now covers the whole generation, prompt and decode, on the same compressed cache.
  Prefill attends a chunk causally over it in one warp-specialized streaming kernel: producer warps unpack the
  records into swizzled shared-memory tiles, consumer warps run QK and PV with an online softmax. Also a complete
  text-generation sample (see its release).

mlops-attention 0.2.1-hotfix

mlops-attention 0.2.1-hotfix

Part of mlops: CUDA operations for NVIDIA GPUs sm_61 … sm_90.

Install

Nothing here downloads anything. Install first:

• PyTorch (CUDA build);

Then, in the directory with the downloaded wheel (cpXY = your Python, torchA.B / cuNNN = your PyTorch and its CUDA):

pip install --no-index -f . mlops-attention

--no-index takes mlops-attention from this directory and its dependencies from what is installed, never from PyPI.
Other combinations: ./setup.sh attention from the source archive.

C / C++: unpack mlops-attention-0.2.1-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libmlops_attention.a (see its README).

Every artifact carries SASS for sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

• mlops-attention-0.2.1-linux-x86_64-cuda12.tar.xz (1012K)
• mlops-attention-0.2.1-src.tar.gz (124K)
• mlops_attention-0.2.1+cu126torch2.7-cp312-cp312-linux_x86_64.whl (5.6M)

Changes

A hotfix of 0.2.0; the API and the cache format are unchanged.

• Prefill on sm_70+: in 0.2.0, prefill() / mlops_fbla_prefill hung on every GPU from Volta on. The kernel
  shuffled over the whole warp in branches that only one half of it takes (a row's running maximum, the output's
  row sums); Pascal runs a warp in lockstep, Volta and later schedule its halves independently. Decode was not
  affected.
• Builds with nvcc 12.8+: 0.2.0 launched the format-independent select kernels from the other format's
  object, and nvcc 12.8+ keeps a kernel template's launch stub local to its object
  (-static-global-template-stub), so the library did not link. They are now launched through host functions
  compiled with them.
• sm_89 from source with nvcc 12.1+: build/arch.mk dropped sm_89 from attention (and positional) with nvcc
  12.1 or newer. The prebuilt artifacts, made with CUDA 12.0, were not affected.
• Tested on an NVIDIA L40S (sm_89, CUDA 13.4) besides the GTX 1080: the C++ tests, pytest, and
  compute-sanitizer's memcheck, racecheck and synccheck; this release's static library passes the C++ tests on
  the L40S.

mlops-attention 0.2.0

mlops-attention 0.2.0

Part of mlops: CUDA operations for NVIDIA GPUs sm_61 … sm_90.

Install

Nothing here downloads anything. Install first:

• PyTorch (CUDA build);

Then, in the directory with the downloaded wheel (cpXY = your Python, torchA.B / cuNNN = your PyTorch and its CUDA):

pip install --no-index -f . mlops-attention

--no-index takes mlops-attention from this directory and its dependencies from what is installed, never from PyPI.
Other combinations: ./setup.sh attention from the source archive.

C / C++: unpack mlops-attention-0.2.0-linux-x86_64-cuda12.tar.xz (tar -xJf) and link lib/libmlops_attention.a (see its README).

Every artifact carries SASS for sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86, sm_87, sm_89, sm_90 (CUDA 12.0).

Assets

• mlops-attention-0.2.0-linux-x86_64-cuda12.tar.xz (1012K)
• mlops-attention-0.2.0-src.tar.gz (124K)
• mlops_attention-0.2.0+cu126torch2.7-cp312-cp312-linux_x86_64.whl (5.6M)

Changes

• Prefill: prefill(q, k, v, cache) / mlops_fbla_prefill append a chunk of tokens to the cache and, with
  queries, return their causal attention over the whole cache (position tokens + i reads tokens
  0 .. tokens + i). The groups the chunk closes are quantized on the GPU, bit for bit the encoder's records;
  one streaming kernel computes the attention. Prefill is compute-bound, so it reads every closed group in full
  (decode with every block refined, the same output in both modes): producer warps turn
  the coarse and fine records into swizzled tiles, consumer warps run QK and PV by DP4A with an online softmax.
  No scratch.
• empty(batch, kv_heads, capacity, mode) makes a cache to prefill into; encode and append run on the GPU
  through the same path.
• Speed (GTX 1080): the prefill attention runs at 8 equivalent FP32 TFLOPS (4+2, 64K context), 2× PyTorch's
  FP32 SDPA on Qwen3-0.6B's own q / k / v. Qwen3-8B-NVFP4 end to end against llama.cpp on the same weights
  (llama-bench's tests): prefill 1.9× and decode 1.5× faster at a 16K context (llama.cpp's KV in q8_0),
  prefill 2.1× faster at 32K, where llama.cpp fits only a q4_0 cache. Table: docs/llamacpp-compare.md.
• Samples: 02_streaming.py (one cache through prefill and decode), 03_transformers.py (FBLA inside a
  transformers model), 04_generate.py (a complete generator without transformers' model: Qwen3 / Qwen2 / Llama
  from safetensors, BF16 or ModelOpt NVFP4, NVFP4EffMA linear layers, mlops.positional RoPE, the layers that do
  not fit streamed from host memory). bench/llama_bench.py runs llama-bench's tests on the latter.
• C API: mlops_fbla_prefill; mlops_fbla_cache's pointers are no longer const (prefill writes the cache).
  The pkg-config file now links libm: with 0.1.0's, a C program failed to link (log2f). The select kernels
  are compiled once instead of in both format objects.

No dependency besides PyTorch (04_generate.py also uses nvfp4effma, mlops-positional, safetensors and the
model's tokenizer through transformers).

mlops 0.4.0 (meta package)

mlops 0.4.0

The meta package: installs every mlops package (mlops-optimizers~=0.2.0, mlops-positional~=0.1.0, mlops-attention~=0.1.0). It has no code of its own.

Install

Install PyTorch and nvfp4effma first, put this wheel and the wheels of the packages (their releases) in one
directory, then:

pip install --no-index -f . mlops

From sources: ./setup.sh all.

Assets

• mlops-0.4.0-py3-none-any.whl (20K)

Changes

• mlops-attention 0.1.0: FBLA compressed-KV FlashAttention decode for DP4A GPUs, in 3+3 and 4+2 modes
  (see its release).