v0.18.0

Highlights

NVFP4 training

NVFP4 for dense linears is now supported for training (prototype), implementing the recipe from Pretraining Large Language Models with NVFP4. This is a prototype feature and the API may change without notice. Hardening, MoE support, and torchtitan integration are planned.

Example usage:

import torch
from torchao.quantization import quantize_
from torchao.prototype.moe_training.nvfp4_training.nvfp4_training import (
    NVFP4TrainingConfig,
)

# Blackwell (SM100+); bfloat16 model with Linear dims divisible by 128
model = model.cuda().bfloat16()

# Replace nn.Linear with NVFP4Linear
quantize_(model, NVFP4TrainingConfig())

# Train as usual
out = model(x)
out.sum().backward()

• [nvfp4_training] Support NVFP4 linear training on Blackwell by @rdspring1 in #4369
• [nvfp4_training] Add Triton kernel for global amax of columnwise RHT (SM90+) by @rdspring1 in #4247

BC-Breaking / Deprecations

Minimum PyTorch version is now 2.11

Support for PyTorch older than 2.11 has been dropped from the code, and CI has moved to newer PyTorch versions.

• drop PyTorch 2.7 and older support from the code by @vkuzo in #4395
• drop PyTorch<2.8 support from the code by @vkuzo in #4398
• drop PyTorch<2.9 support by @vkuzo in #4399
• drop PyTorch <2.10 support by @vkuzo in #4400
• drop PyTorch <2.11 support by @vkuzo in #4401
• switch CI to PyTorch 2.12 by @vkuzo in #4402
• Pin CI to PyTorch 2.13 for release; drop unused torchaudio by @vkuzo in #4638

Removed: v1 tensor-subclass / layout system

The deprecated v1 AffineQuantizedTensor stack and layout system have been fully removed, completing the migration to v2 tensor subclasses.

• Delete AffineQuantizedTensor, AQTTensorImpl, and Layout by @andrewor14 in #4245
• Delete deprecated TensorCoreTiledLayout and related code by @jerryzh168 in #4153
• Delete deprecated PlainLayout, PlainAQTTensorImpl and related v1 code paths by @jerryzh168 in #4151
• Remove LinearActivationQuantizedTensor and all related code by @jerryzh168 in #4258
• Remove LinearActivationWeightObservedTensor and insert_observers_ by @jerryzh168 in #4261
• Remove WeightTensorWithLinearActivationScaleMetadata and related code by @jerryzh168 in #4262
• Remove WeightTensorWithLinearActivationQuantizationMetadata by @jerryzh168 in #4263
• Remove unified.py (Quantizer and TwoStepQuantizer ABCs) by @jerryzh168 in #4264
• Remove MSLK FP8 tensorwise support by @cthi in #4461
• Remove check_cpu_version and check_xpu_version helpers by @Xia-Weiwen in #4211

Renamed / moved APIs

• [CPU] Rename Int4WeightOnlyOpaqueTensorConfig to PrototypeInt4WeightOnlyConfig by @Xia-Weiwen in #4205
• Rename Sparse2x4CUTLASSFloat8Tensor to Float8Sparse2x4_2DData2DMetadataTensor (#4343) by @bbeckca in #4343
• Move NF4Tensor to quantization.quantize_.workflows by @andrewor14 in #4256

Behavior changes

• Use fixed scale for Float8 softmax quantization instead of observer by @jerryzh168 in #4260

Training / QAT

mxfp8

• refactor autograd func by @danielvegamyhre in #4176
• remove unused block_size arg by @danielvegamyhre in #4177
• register sharding rules for cutedsl 2d quant kernel by @danielvegamyhre in #4178
• add cutedsl kernel for 32x1 mxfp8 quantization on 2d tensors by @danielvegamyhre in #4239
• update triton_to_mxfp8_dim0 nan handling; fix offset int32 overflow issue by @danielvegamyhre in #4201
• on-device validation of group sizes in cutedsl quant kernels by @danielvegamyhre in #4253
• Fix race condition in mxfp8 CUDA kernels by @lw in #4278
• integrate 1x32 cutedsl quantization kernel by @danielvegamyhre in #4306
• Support 32x32 scaling for weights in MXFP8 weight quantization kernel by @alexsamardzic in #4254
• add MXFP8Linear by @danielvegamyhre in #4374
• Add "32x1 transposed" variant to MXFP8 3D quantization kernel by @alexsamardzic in #4383
• add support for bias parm in _to_mxfp8_then_scaled_grouped_mm by @danielvegamyhre in #4386
• don't use offs as key in cutedsl compile cache by @danielvegamyhre in #4442
• Add "32x32 transposed" variant to MXFP8 3D quantization kernel by @alexsamardzic in #4447
• Flydsl mxfp8 quantize by @zstreet87 in #4357
• use versioned TMA lookup with CUDA 13 by @lw in #4653
• make RCEIL NaN handling CSE-friendly by @lw in #4654

float8

• fix undefined values for tail elements in act quant kernels by @iamzainhuda in #4186
• add sharding rules for fp8 blockwise kernels by @iamzainhuda in #4220
• fix fp8 grouped mm compile issue by @rishisinhanj in #4233
• Optimize triton_fp8_per_group_colwise_scales for AMDGPU by @wenchenvincent in #4113
• fix blockwise FP8 scaled_mm scale layout in Float8BlockwiseLinear by @iamzainhuda in #4229
• Add fused FP8 colwise 3D scale-and-cast Triton kernel for MoE forward by @rishisinhanj in #4311
• support linear op overrides in Float8TrainingWeightWrapperTensor by @danielvegamyhre in #4325
• Handle saturated eager float8 casts by @iamzainhuda in #4364
• fix cc1 typo in Float8LinearConfig.__post_init__ operand-precision assertion by @GotFusion in #4115
• fix == typo in float8_transpose axiswise dim remapping by @GotFusion in #4038
• Add XPU support for blockwise FP8 by @Stonepia in #4096
• Add blockwise FP8 MoE functionality with emulated GEMM by @iamzainhuda in #4460
• Fix mx_mm backward crash on non-contiguous grad_output by @ultism in #4470
• Integrate DeepGEMM FP8 blockwise MoE backend by @iamzainhuda in #4473
• work around Triton fp8 store miscompile in compiled training by @vkuzo in #4652
• emit tl.where inline for the Triton fp8 store workaround by @vkuzo in #4657

nvfp4

• emulated nvfp4 grouped gemm by @roycho96 in #4240

QAT

• fix: use Int4WeightOnlyConfig.group_size in fake quant configs by @oriollinan in #4518
• Fix Int4WeightFakeQuantizer and Float8FakeQuantizer missing enabled attribute by @javierdejesusda in #4336

other

• Implement aten.cat for Int8QuantizedTrainingLinearWeight (#2619) by @lollinng in #4465
• Fix DTensor errors in PruneOptimizer by @lisjin in #4267
• DTensor proxmap fix + low-rank speedups by @lisjin in #4377
• Add QKGrouper and refactor AttentionHeadGrouper for vmap by @lisjin in #4378
• Add KElementGrouper by @lisjin in #4379
• Simplify iterative reweight test cases by @lisjin in #4380
• Add NMSparseConstraint and MinSparsityConstraint by @lisjin in #4381
• Low-rank from internal repo by @lisjin in #4586
• Add GlobalMinSparsityConstraint for cross-layer budget allocation by @lisjin in #4587
• Fix SVD class leakage into nn.Linear from #4587 by @lisjin in #4613
• Add iterative reweighting to PruneOptimizer by @lisjin in #4283

Inference

• enable inline asm for mxfp8+rceil+compile+cuda+sm100 by @vkuzo in #4282
• Add PerGroup granularity for Float8WeightOnly config by @jerryzh168 in #4174
• Add embedding op support for Float8Tensor with PerGroup quantization by @jerryzh168 in #4175
• Add better debug print for failed prepare by @JakeStevens in #3950
• Create recipe for flux2pro running on AMD (#4200) by @hualazimi0425 in #4200
• add hadamard option to low precision attention api by @howardzhang-cv in #4194
• Add RunningAbsMaxSmoothQuantObserver for memory-efficient calibration (#3946) by @jcaip in #3946
• support pinning for mx and nvfp4 tensors by @vkuzo in #4192
• Skip linear+bn fusion when input is higher than 2-D by @Lidang-Jiang in #4242
• Add alg_id argument to SemiSparseWeightConfig (#4238) by @RandySheriff in #4238
• Adding V-only Hadamard support to low precision attention API by @howardzhang-cv in #4249
• Prevent folding of mutable input to copy_ and put_ operators by @tom-arm in #4181
• fix fb-built pytorch cannot use fp16 2:4 sparsity (#4280) by @ZihaoLiu0927 in #4280
• Add Sparse2x4HIPSPARSELTFloat8Tensor (#4277) by @bbeckca in #4277
• Fix int8 dynamic activation quantization accuracy regression from v2 tensor migration by @htzo in #4326
• Add padding in linear dispatch of Float8Sparse2x4_1DData1DMetadataTensor (#4382) by @bbeckca in #4382
• read dim from kwargs in Float8Tensor aten.split handler by @GotFusion in #4429
• Move collect_producer_nodes to graph_utils.py by @tom-arm in #4294
• Use transpose_result=True for fp8_sparse_mm (#4433) by @gyllstromk in #4433
• Convert quant_lift_up to CustomGraphPass with uuid() by @frgossen in #4424
• Add RopeSDPAFusionPass(CustomGraphPass) with uuid() by @frgossen in #4423
• Add _grouped_mm dispatch handler to Float8Tensor by @xiaowangintel in #4390
• Use smallest_normal instead of eps to clamp quant scales by @GregoryComer in #4534
• Gate int8 channelwise neondot qmatmul behind runtime FEAT_DotProd check by @metascroy in #4571
• add warnings for float8 rowwise inference on CUDA 12.9 by @vkuzo in #4588
• Fix NF4Tensor narrow() crash with recent PyTorch nightly by @vkuzo in #4208
• Make per_tensor_scale optional for triton kernel path by @jerryzh168 in #4188
• make NVFP4Tensor handle per-expert outer scale by @vkuzo in #4315
• Add reduce_range to avoid overflow in int8 tensor by @cyxlily in #4266
• make nvfp4_linear handle a pre-quantized activation (#4504) by @larryliu0820 in #4504

Docs

• fix broken quantize_ link in first_quantization_example by @Anai-Guo in #4317
• update supported configs list, fix unclickable link, add blog ref by @Anai-Guo in #4305
• fix RST warnings in FqnToConfig and Int8DynamicActivationIntxWeightConfig docstrings (#3863) by @Anai-Guo in #4475
• replace TorchTune QAT section with Unsloth by @Anai-Guo in #4312
• fix Sphinx toctree doc build warning for performant_kernels.rst (#3863) by @brittneylilly in #4515
• annotate and document DuplicateDQPass (#4541) by @Anai-Guo in #4565
• Fix broken links in first quantization example docs (Issue #4286) by @SyedIshmumAhnaf in #4291
• correct image path by @itsPronay in #4268
• Calibration is a must for both static and dynamic quantization by @Xia-Weiwen in #4307

Devs

No-GIL support

• Enable building under a free-threaded Python interpreter by @rgommers in #4428
• Re-enable building under a free-threaded Python interpreter by @nascheme in #4537
• cpu: lock ukernel registration tables by @nascheme in #4561
• kernel: avoid partial Triton lazy initialization by @nascheme in #4553
• kernel: synchronize autotuner cache updates by @nascheme in #4554
• optim: cache qmap values as tuples by @nascheme in #4552
• snapshot FSDP precomputed scale by @nascheme in #4555

Other

• preserve mutable buffer inputs during prepare by @mansnils in #4466
• Revert "Revert "pt2e: preserve mutable buffer inputs during prepare"" by @tom-arm in #4564
• remove per kernel fp8 triton quantized benchmarks by @iamzainhuda in #4166
• Add weekly test for Intel GPU by @zxd1997066 in #4190
• add torch.compile test for Float8BlockwiseLinear by @iamzainhuda in #4187
• Enable FP8 QAT test on MI300/MI350 by @brucechanglongxu in #4132
• make alignment size configurable in benchmark script by @danielvegamyhre in #4246
• Patch AArch64 detection and enable Python 3.12 in Linux AArch64 nightly wheels build by @puneetmatharu in #4164
• URL should use meta-pytorch instead of pytorch by @nWEIdia in #4284
• test(moe): skip jagged rowwise scales test. by @jagadish-amd in #4304
• add FSDP and TP tests for Float8BlockwiseLinear by @iamzainhuda in #4295
• Add blockwise FP8 in roofline_utils and separate benchmark script by @iamzainhuda in #4344
• emulated mode test fix by @danielvegamyhre in #4396
• Fix build failure with GCC 14 in rowwise_scaled_linear_sparse_cutlass_f8f8.cu by @rgommers in #4426
• Fix Python 3.14 compatibility of quantization/pt2e by @rgommers in #4427
• fix(setup.py): exclude top-level test/ from find_packages by @Anai-Guo in #4361
• Add algorithm search to FP8 sparse linear benchmark (#4432) by @gyllstromk in #4432
• clean up tutorials by @vkuzo in #4449
• Add HIP device synchronize event to profiler overhead filter by @jagadish-amd in #4450
• Fix typo in e2e model level benchmarks - h100->b200 by @Pqlet in #4547
• device agnostic test base, compile, numerics_integration by @arlesniak in #3823
• Route ROCm regression CI to MI350 ecosystem runners by @amdfaa in #4621

New Contributors

• @JakeStevens made their first contribution in #3950
• @hualazimi0425 made their first contribution in #4200
• @rishisinhanj made their first contribution in #4233
• @Lidang-Jiang made their first contribution in #4242
• @roycho96 made their first contribution in #4240
• @tom-arm made their first contribution in #4181
• @ZihaoLiu0927 made their first contribution in #4280
• @nWEIdia made their first contribution in #4284
• @SyedIshmumAhnaf made their first contribution in #4291
• @itsPronay made their first contribution in #4268
• @jagadish-amd made their first contribution in #4304
• @CaoE made their first contribution in #4099
• @Anai-Guo made their first contribution in #4317
• @Dev-next-gen made their first contribution in #4297
• @rdspring1 made their first contribution in #4247
• @htzo made their first contribution in #4326
• @rgommers made their first contribution in #4426
• @GotFusion made their first contribution in #4429
• @yuchengliu1 made their first contribution in #3986
• @gyllstromk made their first contribution in #4432
• @frgossen made their first contribution in #4424
• @jbschlosser made their first contribution in #4462
• @lollinng made their first contribution in #4465
• @ultism made their first contribution in #4470
• @mansnils made their first contribution in #4466
• @waynehacking8 made their first contribution in #4480
• @GiGiKoneti made their first contribution in #4506
• @nascheme made their first contribution in #4555
• @oriollinan made their first contribution in #4518
• @Pqlet made their first contribution in #4547
• @brittneylilly made their first contribution in #4515
• @javierdejesusda made their first contribution in #4336
• @zstreet87 made their first contribution in #4357
• @georgehong made their first contribution in #4593

Full Changelog: v0.17.0...v0.18.0-rc1