AOTriton 0.15 Beta

Overview

AOTriton 0.15 is a maintenance release in the maintenance-feature cadence
introduced in 0.13. It keeps the 0.14 kernels, the AITER pin and, for most
architectures, the Triton compiler. It is ABI compatible with 0.14b: an
application built against 0.14b can switch to 0.15b as a drop-in replacement
(replace aotriton.images too).

The tuning database changes in this release are regression fixes: they
restore performance that 0.14b was expected to deliver but lost to tuner
defects. The compiler changes only for the gfx110x pack, whose tables were
rebuilt, and for the gfx1250 tech preview.

Headline changes

Performance Regression Fixes

The tuner used to time every cell of the tuning database at batch 3 x 5
heads. At short sequence lengths this leaves the GPU mostly idle, so launch
overhead decided the pick: small tiles and the single-launch fused backward,
which lose once batch x heads reaches the hundreds. On gfx1100, a second
tuner defect hid every large tile of the widened 0.14 attn_fwd search
space (#250).

The tuner now times each cell at batch x heads x seqlen_q ~ 2^17 and keeps
batch 3 x 5 heads for accuracy checks only (#270). The affected tables are
fixed as follows:

• gfx950: BERT-family and gpt2 training, slower in 0.13b/0.14.2b than in
  0.11.2b, are re-picked from measurements at the reported shapes (#267)
• gfx1100: the 0.14 tables are rolled back to 0.13b's and retuned on top of
  that base with the corrected tuner (#267, #270)
  • gfx1101-gfx1103 copy these cells; gfx115x and gfx117x inherit them where
    their own tables do not override

gfx950, BERT-family and gpt2 Workloads

Unforced operator, median ms (seqlen x batch). 0.11.2b is the last release
before the regression, timed through perfmon. 0.14 DB and 0.15b DB are
two AOT builds that differ only in the tuning database, timed with the same
script.

Workload	Pass	0.11.2b	0.14 DB	0.15b DB
roberta-large 384 x 512	fwd	1.86	5.79	1.89
roberta-large 384 x 512	bwd	5.41	10.44	6.16
bert-large-uncased 512 x 192	bwd	3.23	5.72	3.40
distilbert-base-uncased 512 x 384	bwd	4.78	8.46	5.00
gpt2 1024 x 72	fwd	0.74	0.865	0.731
gpt2 1024 x 72	bwd	2.44	3.70	2.45

The 0.15b DB column was measured on a GPU shared with other jobs. On an idle
GPU through perfmon, the fixed times are 3-8% lower: within 4% of 0.11.2b,
and faster than 0.11.2b for the roberta-large forward and gpt2.

gfx1100, Forward

Median ms, the same two-build comparison as above:

Shape	0.14 DB	0.15b DB
bf16 hdim 128, batch 3 x 5 heads, seqlen 2048	0.831	0.474
fp16 hdim 64, batch 3 x 5 heads, seqlen 2048	0.489	0.249
bf16 hdim 128, batch 1 x 16 heads, seqlen 8192	15.61	8.70
LTX (bf16, 32 heads, hdim 128), seqlen 6630	20.74	11.81
LTX (bf16, 32 heads, hdim 128), seqlen 15300	110.3	61.2

Diffusion workloads (LTX, Wan, SD 3.5, Qwen-Image, Hunyuan, Flux.2) lost
10-60% of wall time with 0.14b; this recovers it.

gfx1100, Backward

0.14b's backward was up to ~3x slower than 0.13b at short sequence lengths.
0.15b starts from 0.13b's tables (plus the #267 forward cells) and retunes
on top of them. Median ms, the 0.13b-based tables vs 0.15b:

dtype	hdim	seqlen	0.13b-based	0.15b
fp16	64	2048	5.15	5.16
fp16	64	4096	9.56	9.43
fp16	128	128	4.82	3.19
fp16	128	2048	18.47	17.69
fp16	128	4096	35.03	32.65
bf16	128	512	11.39	7.69
bf16	128	2048	20.19	17.09

Bottom-Right Aligned Sliding Windows (VarlenBits::principal_diagonal_choice)

AOTriton measures numeric sliding windows from the top-left diagonal, while
flash-attention measures them from the bottom-right one. Dense callers can
shift the window on the host, but under varlen the shift differs per
sequence, so bottom-right numeric windows were not expressible
(pytorch/pytorch#201930).

VarlenBits::principal_diagonal_choice (bit 18) selects the diagonal per
call (#268):

• VarlenPrincipalDiagonal::TOP_LEFT (0, default) keeps the previous
  behavior
• VarlenPrincipalDiagonal::BOTTOM_RIGHT (1) follows flash-attention: a
  numeric window becomes L = wl - d, R = wr + d with
  d = seqlen_k - seqlen_q taken per sequence
• Sentinel windows (causal masks) are never shifted

PyTorch uses it to fix pytorch/pytorch#203417 (numeric sliding windows
aligned top-left on ROCm). See docs/Generalized Sliding Window Attention.md
for the semantics and how to reproduce flash-attention's masks.

Human-Readable Tuning Database

The tuning database is no longer shipped as sqlite archives (5.5 MiB binary)
but as git-diffable text files (1.1 MiB), so tuning changes can be reviewed
as diffs and submitted incrementally (#257). The format is specified in
docs/Human Readable Tuning Database Format.md:

• amd/<arch>/<iface>.txtdb tensor files, one block per functional, with
  run-length encoding
• <iface>.<NNNN>-<name>.dbpatch files hold only the cells that differ, and
  apply in numeric order
• #! base references another architecture's table (not a symlink, which
  breaks on Windows)
  • Replaces AOTRITON_TUNING_DATABASE_REUSE

All 0.15b database fixes above ship as .dbpatch layers.

Major Changes from Release 0.14 Beta

API Changes

• VarlenBits::principal_diagonal_choice (#268)
  • attn_fwd_params::kVersion 4 -> 5, attn_bwd_params::kVersion 7 -> 8
  • 0.14b callers are translated with TOP_LEFT; a 0.14b params struct with
    bit 18 set (reserved there) is rejected with hipErrorInvalidValue
• GPU_AMD_ARCH_GFX1250_STRICT_MOD0 for runtimes that report
  gfx1250-strict (#264)

GPU Targets

• Experimental: gfx1170 and gfx1171 added (#241)
  • Triton pinned through the alternative-wheel config (#254)
  • Use gfx1150's and gfx1151's tuning, respectively (#257)
• gfx1152/gfx1153 now use gfx1151's tuning instead of gfx1100's (#257)
• gfx1150/gfx1151 use their own tuning, as .dbpatch layers over gfx1100
  (#257)
  • The previous reuse map silently preferred gfx1100's rows
• Tech preview: gfx1250 and gfx1250-strict, built with triton==3.9.0 from
  PyPI and a tuned database (#264)

Triton Compiler and Databases

• gfx1100-gfx1103 build with triton==3.9.0 (#270); other arches keep
  their 0.14b pins
• Tuning database fixes for gfx950 and gfx1100, see
  Performance Regression Fixes (#267, #270)

Kernel Bug Fixes

• bwd_kernel_fuse: fix NaN or zeroed gradients for sm_scale <= 0 (#245)
• gfx1201 flyc: round P to nearest instead of truncating to bf16 (#266)
  • Truncation biased O low; bf16 dQ at head_dim >= 128 was ~6x less
    accurate than on gfx1151. Max dQ error drops from 4.4-7.5e-3 to
    0.97-1.23e-3
• attn_fwd: the early-exit zero store is bounded by Hdim_vo, not
  BLOCK_DMODEL (#268)

Build Changes

• BREAKING --database_root / AOTRITON_TUNING_DATABASE_ROOT renamed
  --external_tuning_database / AOTRITON_EXTERNAL_TUNING_DATABASE (#257)
• ROCm installed through TheRock is found without setting ROCM_PATH or
  CMAKE_PREFIX_PATH, via rocm-sdk (#258)
  • Resolution order: ROCM_PATH -> rocm-sdk path --root -> /opt/rocm
  • Configuration stops with rocm-sdk's own error when it is on PATH but
    fails
• Fix the Win32 build by removing AOTRITON_API from util.h (#258)
• Fix builds without NDEBUG (e.g. no CMAKE_BUILD_TYPE), which failed on
  a missing <iostream> (#262)
• AOTRITON_TRITON_WHEEL_PEP508 takes any pip requirement; altwheel YAML
  entries accept {pep508: <req>} (#264)

pgDEF and Perfmon

• pgDEF: the distributed execution framework of Tuner v3.5 is extracted
  into aotriton.pgdef, an application-independent framework on PostgreSQL
  (#252). The tuner is ported onto it as its first application.
  • CAVEAT: no migration path. The tuning database schema changes
    completely; re-initialize the work directory
  • BREAKING .tune scripts require --category <app>[,<sub category>],
    and python -m aotriton.tune.dispatch requires --pool (or --arch)
  • Claimed tasks have a lease, so restarting a worker no longer leaves its
    tasks stuck in running
  • Idle connections get TCP keepalives, and CPU handlers retry once after a
    lost connection, which used to lose result rows (#270)
  • See docs/Distributed Execution Framework.md for the design
• Perfmon: the second pgDEF application, which measures the public flash
  operators of released builds end to end (#261)
  • Each entry is timed through the unforced operator (what users get) and
    through the operator forced to each backend. A forced backend that beats
    the operator's own choice is a fault in the tuning database
  • Inputs are sized to fill the GPU: 704 synthetic entries plus 10 shapes
    from regression reports (roberta-large, bert-large-uncased,
    distilbert-base-uncased, gpt2)
  • Results are committed to modules/flash/perfdb/ as a human-readable
    performance database, with data for 0.12.1b, 0.13b and 0.14.2b on
    gfx942, gfx90a, gfx1201 and gfx1100
  • python -m aotriton.perfmon.perfdb compare <tag> <tag> compares
    releases; perfmon reproduced the gfx950 BERT-family and gpt2 regressions
    fixed in this release
• .tune/ is split into .def/ (shared), .tune/ and .perfmon/; the WebUI
  gains a perfmon page (#261)

Minor Changes from Release 0.14 Beta

• python -m aotriton.unaks2 lists and extracts kernel images from .aks2
  archives (#256)
• docs/attention-kernel-numerical-error-lessons/: 16 numerical mistakes of
  the Triton attention kernels, with fixing commits, guarding tests and
  mutants (#249, #266)
  • New tests test_common_mistakes and test_null_logsumexp (#249)
• The tuner runs natively on hosts without docker (#270)
• modules/flash/tests/adiffs/<arch>.txt refreshed from the 0.15b level-3
  runs; gfx942 added (#270)

Known Problems

• gfx950: fp32 forward with head dimension 8-64 and seqlen_q of about 2048 or
  more is not run-to-run deterministic in the BLOCK_M=256 kernels. Repeated
  calls with identical inputs can differ by up to ~1e-2 absolute, so
  gradients also vary between runs, even in deterministic mode. The fp32
  entries in modules/flash/tests/adiffs/gfx950.txt record this problem, not
  an accuracy limit. To be addressed in 0.15.1b.
• gfx942: the AITER backward fails the two test_common_mistakes numerical
  checks (split and fused pass).
• gfx90a: the hardware flushes fp16 subnormals; 4 window-(0,0) varlen tests
  fail.
• fp32 8192x8192 unit tests run out of memory on 32-48 GB GPUs.
• gfx1100: some retuned cells were tuned on a 70-CU part, the rest on a
  96-CU one.
• gfx115x stays on Triton 3.8 and inherits the retuned gfx1100 cells; this
  combination is untested on gfx115x hardware.
• The tuning choices of gfx1152, gfx1153, gfx1170 and gfx1171 are not
  validated, due to hardware availability.
• gfx1250 and gfx1250-strict remain tech previews.

What's Changed (Auto-Generated)

• Fix SDPA fused backward for non-positive sm_scale. by @anhminhnguyenhoang in #245
• Numerical Error Lessons from the Triton Attention Kernels, and a Test that Catches Them by @xinyazhang in #249
• Widen the gfx1100 attn_fwd tuning space to PADDED_HEAD=True too by @sstamenk in #250
• PostgreSQL based Generic Distributed Execution Framework (pgDEF) by @xinyazhang in #252
• Add gfx117X target support by @mstankov-amd in #241
• Pin gfx117x Triton compiler with altwheel config by @arsic3 in #254
• Add a utility tool aotriton.unaks2 by @xinyazhang in #256
• Auto-detect ROCM_PATH and CMAKE_PREFIX_PATH via rocm-sdk by @xinyazhang in #258
• Human-Readable Tuning Database by @xinyazhang in #257
• Include in generated op/shim/flyc sources by @k-artem in #262
• Enable gfx1250/gfx1250-strict with tuned database. by @xinyazhang in #264
• In gfx1201's FlyDSL kernel, replace bf16_trunc_pack_v8 with bf16_rne_pack_v8 by @xinyazhang in #266
• Fix tuning db for gfx950 and gfx1100 by @xinyazhang in #267
• Add VarlenBits::principal_diagonal_choice to anchor sliding windows at the bottom-right diagonal by @xinyazhang in #268
• Perfmon: Cross-Version Performance Monitor and Human-Readable Performance Database by @xinyazhang in #261
• End of 0.15b: Tuning at GPU-Filling Sizes and the gfx1100 Tuning Database Rebuild by @xinyazhang in #270

New Contributors

• @anhminhnguyenhoang made their first contribution in #245
• @mstankov-amd made their first contribution in #241
• @arsic3 made their first contribution in #254
• @k-artem made their first contribution in #262

Full Changelog: 0.14.2b...0.15b