perf(LTX25-DECODE-THREADS): the video VAE decode had 20 cores and used one (#1009)

[localai-bot]

Lever 3 of the LTX25-DECODE-SPEED investigation (#1006, PR #1038 — #1018 was
the earlier pull request and is now closed). Closes #1009.

ParallelForRows (src/vt/cpu/cpu_threadpool.cpp:413) is synchronous and 10+ CPU
kernels in this tree dispatch through it. Zero of them were in the LTX-2.5 conv
video VAE decode, whose 42 convolutions carry ~7.25 TFLOP at 448x256/25f and ran
on one core of twenty. Three sites now dispatch: CausalConv3d's output nest, its
padding gather, and Linear3d.

The axis is the whole risk, so it is argued at the site

The sibling dtype row (#1008, d1b0ea3a8) had to change this convolution's
summation order to a blocked one to stay inside a 5e-06 tolerance. Parallelism
is the second thing that can change a summation order.

The partition is the output line (oc, ti, hi), out.w contiguous elements.
Volume::At(oc, ti, hi, wi) is ((oc*t + ti)*h + hi)*w + wi, so row r is exactly
[r*out.w, (r+1)*out.w) of out.data — no element is written twice — and the
entire ci * kernel^3 reduction stays inside one output element's body in the
blocked order #1008 shipped. A worker therefore executes the serial arm's
instruction sequence, in the serial arm's order, on the serial arm's values, for
every element it owns. The result cannot depend on the worker count or on which
worker stole which chunk, and the second half matters: ParallelForRows steals
through an atomic cursor, so the row-to-thread assignment is genuinely
non-deterministic run to run.

Splitting the reduction axis ic into per-thread partials would also be a legal
convolution. It is rejected in the comment at the site, because it would make the
summation order a function of the thread count.

The numerics did not move at all

Both suites were rebuilt with kLtx2GoldenTol set to 0.0 before the change and
again after, so every golden reports its max|diff| rather than its verdict.

All 34 recorded margins — 23 in test_ltx2_vae, 11 in test_ltx2_tiling —
came back byte-for-byte identical, compared by diffing the two sorted lists
rather than by eye: VAE_MARGINS_IDENTICAL (23 values),
TILING_MARGINS_IDENTICAL (11 values).

golden arm	before (serial)	after (20-thread global pool)	tol
Conv video decoder	1.72853e-06	1.72853e-06	5e-06
non-causal Conv video decoder	2.08616e-06	2.08616e-06	5e-06
norm_eps-binding video decoder	1.54972e-06	1.54972e-06	5e-06
tiled decode, untiled control A	2.74181e-06	2.74181e-06	5e-06
tiled decode, untiled control B	2.80142e-06	2.80142e-06	5e-06
every other arm in both suites	unchanged	unchanged	—

Those before-values are also the ones ltx25-decode-dtype.md §8.1 recorded on its
own host, which is an independent check that this box reproduces the sibling row.
No tolerance was touched.

That table is itself a threading gate: the suite runs on the global pool,
hardware_concurrency wide, so every LTX-2.5 video golden after this change
executes on 20 workers, and the "Conv video decoder" fixture carries a res_x_y
block so Linear3d and conv_shortcut are on that path too.

Two cases, because one of them measures nothing on its own

A thread-count A/B is green on a serial implementation. Shipping only that would
have been a test that passes while measuring nothing.

• "the decode DISPATCHES its convolutions to the CPU threadpool" reads the
  pool's public work-stealing cursor through ChunkAdd(0), which is a
  non-mutating read. A fresh pool reads 0; a pool that has run a partitioned
  dispatch reads at least nth. Asserting 0 before the decode is the
  instrument's own positive control. Before this change it fails
  CHECK( 0 > 0 ).
• "the decode is BIT-IDENTICAL across thread counts" decodes the same latent
  at 1, 2, 3, 5 and 8 workers and memcmps every arm against the 1-worker one,
  which short-circuits to the pre-change serial path. 3 and 5 are there because
  nchunk derives from nth * 4, not from nth — 45/30/18/12 are the chunk
  strides at this fixture's 360 conv_in output lines, four different partitions
  of the same output. It is NOT that 3 and 5 fail to divide the row counts: 360
  and 15 are both divisible by each. That was the comment's original claim and it
  was false; see the repairs below.

Both enter through Ltx2VideoDecodeStreaming — what the render path calls at
src/vllm/multimodal/ltx2_video.cpp:3258 — and both assert an analytically
derived value of exactly 7, not a recorded one, because #1008 recorded that a
zero-filled stub satisfies an expectation of zero.

The CPU A/B

Same binary, VLLM_CPP_CPU_THREADS the only variable, one decode through
Ltx2VideoDecodeStreaming, 14 runs per count across an ascending and a descending
sweep so an ordering drift shows as spread rather than hiding in a mean.

threads	runs	min s	median s	max s	spread	speedup	efficiency
1	14	1.9859	2.0418	2.1172	6.4%	1.00x	100%
2	14	1.0266	1.0552	1.0843	5.5%	1.93x	96.7%
4	14	0.5464	0.5555	0.5674	3.8%	3.68x	91.9%
8	14	0.2920	0.3013	0.3072	5.0%	6.78x	84.7%
16	14	0.2129	0.2232	0.2597	21.0%	9.15x	57.2%
20	14	0.2024	0.2234	0.2534	22.8%	9.14x	45.7%

A second shape at the checkpoint's real base_channels of 128: 5.1015 s at one
thread against 0.5276 s at twenty, 9.67x. That is the weakest number here —
n = 3 against the table's 14, no min/median/max, same contended box — so it
corroborates the table's shape at a second channel width and is not independently
a three-significant-figure result. The public records carry ~9x at 16-20 workers with the conditions rather than either decimal.

The load it was taken at. One-minute load average 4.03 to 6.77 on a box whose
one-minute average had been between 2 and 94 the same day, with one non-agent
process holding ~1.07 cores throughout. That process is part of why 16 and 20
spread 21-23% where everything at or below 8 spreads under 7%.

No ceiling is declared. The implied serial fraction at 9.14x on 20 workers is
6.3%, which is the right order for PixelNorm, Silu, ApplyAdaLn, the residual
add and expand — every one still serial, every one listed under ## Owed.
Memory bandwidth is the second candidate and is not separated here.

Determinism, proven a second time: the output checksum was bit-identical across
all 84 A/B decodes — six worker counts, two sweep directions, two shapes — on
pseudo-random weights rather than the engineered fixture.

What is NOT claimed

No end-to-end render speedup, no ratio against any oracle, no composition figure
with #1008. There is no GPU here, dgx.casa was unreachable throughout, and
ltx_core is not installed. The harness shape is synthetic and says so; what
generalises from it is the scaling, not the absolute wall.

ThreadSanitizer, with the instrument controlled first

RelWithDebInfo + VLLM_CPP_SANITIZE=thread: test_ltx2_vae 42/42,
test_ltx2_tiling 10/10, test_ltx2_video 57/57, all EXIT=0, zero
WARNING: ThreadSanitizer.

Two instrument problems had to be settled before that meant anything. The binaries
would not start at all — FATAL: ThreadSanitizer: unexpected memory mapping,
EXIT=66, an ASLR-against-shadow-layout failure that a && chain would have read
as a race; setarch x86_64 -R fixes it. And a sanitizer that reports nothing is
indistinguishable from one that is not instrumenting, so a deliberate unsynchronised
write was compiled into CausalConv3d's parallel body in the same lane: 87
WARNING: ThreadSanitizer: data race, EXIT=66, then reverted, rebuilt, and back
to 0 and EXIT=0.

Mutations, three facts each

mutation	numstat	built	exit	detected by
T0 — all three dispatches reverted	13/8	yes, 0 errors	1	the dispatch case, CHECK( 0 > 0 )
T1 — CausalConv3d's output loop alone	3/2	yes, 0 errors	0	nothing. 42/42 and 10/10 pass
T2 — the padding gather alone	5/3	yes, 0 errors	0	nothing. 42/42 and 10/10 pass
T3 — Linear3d alone	5/3	yes, 0 errors	0	nothing. 42/42 and 10/10 pass
D1 — chunk-dependent value, visible at 1 worker	1/0	yes, 0 errors	1	10 cases + 2 tiling cases
D2 — the same defect INVISIBLE at 1 worker	1/0	yes, 0 errors	1	the bit-identity memcmp, on all four non-base arms
R — production call site deleted	17/2	yes, 0 errors	1	the dispatch case on the cursor AND the value; the identity case's non-degeneracy REQUIRE
T1, first attempt	4/2	NO, 45 errors	—	nothing — a mutation that does not build establishes nothing

T1's failed first attempt is in the table on purpose. One unbalanced brace
closed the anonymous namespace early and produced 45 -Werror errors that read as
unrelated unused-function complaints hundreds of lines away. The runner refused
to draw a verdict rather than running a stale binary and printing a plausible 42/42.

T1, T2 and T3 are an honest gap and it is owed. One work-stealing cursor is
shared, so reverting any single site leaves the other two dispatching and the case
reads non-zero. It gates "at least one of the three sites dispatches", and T0 is
what holds the conjunction. T3 additionally cannot be seen by that fixture at all,
since Linear3d is only reached through a res_x_y block. What does bound each
site is the golden table above — the "Conv video decoder" arm reaches all three at
20 workers and did not move — and the wall-clock, which is what a serial
convolution would actually cost.

D1 is beside D2 because it is the weaker of the two. D1 perturbs the first row
of every chunk including the first, so the 1-worker arm moves too and the case
fails on its value assertion before reaching the memcmp. D2 perturbs only chunks
that do not start at row 0, which is invisible at one worker, so the memcmp
across worker counts is the only thing that can report it. It does, on all four.

Gate

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF, -j6, ctest -j4.
Run twice, the second at this branch head so a green gate chains to the push.

CONFIGURE_EXIT=0, BUILD_EXIT=0, : error: count 0, ctest -N 492,
CTEST_EXIT=0, 100% tests passed, 0 tests failed out of 492 in both runs
(308.99 s and 316.87 s). Two pre-existing skips,
test_modelopt_mixed_precision_checkpoint and test_voxtral_e2e.

No space left 0 and BFD internal-error/assertion 0 across every log,
both greps positive-controlled against a synthetic file carrying the real message
forms — 1 and 2 hits there, 0 in the real logs. check-doc-checkpoint --commit
green on each of the three commits and armed (b5618b305 exits 1).

Load average 32-52 on the first run and 82-94 on the head run, on a shared
20-core box; none of the load-dependent suites flaked in either. Free disk
21-30 GiB of 447 GB; the 834 MiB sanitizer tree was removed after use.

One deliberate omission

No .agents/issue-index.md row is appended for #1009. That row already exists
at .agents/issue-index.md:279 on PR #1038, branch row/LTX25-DECODE-SPEED-R2,
which filed the issue and is unmerged. #1018 was the pull request that carried it
first; #1018 is closed and #1038 supersedes it. .gitattributes sets merge=union on that file and
scripts/check-agent-record.py refuses a duplicate issue number, so a second copy
here would turn main red for every branch the moment #1038 merges — which is what
a duplicate #995 row did on 2026-08-16. The sibling dtype row made the same call
for #1008. The link lives in the spec and in this body; the index link arrives with
#1038. One index row IS appended by this branch, for the new issue #1044 below,
and #1044 is not among the ids #1038 appends.

The review, and the five findings repaired after it

A fresh reviewer returned PASS with no blocking findings: the gate reran at
492/492, reduction safety was verified from Volume::At's index arithmetic,
determinism was proven by a mutation caught at all four worker counts,
ThreadSanitizer was clean against an 84-race positive control, the keyed records
were proven key by key, all nine anchors landed exactly, and a correctly-shaped
zero-filled buffer fails both new cases, so the zero-stub trap is closed.

Five non-blocking findings followed. Each was re-verified before it was
repaired, because a finding is a hypothesis; none of the five was rejected.

#	Finding	Verified how	Repair
F1	The T1/T2/T3 gap was argued in §8.6 prose with no issue and no ## Owed entry	§7's table has no per-site row; ParallelForRows seeds ONE pool cursor (cpu_threadpool.cpp:438, advanced :455), so two surviving sites keep it non-zero	Filed #1044, added it to §7 ## Owed and to the issue index, owned by LTX25-DECODE-THREADS
F2	FEATURES.md and USAGE.md carried 9.14x/9.67x bare, against a 21-23% spread	Cell widths measured with check-public-doc-tables.py's own parser	~9x at 16-20 workers with the conditions; the bare 9.67x at c=128 is gone from FEATURES.md
F3	The #1009 index row was cited at :275 on PR #1018	#1018 is CLOSED; the row is at line 279 of the index on row/LTX25-DECODE-SPEED-R2 (PR #1038)	Citation corrected here and in the spec, and the two other spec pointers at the closed pull request with it. The DECISION is unchanged
F4	§8 cited evidence at dac85969c, which does not resolve on the branch	git merge-base --is-ancestor dac85969c HEAD exits 1	§8 cites d653f7319 and states why the measurement transfers
F5	The determinism case's stated reason was arithmetically false	360 and 15 are both divisible by 3 and by 5; the strides are 45/30/18/12	The comment now states the real mechanism and says outright not to "fix" the row counts

F1 is owed, not implemented. #1044 carries the closing test the reviewer
supplied — a per-dispatch Threadpool::RunCount() bumped in Run() and an
EXACT expected count rather than > 0, plus a fixture carrying a res_x_y
block because Linear3d is unreachable with decoder_blocks empty. A new gate
needs its own red-before evidence and its own fresh review, so it is a row.

The index row names its owner rather than leaning on ## Owed, deliberately.
owed_issues() in scripts/check-agent-record.py splits on a bare \n## Owed
and this spec's heading is ## 7. Owed, so nothing listed there is visible to
the unowned ratchet. Measured: the unowned count is 33 before and after,
against UNOWNED_HIGH_WATER = 33. Four other specs have the same numbered
heading (ltx25-decode-dtype, ltx25-token-append, nemotron-h-a2q1-fp8-mamba,
nemotron-h-a2q2-nvfp4-moe-lmhead); that is a record observation this row does
not repair.

Cell widths, since MAX_CELL_CHARS = 220 binds and the BENCHMARKS.md cell
sat at exactly 220. BENCHMARKS.md LTX-2.5 axes 220 to 212;
FEATURES.md decode-threading 210 to 204. Both measured with the checker's
own _table_rows, and check-public-doc-tables.py exits 0.

Keyed records, proven key by key against the merge base. BENCHMARKS.md:
179 unrelated keys byte-identical, only LTX-2.5 axes changed, none added or
removed. FEATURES.md: 194 unrelated keys byte-identical. USAGE.md: 200
unrelated keys byte-identical, no table row touched. .agents/issue-index.md:
origin/main's version is a byte-identical prefix, exactly one appended
line, and it is #1044.

Nothing was re-measured. dgx.casa is down and the A/B harness is
deliberately not in the tree, so every wall-clock figure above stands as the
implementer recorded it. No end-to-end render speedup is claimed here either.

Gate after the repairs. CONFIGURE_EXIT=0, BUILD_EXIT=0, : error: count
0 on a full 1449-target build, ctest -N 492, CTEST_EXIT=0, 100%
tests passed, 0 tests failed out of 492 in 165.47 s, the same two pre-existing
skips. No space left 0 and BFD/internal-error 0 across both logs,
each grep positive-controlled against a synthetic file carrying the real message
forms (2, 1 and 2 hits there, 0 in the real logs). check-doc-checkpoint --commit green on all four branch commits and armed: b5618b305 exits 1.
One-minute load 10 to 26; free disk 40 GiB falling to 21 GiB.

Pushed with --no-verify, and why

The pre-push hook refuses this branch on check-public-doc-tables.py:

docs/BENCHMARKS.md has 36 prose paragraphs, over the 35 budget
docs/FEATURES.md has 22 prose paragraphs, over the 21 budget

This branch did not cause it. Matched-arm check: origin/main alone, in a
detached worktree with no branch content, fails with the identical numbers.
Bisected to e34d71379 (#1054, an AppleClang capture fix that also added +5
lines to BENCHMARKS and +4 to FEATURES); 283c7e492 immediately before it exits
0. Filed as #1055.

AGENTS.md: "Hooks are bypassable convenience, not evidence", and "a commit that
needs an exception argues for it in its own message". This is that argument. The
same checker run against this branch's own edits is clean -- it changes exactly
one key in BENCHMARKS (LTX-2.5 axes) and adds exactly one in FEATURES
(LTX-2.5 Conv VAE decode threading), with every unrelated key proven
byte-identical to origin/main.

Gate at the merged tree

CONFIGURE_EXIT=0, BUILD_EXIT=0, : error: 0, 493 targets linked, ctest -N
495, 494 of 495 passed. The one failure is test_serve_low_tools, which is
#428 ("the concurrency-cap assertion races the server-side counter and reads
3 under load") -- it passes 3/3 when re-run alone, and this branch touches no
serve or tools file. Attribution verified against the issue that names the test,
not assumed from a family.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]