MiniMax H3: SageAttention FP8 PV kernels produce noise above ~160k tokens on sm_120

[AI4VFX]

Edit 2026-08-09. Read this, the report below is old.

This works now on ComfyUI 0.31.1. 1920x1088, 15 s, sage on via the KJNodes node set to auto, 54 min against 94 with sage off. Same sage build as when it broke. Two things changed though, ComfyUI and my text encoder, so I'm not calling it fixed yet. More here.

If you're on 0.30.1 or older and getting noise on sm_120, update first.

Ignore the workaround I put here on 08-08. I said to use the KJNodes node on sageattn_qk_int8_pv_fp16_cuda. It doesn't work on sm_120, it aborts on the first attention call, and fp16_triton throws cudaErrorIllegalAddress. auto is the only mode that runs on that arch, and it isn't any of the entries under it in the dropdown. Table. Sorry if anyone tried it.

What it looked like on 0.30.1, counting tokens as ceil(frames/4) * (W/32) * (H/32), all on an RTX PRO 6000 with the bf16 models:

res	duration	frames	tokens	result
1920x832	15 s	362	142k	clean
1920x1088	12 s	294	151k	clean
1216x704 (2 pass)	30 s	736	154k	clean
1920x1088	13 s	328	167k	noise
1920x832	18 s	447	175k	noise
1920x1088	15 s	362	186k	noise

Clean under about 154k, noise from about 167k, nothing in between. The same 186k render with sage off was always fine, so the wall was never in H3.

Don't merge the one line fix below. It turns sage off for H3 everywhere, including Ampere where it works fine. @pepikir's breakdown is still the best account of what's going on.

The rest is the original report. The smooth_k theory in it is dead. The audio VAE bit still holds.

---

Summary

Running MiniMax H3 with --use-sage-attention produces pure noise in both the video and audio streams. Not degraded output — noise. The H3 DiT's attention call does not pass low_precision_attention=False, so it gets routed to attention_sage like any other model, and SageAttention's int8 QK path breaks it completely.

Every comparable model in the tree already opts out. H3 (added in #15224, merged as e2ab36d) appears to be the omission.

Expected vs actual

• Expected: either correct output, or (like LTX) an automatic fallback to PyTorch attention.
• Actual: 55 minutes of sampling → noise, no warning, no error.

Root cause

⚠️ Superseded, see the edit note at the top. The opt-out is not missing in any meaningful sense; the kernel selected on sm_120 is the actual fault.

comfy/ldm/modules/attention.py:550 provides the opt-out:

def attention_sage(q, k, v, heads, mask=None, attn_precision=None, skip_reshape=False, skip_output_reshape=False, **kwargs):
    if kwargs.get("low_precision_attention", True) is False or (mask is not None and not SAGE_ATTENTION_SUPPORTS_MASK):
        return attention_pytorch(q, k, v, heads, mask=mask, skip_reshape=skip_reshape, skip_output_reshape=skip_output_reshape, **kwargs)

Models that use it today:

File	Line
comfy/ldm/lightricks/model.py	405 — comment: "sageattn mask support is unreliable"
comfy/ldm/audio/dit.py	433, 434, 437
comfy/ldm/audio/vae_sa3.py	141, 142, 145
comfy/ldm/ace/ace_step15.py	244
comfy/ldm/hunyuan3dv2_1/hunyuandit.py	346, 416
comfy/ldm/sam3/{sam,detector,tracker}.py	several
comfy/image_encoders/dino3.py	97
comfy/ldm/triposplat/model.py	106

MiniMax H3 does not — comfy/ldm/minimax/model.py:181:

out = optimized_attention(q, k, v, self.heads, mask=None, skip_reshape=True, transformer_options=transformer_options)

Proposed fix (one line)

⚠️ Retracted, don't merge this. It would disable sage for H3 on every GPU and kernel, including the ones where it works. See the edit note at the top.

out = optimized_attention(q, k, v, self.heads, mask=None, skip_reshape=True,
                          low_precision_attention=False, transformer_options=transformer_options)

All attention backends (attention_basic, attention_sub_quad, attention_split, attention_xformers, attention_pytorch, attention_sage, attention_flash) declare **kwargs, so this is inert for every non-sage path — verified locally, no behavioural change with sage disabled.

I've applied this locally and it resolves the issue. Happy to open a PR if useful.

Evidence that the fault is in the DiT, not the VAEs

Worth recording, since it rules out a decode-side explanation:

Decoder	Attention call	Sage-swapped?
Video VAE comfy/ldm/minimax/vae.py:239	optimized_attention(...)	yes
Audio VAE comfy/ldm/minimax/audio_vae.py:247	comfy.ops.scaled_dot_product_attention(...)	no

The audio VAE never imports optimized_attention, so --use-sage-attention cannot have affected it — yet the audio output was still noise. The audio latent was therefore already corrupt leaving the sampler, placing the fault upstream of both decoders in the DiT's attention.

(This part still holds.)

Possible contributing factor

⚠️ Retracted. @pepikir probed this directly and smooth_k=True vs False gave identical relative error, so it isn't the trigger.

comfy/ldm/modules/attention.py:573 hardcodes smooth_k=False:

sage_kwargs = {"is_causal": False, "tensor_layout": tensor_layout, "sm_scale": kwargs.get("scale", None), "smooth_k": False}

SageAttention's own default is smooth_k=True, and channel-wise mean subtraction of K before int8 quantization is what its paper credits for keeping int8 QK accurate. H3's attention (model.py:156-181) applies QK-RMSNorm plus a partial rope (rot = rope_freqs.shape[-3] * 2, 96 of 128 dims rotated), which leaves rotated and unrotated channels with markedly different statistics — the kind of channel-wise outlier structure smooth_k exists to handle. Speculative and not instrumented; the opt-out above makes it moot for H3, but it may explain why other models needed the same escape hatch.

Reproduction

• ComfyUI v0.30.1, torch 2.11.0+cu130, Python 3.13, Windows 11
• RTX PRO 6000 Blackwell (96 GB), sageattention installed
• minimax_h3_fl2va_bf16.safetensors + qwen3vl_32b_minimax_h3_bf16.safetensors
• MiniMaxH3ImageToVideo → BasicGuider → SamplerCustomAdvanced (res_multistep, 20 steps), 1920x1088, 362 frames
• Launch with --fast-disk --use-sage-attention → noise.
• Identical graph and seed without --use-sage-attention → correct output.

Same seed both runs, so the two are directly comparable.

[coderabbitai]

🔗 Related PRs

#15270 - fix(minimax): expose H3 attention patch hooks [open]
#15446 - Optimize MiniMax-H3 VAE [open]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[pepikir]

Data point: SageAttention works fine on MiniMax H3 via the KJNodes Patch Sage Attention KJ node — no noise, and a measurable speedup. It looks like only the --use-sage-attention core path is affected, so low_precision_attention=False may be too broad a fix.

Setup: RTX 3090 (sm_86), ComfyUI 0.30.1, torch 2.11.0+cu130, sageattention 1.0.6 (PyPI, pure-Triton wheel), minimax_h3_fl2va_pruned_int8_convrot.safetensors, 864×480, 124 frames, 20 steps, res_multistep/simple, BasicGuider, fixed seed.

Result:

	sampling	20 steps
no sage	9.00 s/it	~180 s
Patch Sage Attention KJ (auto)	6.57 s/it	~131 s

1.37× on sampling. Output is clean — a normal, coherent video with the audio track intact, visually equivalent to the non-sage run. PSNR against the same-seed non-sage render is 32.0 dB average (y 30.3), i.e. a slightly different but equally valid sample, not degradation. Nothing like the pure noise reported here. Zero using pytorch attention instead fallbacks in the log, so sage was genuinely active on the DiT attention.

Possible reason the two paths differ — smooth_k:

# comfy/ldm/modules/attention.py:573 (attention_sage)
sage_kwargs = {..., "smooth_k": False}

# KJNodes get_sage_func()
return sageattn(q, k, v, is_causal=..., attn_mask=..., tensor_layout=...)
#                                      smooth_k not passed -> library default

sageattn's signature is smooth_k: bool = True. Smoothing K (subtracting its per-channel mean before INT8 quantization) is SageAttention's headline mechanism for handling channel-wise outliers, and the paper puts its cost under 0.2%. Core disables it; the KJNodes path gets it.

The KJNodes node also installs via model_options["transformer_options"]["optimized_attention_override"], bypassing attention_sage entirely — so it would also bypass a low_precision_attention=False opt-out if one were merged.

Caveat, so this isn't overstated: I did not reproduce the noise myself — I have not run the --use-sage-attention path, so I can't confirm or refute the original report, only add that the node path works. I also could not isolate smooth_k as the cause: in a standalone probe on H3-shaped tensors (56 heads × 128, including synthetic channel outliers in K), smooth_k=True and False gave identical relative error (0.0129 both), and neither produced noise. So the override mechanism may matter as much as smooth_k, and the real trigger may involve trained weight distributions my synthetic test didn't capture.

Suggestion: before merging low_precision_attention=False for H3, it may be worth testing whether flipping smooth_k to True in attention_sage fixes the core path too. As written, the proposed fix would disable a working 1.37× for anyone on the node route, and H3's attention share grows with resolution (~55% of sampling at 480p, higher at 720p+), so the loss scales up.

[pepikir]

Correction to my comment above. I've now run the core --use-sage-attention path, and it does not produce noise here. Both paths work, and my smooth_k explanation was wrong — retracting it.

Same prompt, same cached conditioning, same seed, 864×480 / 124 frames / 20 steps, RTX 3090 (sm_86), ComfyUI 0.30.1, torch 2.11.0+cu130, sageattention 1.0.6 from PyPI:

path	s/it	PSNR vs no-sage reference	output
no sage	9.00	—	reference
Patch Sage Attention KJ (auto)	6.57	32.01 dB	clean
core --use-sage-attention	6.50	30.30 dB	clean

Zero using pytorch attention instead fallbacks in either run, so sage was genuinely active both times. ~1.37× on sampling either way. So the smooth_k=False difference between attention_sage and the KJNodes wrapper is real in the source but is not what causes the noise — in a standalone probe on H3-shaped tensors, smooth_k=True and False also gave identical relative error (0.0129 both).

So the noise doesn't reproduce on Ampere. A better hypothesis than mine is that this is architecture- or version-dependent. sageattention 1.0.6 on sm_86 uses the Triton INT8-QK / FP16-PV path. On sm_89+ and Blackwell with sageattention 2.x, sageattn auto-dispatches to the FP8-PV CUDA kernels instead — a genuinely different kernel, and a plausible place for a model-specific break that Ampere never reaches.

@AI4VFX — could you post your GPU, sageattention version, and which H3 checkpoint (fp8 vs the int8_convrot W4A4 one)? That would narrow it quickly. I'm on the W4A4 checkpoint, which may also matter.

Worth flagging for the proposed fix: low_precision_attention=False would disable a working ~1.37× for every Ampere user, and H3's attention share grows with resolution and clip length (roughly 55% of sampling at 480p/5s, considerably more at 720p or 15s), so the cost of a blanket opt-out scales up. If the break turns out to be arch- or kernel-specific, gating on that rather than disabling sage outright would keep the speedup where it works.

[AI4VFX]

Thanks for actually digging into this — and you're right to push back. I owe the thread a correction of my own.

Short version: sage isn't broken on H3, and my issue title is too strong. I've run a lot more H3 with sage since I filed this, and it's clean nearly everywhere, on Blackwell too. So I don't think the blanket low_precision_attention=False should be merged as I wrote it — it would cost you your 1.37× and cost me a 1.6×.

Your three questions:

• GPU: RTX PRO 6000 Blackwell, 96 GB (sm_120)
• sageattention: 2.2.0+cu130torch2.9.0andhigher.post4
• Checkpoint: the bf16 one, not int8_convrot — minimax_h3_fl2va_bf16.safetensors with qwen3vl_32b_minimax_h3_bf16.safetensors as the text encoder

So: different arch, different sage major version, different checkpoint from you. Plenty of room for your arch/kernel theory.

That said, here's the part I think matters more, and it's why I don't think your Ampere runs settle it either way yet. On my box the failure tracks sequence length, not hardware. Same GPU, same sage build, same checkpoint, same code path — clean at some sizes, noise at one. Counting tokens as ceil(frames/4) * (W/32) * (H/32) (VAE 16×/4×, then the DiT's (1,2,2) patchify):

run	frames	grid	tokens	result
1920×1088, 5 s	124	60×34	~63k	clean
1216×704, 30 s	~736	38×22	~154k	clean
1920×1088, 15 s — the original report	362	60×34	~184k	noise

Note the middle row: 6× the frames of the first and still fine, so it isn't frame count.

Your run is 864×480 / 124 f, which comes out around 12.6k tokens — roughly 1/12th of my verified-clean ceiling and 1/15th of where I actually saw it break. I'd fully expect that to be clean on Blackwell as well. Which is the problem: right now "Ampere is fine" and "short sequences are fine" are the same experiment, so your result can't distinguish them.

If you're up for one more run, the test that would actually settle it is pushing an Ampere render up near ~180k tokens — 1280×720 at ~15 s, or just extend frames at your current res until you're in that range. Clean there and your kernel theory wins outright. Noise there and it's sequence length, and we can both stop blaming each other's GPUs.

One more caveat I should have flagged earlier, in fairness: the graph that produced the noise also contained a PathchSageAttentionKJ node set explicitly to sageattn_qk_int8_pv_fp8_cuda++, not auto. My later clean runs used the plain global flag with no node at all. So I can't cleanly separate "long sequence" from "forced fp8-PV kernel" on my own evidence either. Two live suspects, narrowed but not settled.

And agreed on smooth_k, including your retraction — I floated the same theory in the issue body and I don't think it holds up either.

For what it's worth, what I run locally now isn't the hardcoded opt-out I proposed: I gate it on an env var, so the default behaves like the proposed fix but a dedicated launcher turns sage back on for H3. That's been the practical answer here. If upstream wants something, a narrower gate — or even just a warning above some sequence length — seems much better than switching sage off for H3 across the board.

Happy to retitle this issue to something more accurate once we've worked out which of the two it is.

[mapuamap]

try with node Sage-KJ fp32

rtx 6000 blackwell 1920*1088 15sec 184k tokens

[pepikir]

@mapuamap's run is the one that separates the two suspects, and I think it settles it: it's the FP8 PV path, not sequence length.

Same GPU as @AI4VFX (RTX PRO 6000 Blackwell), same 1920×1088 / 15 s / ~184k tokens that failed, and swapping only the PV kernel makes it clean.

What "Sage-KJ fp32" actually is. There's no mode named fp32 in KJNodes — the dropdown is disabled / auto / sageattn_qk_int8_pv_fp16_cuda / sageattn_qk_int8_pv_fp16_triton / sageattn_qk_int8_pv_fp8_cuda / sageattn_qk_int8_pv_fp8_cuda++ / sageattn3 / sageattn3_per_block_mean. Exactly one of those passes a bare pv_accum_dtype="fp32":

# KJNodes get_sage_func()
elif sage_attention == "sageattn_qk_int8_pv_fp16_cuda":
    return sageattn_qk_int8_pv_fp16_cuda(..., pv_accum_dtype="fp32", ...)
elif sage_attention == "sageattn_qk_int8_pv_fp8_cuda":
    return sageattn_qk_int8_pv_fp8_cuda(..., pv_accum_dtype="fp32+fp32", ...)
elif sage_attention == "sageattn_qk_int8_pv_fp8_cuda++":
    return sageattn_qk_int8_pv_fp8_cuda(..., pv_accum_dtype="fp32+fp16", ...)

So the clean run is INT8 QK, FP16 PV, single-level FP32 accumulation — no FP8 anywhere. @mapuamap, can you confirm that's the mode you picked?

Every data point in the thread, sorted by PV path rather than by GPU:

reporter	arch	PV path	tokens	result
AI4VFX	sm_120	FP8 (++, fp32+fp16)	~184k	❌ noise
AI4VFX	sm_120	FP8 (global flag → auto)	~184k	❌ noise
mapuamap	sm_120	FP8 (global flag → auto)	~184k	❌ noise
AI4VFX	sm_120	FP8 (global flag → auto)	~154k	✅ clean
AI4VFX	sm_120	FP8 (global flag → auto)	~63k	✅ clean
mapuamap	sm_120	FP16 PV / FP32 accum	~184k	✅ clean
me	sm_86	FP16 PV / FP32 accum	12.6k → 56,120	✅ clean

Every noise report is FP8 PV. Every clean report at high token count is FP16 PV.

And my Ampere runs were never evidence about Ampere. @AI4VFX was right that my 12.6k run couldn't distinguish "Ampere is fine" from "short sequences are fine" — but the reason is simpler than either of us assumed. I'm on sageattention 1.0.6, which has no arch dispatch and no FP8 kernel at all; sageattn() is one Triton path:

q_int8, q_scale, k_int8, k_scale = per_block_int8(q, k, sm_scale=sm_scale, tensor_layout=tensor_layout)
o = attn_false(q_int8, k_int8, v, q_scale, k_scale, ...)
# and in the kernel:
acc = tl.zeros([BLOCK_M, HEAD_DIM], dtype=tl.float32)

INT8 QK, FP16 PV, FP32 accumulator. That is arithmetically the same class as @mapuamap's clean Blackwell run. So my runs weren't a test of Ampere-vs-Blackwell — they were another sample of the PV path that works, and they land in the same row of the table as the fix. I did extend Ampere clean to 56,120 tokens (1280×736 / 243 f, 4.5× my original figure), but that ceiling is beside the point now; the kernel is the variable, not the arch and not the length.

This also reconciles the sequence-length correlation rather than contradicting it. FP8 PV accumulation error grows with the number of accumulated terms, so token count sets when the FP8 path crosses into garbage — clean at 63k, clean at 154k, destroyed at 184k — while the FP16/FP32 path never crosses at all. Both observations were true; length is the trigger, FP8 is the cause. Only the cause is worth gating on.

Which makes low_precision_attention=False the wrong lever. Using @mapuamap's own wall-clock, all three rows on the same box:

config	time	vs no-sage
global sage flag	3544 s	1.71× — but noise
Sage-KJ, FP16 PV / FP32 accum	4503 s	1.35× — clean
no sage	6073 s	1.00×

The blanket opt-out takes everyone to the 1.00× row. Pinning the FP16-PV/FP32-accum kernel for H3 in attention_sage — instead of falling back to attention_pytorch — keeps 1.35× on the exact hardware and sequence length that currently fails. A sequence-length warning becomes unnecessary if the kernel that degrades with length isn't selected in the first place.

Worth noting the KJ node installs via optimized_attention_override, so it bypasses attention_sage entirely — whatever lands upstream won't affect the node route either way.

Caveats, so this isn't overstated:

• @mapuamap's is a single run per row with no sageattention version given, and I'm inferring the mode from the label.
• All three of their rows include Spectrum (ComfyUI-Spectrum-MiniMax-H3), which is an approximate accelerator that skips transformer blocks and admits to changing the trajectory. It's constant across the three rows so the comparison holds, but these aren't native-path renders.
• The only FP8 mode confirmed noisy is fp32+fp16 accumulation plus whatever auto dispatches to on sm_120. sageattn_qk_int8_pv_fp8_cuda (fp32+fp32) is untested here — if that one is also clean at 184k, the culprit narrows further to the FP16 second-level accumulator rather than FP8 PV generally. That's a cheap next run for anyone on Blackwell.

Retitling to something like "MiniMax H3: FP8 PV sage kernels produce noise at long sequence lengths on sm_120" would match the evidence better than the current title.

[AI4VFX]

This lines up with what I've been seeing, and I think you've got it right that FP8 PV is the cause.

I ran into the wall again from the other direction yesterday, which tightens the bracket a bit. All of these are the global flag (so FP8 auto-dispatch), same box as before, bf16 checkpoint. No Spectrum, though I did have EasyCache on for some of them and it made no difference to the noise either way.

res	duration	frames	tokens	result
1920×832	15 s	362	142k	clean
1920×1088	12 s	294	151k	clean
1920×1088	13 s	328	167k	noise
1920×832	18 s	447	175k	noise
1920×1088	15 s	362	186k	noise

So the crossover is somewhere between 154k and 167k, tighter than the 154k/184k I had before. And it's a cliff, not a slope. 151k is clean, 167k is garbage, nothing soft in between.

The pair worth looking at is the two 15 s rows. Same duration, same 362 frames, same temporal rope positions, one clean and one noise, only the token count differs. And 832 breaks too once I push it to 18 s, so it isn't a height thing either. That's your "length is the trigger" showing up at two different aspect ratios, which I think makes that half of it pretty solid.

I hadn't tried the FP16 PV kernel at 186k. I'd written off full res 15 s and dropped to 12 s to stay under the wall, so mapuamap's run is the first sign I don't have to. Worth a shot, and I can do the fp32+fp32 one at the same time since you asked for it.

Agreed on the retitle, the current one is wrong either way.

[AI4VFX]

Ran the rest of the kernels on the Blackwell box. Identical settings for every row below: 1920x1088, 15 s / 362 frames / ~186k tokens, 20 steps, res_multistep, same seed, and nothing else in the attention chain (no EasyCache, no Sol-Attn, no Spectrum). Short version is that none of them work here, and I think it changes the conclusion.

KJ mode	step time	result
auto	~110 s/it	completes, noise
sageattn_qk_int8_pv_fp8_cuda++ (fp32+fp16)		crashed. this is the mode from my original report, where it ran to completion and gave noise
sageattn_qk_int8_pv_fp8_cuda (fp32+fp32)	176 s/it	clean step 1, then cudaErrorIllegalAddress
sageattn_qk_int8_pv_fp16_triton	225 s/it	clean step 1, then cudaErrorIllegalAddress
sageattn_qk_int8_pv_fp16_cuda (fp32)		Fatal Python error: Aborted on the first attention call
sageattn3, sageattn3_per_block_mean		ModuleNotFoundError, sageattn3 isn't installed

The thing I'd missed until now is that auto isn't any of the entries below it. sageattention/core.py:152-159:

elif arch == "sm120":
    if get_cuda_version() < (12, 8):
        pv_accum_dtype = "fp32"
    else:
        pv_accum_dtype = "fp32+fp16"
    return sageattn_qk_int8_pv_fp8_cuda(..., qk_quant_gran="per_warp", ..., pv_accum_dtype=pv_accum_dtype)

KJNodes' fp8_cuda++ calls the same function with the same pv_accum_dtype="fp32+fp16", but never passes qk_quant_gran, so it takes the default per_thread from the signature at core.py:622. So auto and ++ differ by one parameter and it isn't the PV path. That seems worth flagging, since we've been sorting the table by PV path and those two rows share one but behave differently.

Every explicit mode is also something sage never picks on sm120. fp16_cuda is only selected for sm80/sm86. fp16_triton has a comment right there at core.py:154 saying the triton kernel isn't usable on sm120. fp8_cuda with fp32+fp32 isn't what sm120 gets either, it gets fp32+fp16, and always per_warp.

Which is why I don't think pinning an FP16 PV kernel for H3 can work. On sm120 there isn't one to pin. fp16_cuda aborts the process and fp16_triton corrupts memory, and neither falls back to anything. That would move H3 from "noise at long sequences on Blackwell" to "hard crash on Blackwell", which is worse than where we are now. If something does land upstream it probably has to be arch-aware.

@mapuamap could you post your sageattention version? A clean 186k run on an explicit mode doesn't fit the dispatch above, so either your build differs from mine (2.2.0+cu130torch2.9.0andhigher.post4) or that mode is doing something I'm not seeing.

One more thing, and this is the bit I'd most like a sanity check on. On the ++ run I got a preview where part of the frame was correct and part was noise, with a clear boundary between the two. That isn't what I'd expect from accumulation error, which should degrade the whole frame fairly evenly. A boundary looks more like something writing out of bounds.

That mode is also the one from my original report, where at these same settings it ran all the way through and produced noise rather than crashing. Numerical error ought to be deterministic, so getting a completed noisy render one time and an illegal address another makes me wonder whether the noise and the crashes are one bug rather than two, with the length correlation just being where the bad access starts landing. (Fair warning that the original run was a few weeks of Comfy versions ago and the graph around it wasn't as stripped down as tonight's, so that pair isn't as clean as the six rows above.)

Caveats so this isn't overstated. Single runs each. I didn't save the half-and-half frame, I'll grab it if it happens again. Both illegal-address crashes got through step 1 before dying, which argues against a plain indexing overflow on the first call. And VRAM is a live alternative I haven't instrumented, this model already peaks around 75 of 96 GB without sage and the fp32 accumulation paths want a much bigger workspace.

[pepikir]

That kernel table settles the part I had wrong, so let me withdraw the FP16-PV suggestion up front rather than defend it. If fp16_cuda aborts the process and fp16_triton corrupts memory on sm_120, there is nothing to pin there and "noise" would become "crash", which is strictly worse. Your original instinct — take H3 off the low-precision path — is the right lever on that arch. My objection to it was that it costs everyone the 1.35x, and that objection only holds where a working kernel exists, which on sm_120 it apparently doesn't. Arch-aware is the only shape that fits both.

On the Ampere half: I can now close it out, and the answer is that it can't be tested.

I have it verified clean to 56,120 tokens (1280x736, 243 f) — 4.5x the 12,555 I originally posted, so the ceiling moved a long way. But the number that matters is the one I should have computed weeks ago: at H3's own limits, Ampere tops out at 91,728 tokens. That's the 768x1344 area cap (MAX_PIXELS = 768 * 1344 in nodes_minimax_h3.py:27) at the top of the trained frame range (362), i.e. 1344x768 x 362 f.

	tokens	vs your crossover
my verified-clean ceiling	56,120	37% of 151k
max reachable at H3's limits	91,728	61% of your clean 151k row
your last clean row	150,960	—
your first noise row	167,280	—

So Ampere cannot reach even your clean row, let alone the crossover. Getting 1280x736 to ~186k needs about 800 frames — 33 s, more than 2x past the trained range — and your exact 1920x1088 is 2.02x over the pixel cap. The Ampere run you asked for in your first reply is not "expensive", it's unreachable without leaving the model's supported envelope entirely, and a render outside that envelope wouldn't tell us anything about kernels. I'd treat "is this arch or length" as unanswerable from the Ampere side and drop it. My data only ever supported the weak claim, which is that nothing goes wrong below ~56k.

On the boundary, since you asked for a sanity check: I think you're right, and I'd go further — it argues the noise and the crashes are one bug.

A sharp spatial boundary is hard to get from accumulation error. FP8 PV error is per-token and roughly uniform in magnitude, so it should degrade the whole frame together, and it should get worse gradually with length. You're seeing neither: clean at 151k, garbage at 167k, nothing in between, and a frame that is partly correct.

What does produce a boundary is a contiguous run of tokens being wrong while the rest are fine. Tokens map to spatial patches, so a bad tail of the sequence lands as a bad region of the frame. That's the signature of something tile-indexed going out of range past a threshold, not of arithmetic precision.

The auto vs ++ finding is what makes me think this is the scale buffers specifically. You're right that they share pv_accum_dtype="fp32+fp16" and differ only in qk_quant_gran — and that parameter's whole job is how many quantization scale entries get allocated and indexed (per_warp vs per_thread is a large difference in that buffer's size and stride). If behaviour changes with the granularity parameter while the PV accumulator is held constant, the PV path isn't the discriminating variable. That also explains why sorting the table by PV path kept producing a table that didn't sort.

One overrun would then explain both symptoms: it reads or writes past the end of a scale buffer, and where it lands is what varies. Land in mapped memory and you get garbage scales for the tail blocks — a completed render, noise in a region, deterministic-looking. Land outside and you get cudaErrorIllegalAddress. Same bug, and which one you see depends on the allocator, which is why your ++ run gave noise weeks ago and a crash last night on a differently-shaped graph. It also fits both crashes surviving step 1: the first call warms the allocator and the second lands differently.

Two cheap things that would confirm or kill this, in order of cost:

1. CUDA_LAUNCH_BLOCKING=1 on either illegal-address mode. Nearly free, and it attributes the fault to the actual kernel launch instead of wherever the async error surfaced. If it names a quant/scale kernel rather than the attention kernel itself, that's most of the answer.
2. compute-sanitizer --tool memcheck on the same run. This names the offending access and the buffer directly. It's a heavy slowdown on a 176 s/it config, but you only need it to reach step 2 — both your crashes died there, so it should fault early rather than needing the full 20 steps.

If (1) points at a scale kernel, the fix is upstream in sageattention rather than anything ComfyUI can gate, and the ComfyUI-side change stays what you proposed: don't take that path for H3 on sm_120.

Caveats:

• The ceil(frames/4) * (W/32) * (H/32) figure we've both been quoting reproduces all five of your rows exactly, so it's a consistent metric for comparing runs, but I don't think it's the model's literal sequence length. video_latent_t() at nodes_minimax_h3.py:39 gives 107 temporal positions for 362 frames, not 91, which would put your 186k config nearer 218k of actual attention tokens. It doesn't change any ordering or ratio here, but if the threshold turns out to sit at a round number in some buffer, it's worth pinning down which count that number is in.
• My 56,120 is one clean run, not a bisection — the true Ampere ceiling is somewhere above it, and I stopped looking once it was clear the interesting range was out of reach.
• The overrun story is inference from your two symptoms plus the granularity parameter. I haven't run sanitizer on it and I can't, since the failure lives at lengths my hardware can't produce.

[LiJT]

Yes this is true, 2.0MP 1920x1088 15 seconds video generation failed on 5090 for 1hr computing.

[pepikir]

@LiJT thanks — that's the same row we already have, but the board it's on is the useful part.

Your config is the known-bad one. 1920x1088 at 15 s is 362 frames, so ceil(362/4) * (1920/32) * (1088/32) = 91 * 60 * 34 = 185,640 tokens, the same ~186k row @AI4VFX has failing on the global flag. Your screenshot shows 20/20 at 206.75 s/it and Prompt executed in 01:09:01, with the frame uniformly noisy — that matches the auto rows rather than the half-and-half boundary AI4VFX caught once on ++.

But you're on a 5090, and that's the control this thread has been missing. Every failing run so far is an RTX PRO 6000 with 96 GB. AI4VFX flagged VRAM as the one alternative nobody had instrumented — H3 already peaks around 75 of 96 GB without sage, and the fp32 accumulation paths want a much bigger workspace. Your card is the same arch (GB202, sm_120) with 32 GB, a third of the memory, and it fails at the same config. That's consistent with the kernel explanation, but a single point can't place a threshold, and the threshold is where the answer is.

So the run I'd most like from you. AI4VFX's crossover on the 96 GB box sits between exactly two rows:

res	duration	frames	tokens	96 GB result
1920x1088	12 s	294	151k	clean
1920x1088	13 s	328	167k	noise

Those same two on 32 GB. If the boundary lands in the same place with a third of the memory, VRAM is dead and the kernel wins outright. If it moves down, VRAM is live and every row in the table needs re-reading.

Cheap version of that ask: enable latent previews and look at step 1-2 instead of waiting for step 20. I don't know whether the auto noise is visible that early — worth checking once on the config you already know fails, and if it is, the bisect is minutes rather than the two hours it currently looks like.

Two one-line questions, in order of how much they'd narrow things:

1. Your CUDA version (python -c "import torch; print(torch.version.cuda)"). sageattention/core.py picks pv_accum_dtype="fp32" for sm120 below CUDA 12.8 and "fp32+fp16" at or above it. If you're under 12.8 and still get noise, that rules out the FP16 second-level accumulator, which is one of the two suspects left standing.
2. Global --use-sage-attention, or a Patch Sage Attention KJ node — and your sageattention version.

One warning, because you will hit it if you follow the issue body. The workaround at the top — KJNodes set to sageattn_qk_int8_pv_fp16_cuda — does not work on sm_120. AI4VFX went through every mode on the same GPU class afterwards and that one gives Fatal Python error: Aborted on the first attention call, while fp16_triton gives cudaErrorIllegalAddress. The body predates that comment and hasn't been updated. @AI4VFX — worth editing it, since people arriving here read the body and not ten comments.

What is actually clean on sm_120 today is staying under the crossover, and both of these are measured rather than extrapolated:

res	duration	tokens	result
1920x832	15 s	142k	clean
1920x1088	12 s	151k	clean

So you can keep the full 15 seconds at 1920x832, or keep the height and drop to 12 s. Failing that it's --use-sage-attention off for H3 and the 1.00x baseline.

Caveats: I'm inferring your kernel path from a screenshot that only shows step time and the result, so if you were on a node rather than the global flag, question 2 changes what your run means. And I'm on Ampere and cannot reach any of these token counts on my own hardware, so all of the above is a reading of other people's runs, not mine.