Skip to content

feat(minimax-h3): tiny-VAE (taeh3) step previews with x-hat-0 estimates - #147

Open
lstein wants to merge 1 commit into
minimax-h3/07-quantized-text-encoderfrom
minimax-h3/08-tae-preview
Open

feat(minimax-h3): tiny-VAE (taeh3) step previews with x-hat-0 estimates#147
lstein wants to merge 1 commit into
minimax-h3/07-quantized-text-encoderfrom
minimax-h3/08-tae-preview

Conversation

@lstein

@lstein lstein commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Summary

Step previews for MiniMax H3 denoising were useless until the penultimate step: the callback projected the raw noisy latents xₜ through a placeholder grayscale channel mean. This PR fixes both halves.

1. x̂₀ previews (backend loop)

denoise() now hands the step callback the scheduler's own predicted-clean estimate of the generated video rows — x0 = x_t + sigma*v (H3's data-ward velocity), computed before the in-place Euler update, condition rows excluded, float32. Unlike xₜ, x̂₀ is decodable at every step, so structure appears within the first few steps. At the final step (sigma_next = 0) the estimate equals the returned latents exactly — pinned by a new test.

2. Tiny-VAE decode (taeh3)

The denoise node decodes that estimate with madebyollin's taeh3 tiny decoder (MIT, taehv @ 62f7591), fetched once (~23 MB) through the remote-model cache, with graceful fallback if the download or the decode fails. Vendored decoder-only in invokeai/backend/minimax_h3/taehv_decoder.py; bit-exact against the reference implementation on identical input.

Ground-truth verification against the released H3 video VAE settled the space convention: taeh3 consumes the normalized latent space ((z - mean)/std — exactly the denoise loop's space). A real-VAE encode of a synthetic test image round-trips through taeh3 at ~21 dB PSNR; feeding unnormalized latents produces garbage. Per-step preview cost: a 2-latent-frame window ending at the middle frame decodes a full 768×1344 frame in ~16 ms (vs ~50 s/step for the int8 transformer).

3. Real fallback factors

The no-download fallback (step_callback.py) gets a real 24×3 latent→RGB projection + bias, least-squares fitted here against the released H3 VAE encoder over real photos + synthetic gradients (fit rms ~0.09 in [-1,1]) — fitted, not copied from ComfyUI. Replaces the grayscale placeholder.

Notes

  • Callback contract change: denoise()'s step_callback now receives (step, total, pred_x0_generated_video_rows) (pre-sliced, float32) instead of the full noisy row buffer. The invocation is the only callback consumer.
  • The decoder is locked in VRAM for the duration of the loop (23 MB, far below the reserved working memory), entered after the transformer's RAM load and before its VRAM lock; every preview-path failure degrades to the linear projection — none can abort a generation.
  • Node version bumped to 1.1.0; no field changes, existing workflows unaffected. No frontend changes needed (progress-image plumbing is generic).
  • Adversarial fresh-context review: no blocking defects; both low-severity findings (unguarded VRAM-lock aborting on OOM; cache-record drop/reload noise) fixed in the final commit.

Testing

  • 6 new tests: decoder shape/range/window semantics, state-dict layout pinned against the released checkpoint file, loader strict-load + encoder-key filtering, x̂₀ callback contract (shape/dtype/final-step identity, keyframe case). 138 backend tests pass.
  • Empirical: strict load of the real released taeh3.safetensors; bit-equivalence vs reference; real-VAE round-trip PSNR; 16 ms full-size decode on the W7900.

🤖 Generated with Claude Code

The step preview previously projected the RAW noisy latents x_t through a
uniform grayscale channel mean, so nothing recognizable appeared until the
last couple of steps. Three changes:

- The denoise loop now hands the step callback the scheduler's own
  predicted-clean estimate (x0 = x_t + sigma*v, data-ward velocity) of the
  generated video rows, which is decodable at every step.
- The denoise invocation decodes that estimate with madebyollin's taeh3 tiny
  decoder (MIT, ~23 MB one-time download via the remote-model cache; vendored
  decoder-only, bit-exact against the reference implementation). A full-size
  768x1344 preview frame costs ~16 ms per 50 s step. Verified against the
  real H3 video VAE that taeh3 consumes the NORMALIZED latent space (the
  denoise loop's space): real-VAE encode -> taeh3 decode reconstructs at
  ~21 dB, unnormalized latents produce garbage.
- The no-download fallback path gets real latent->RGB factors + bias,
  least-squares fitted against the released H3 VAE encoder (rms ~0.09),
  replacing the grayscale placeholder.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant