feat(minimax-h3): tiny-VAE (taeh3) step previews with x-hat-0 estimates - #147
Open
lstein wants to merge 1 commit into
Open
feat(minimax-h3): tiny-VAE (taeh3) step previews with x-hat-0 estimates#147lstein wants to merge 1 commit into
lstein wants to merge 1 commit into
Conversation
The step preview previously projected the RAW noisy latents x_t through a uniform grayscale channel mean, so nothing recognizable appeared until the last couple of steps. Three changes: - The denoise loop now hands the step callback the scheduler's own predicted-clean estimate (x0 = x_t + sigma*v, data-ward velocity) of the generated video rows, which is decodable at every step. - The denoise invocation decodes that estimate with madebyollin's taeh3 tiny decoder (MIT, ~23 MB one-time download via the remote-model cache; vendored decoder-only, bit-exact against the reference implementation). A full-size 768x1344 preview frame costs ~16 ms per 50 s step. Verified against the real H3 video VAE that taeh3 consumes the NORMALIZED latent space (the denoise loop's space): real-VAE encode -> taeh3 decode reconstructs at ~21 dB, unnormalized latents produce garbage. - The no-download fallback path gets real latent->RGB factors + bias, least-squares fitted against the released H3 VAE encoder (rms ~0.09), replacing the grayscale placeholder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lstein
force-pushed
the
minimax-h3/07-quantized-text-encoder
branch
from
August 6, 2026 00:28
471df42 to
724ad4d
Compare
lstein
force-pushed
the
minimax-h3/08-tae-preview
branch
from
August 6, 2026 00:28
13fe344 to
5227464
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step previews for MiniMax H3 denoising were useless until the penultimate step: the callback projected the raw noisy latents xₜ through a placeholder grayscale channel mean. This PR fixes both halves.
1. x̂₀ previews (backend loop)
denoise()now hands the step callback the scheduler's own predicted-clean estimate of the generated video rows —x0 = x_t + sigma*v(H3's data-ward velocity), computed before the in-place Euler update, condition rows excluded, float32. Unlike xₜ, x̂₀ is decodable at every step, so structure appears within the first few steps. At the final step (sigma_next = 0) the estimate equals the returned latents exactly — pinned by a new test.2. Tiny-VAE decode (taeh3)
The denoise node decodes that estimate with madebyollin's
taeh3tiny decoder (MIT, taehv @62f7591), fetched once (~23 MB) through the remote-model cache, with graceful fallback if the download or the decode fails. Vendored decoder-only ininvokeai/backend/minimax_h3/taehv_decoder.py; bit-exact against the reference implementation on identical input.Ground-truth verification against the released H3 video VAE settled the space convention: taeh3 consumes the normalized latent space (
(z - mean)/std— exactly the denoise loop's space). A real-VAE encode of a synthetic test image round-trips through taeh3 at ~21 dB PSNR; feeding unnormalized latents produces garbage. Per-step preview cost: a 2-latent-frame window ending at the middle frame decodes a full 768×1344 frame in ~16 ms (vs ~50 s/step for the int8 transformer).3. Real fallback factors
The no-download fallback (
step_callback.py) gets a real 24×3 latent→RGB projection + bias, least-squares fitted here against the released H3 VAE encoder over real photos + synthetic gradients (fit rms ~0.09 in [-1,1]) — fitted, not copied from ComfyUI. Replaces the grayscale placeholder.Notes
denoise()'sstep_callbacknow receives(step, total, pred_x0_generated_video_rows)(pre-sliced, float32) instead of the full noisy row buffer. The invocation is the only callback consumer.Testing
taeh3.safetensors; bit-equivalence vs reference; real-VAE round-trip PSNR; 16 ms full-size decode on the W7900.🤖 Generated with Claude Code