feat(MODEL-MM-indextts2): the DiT conditioning front end, and a tensor that is DEAD in 2.5 (#634) - #745
Merged
Merged
Conversation
…r that is DEAD in 2.5 (#634) Ports `diffusion_transformer.py:206-226` (index-tts @4f8792ff): cond_projection, the four-way concatenation, the style broadcast, the classifier-free-guidance zeroing, and cond_x_merge_linear. The shipped `cond_x_merge_linear.weight` is [512, 864], and 864 is exactly 512 + 80 * 2 + 192, so the concatenation order and widths are pinned by the weight itself rather than by my reading of the code. Two things contradict what a reader expects, and both are now recorded where someone will hit them. `cond_in_module` is FORCED to `cond_projection` upstream: the `content_type` switch that would have selected `cond_embedder` is commented out. So `cond_embedder` is present in `s2mel.pth` and DEAD in 2.5. A port that "restored" the switch, which is exactly what reading the surrounding code suggests, would consult a tensor this model never uses. `class_dropout` is NOT a training-only path. `mask_content` sets it at inference, and it zeroes everything AFTER the first `in_channels` columns -- keeping x, dropping prompt, cond and style. That IS the CFG unconditional branch that `cfm::EulerStepCfg` consumes, so it is ported as a first-class argument rather than an artefact. Gated against upstream on BOTH branches, and the behavioural cases are chosen to separate near-misses rather than merely to be non-vacuous: under the unconditional branch, perturbing x must move the output while perturbing prompt, cond or style must NOT; under the conditional branch all four must move it; and style is proven BROADCAST rather than read per frame by shifting the whole vector and requiring every frame to move by the same delta. 5 cases / 128 assertions. Six mutations, six caught, all compile_err=0, file verified byte-identical: raw cond used instead of the projection, the concat order swapped, x read frame-major instead of channel-major, the unconditional branch zeroing x as well, the unconditional branch keeping style, and style indexed by frame instead of broadcast. Still no render: the DiT BLOCKS between this front end and the ported tail are not wired, and correctness against vLLM-Omni needs the oracle pin (#633). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
added a commit
that referenced
this pull request
Aug 14, 2026
…imator is COMPLETE front to tail (#634) (#748) Ports `gpt_fast/model.py:161-191` (Transformer.forward): N blocks, the U-Net skip routing across them, and the final AdaptiveLayerNorm. With #738, #739, #741 and #745 the S2Mel DiT now exists end to end -- front end, stack, tail -- each gated against upstream's own modules. This is composition only. The block is `dit::Block`, the routing is `dit_skip::Plan`, the merge is `dit_skip::ApplySkip`, and nothing here reimplements any of them. The rotary table is passed IN, emitted by the generator from upstream's own `freqs_cis`, so the gate measures composition rather than a second copy of `precompute_freqs_cis`. One upstream quirk is recorded rather than tidied: a `skip_in_linear` is built on EVERY layer when `uvit_skip_connection` is set, including layers that never receive a skip, so the checkpoint carries unused ones. They load and are left alone. A case asserts that layer 0's skip_in_linear cannot affect the output at all, while a receiving layer's must. 3 cases / 52 assertions against upstream Transformer.forward. Five mutations, five caught, all compile_err=0, file byte-identical after: the skip merged against itself instead of the popped value, emitting a layer's INPUT instead of its output, FIFO instead of LIFO, the final norm dropped, and the receive moved after the block. A SIXTH mutation was withdrawn as badly designed rather than reported as a gate hole. Pushing before the block WITHOUT removing the push after it leaves the stack holding both, and because a layer's input IS the previous layer's output, every pop still returned the same tensor -- a genuine no-op, not a defect the gate missed. Rewritten to emit the input INSTEAD of the output, it fails. Still no render: the talker, the reference encoders and BigVGAN are unported, so nothing yet produces the conditioning this estimator consumes, and correctness against vLLM-Omni needs the oracle pin (#633). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
localai-bot
pushed a commit
that referenced
this pull request
Aug 14, 2026
Merges the PINNED SHA `32d82c64d`, not the ref. `origin/main` is shared with other worktrees in this checkout, and it advanced from `5e646d98c` to `32d82c64d` mid-merge when another session fetched — which made main's own newer files (the `dit_front` / `dit_stack` IndexTTS2 cluster, #745/#748) read as "silently dropped by the merge" in every `git diff origin/main` afterwards. They were never dropped; they are simply not in `5e646d98c`. That first attempt was aborted rather than patched, because the repair it invited — restoring files from the newer main into a tree merged from the older one — produced a CMakeLists listing TUs the tree did not contain. Merging an immutable SHA is what makes the verification below mean anything. Three conflicts, all resolved against that SHA: * `tests/vllm/multimodal/test_ltx2_video.cpp` — both sides added a different include in the same sorted run: this branch's `ltx2_video_vae_encoder.h` and main's `vllm/platforms/interface.h`. Both are kept, in sorted order. Proved by COMPILING, not by reading: `BUILD_EXIT=0` and the suite runs. * `.agents/roadmap_v1.md` and `docs/FEATURES.md` — keyed records, resolved by taking main's file WHOLESALE and reapplying this row's key onto it, never by accepting the automatic three-way result. Each file is now byte-identical to `32d82c64d` apart from exactly one line: the `#644` row and the LTX-2.5 DiT row. An earlier merge in this campaign auto-merged two `#644` rows into a duplicate that `check-agent-record` caught; taking one side wholesale is what makes that shape impossible rather than merely unlikely. The combined `#644` row names both campaign rows and both specs. The FEATURES row carries main's prompt-AdaLN facts plus this row's served image-conditioning arm, inside the 220-char cell and 600-char entry budgets. `docs/USAGE.md`, `CMakeLists.txt` and `tests/CMakeLists.txt` auto-merged, and were each verified against the pinned SHA rather than trusted: both CMake files are PURELY ADDITIVE (zero removed lines; this branch adds the two `ltx2_video_vae_encoder_load` / `ltx2_image_preprocess` TUs and the one `test_ltx2_image_cond` registration), and USAGE's only removal is the paragraph this row supersedes, with main's own additions still present. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(MODEL-MM-indextts2): the DiT conditioning front end, and a tensor that is DEAD in 2.5 (#634)
Ports
diffusion_transformer.py:206-226(index-tts @4f8792ff): cond_projection,the four-way concatenation, the style broadcast, the classifier-free-guidance
zeroing, and cond_x_merge_linear. The shipped
cond_x_merge_linear.weightis [512, 864], and 864 is exactly512 + 80 * 2 + 192, so the concatenation order and widths are pinned by the
weight itself rather than by my reading of the code.
Two things contradict what a reader expects, and both are now recorded where
someone will hit them.
cond_in_moduleis FORCED tocond_projectionupstream: thecontent_typeswitch that would have selected
cond_embedderis commented out. Socond_embedderis present ins2mel.pthand DEAD in 2.5. A port that"restored" the switch, which is exactly what reading the surrounding code
suggests, would consult a tensor this model never uses.
class_dropoutis NOT a training-only path.mask_contentsets it atinference, and it zeroes everything AFTER the first
in_channelscolumns --keeping x, dropping prompt, cond and style. That IS the CFG unconditional
branch that
cfm::EulerStepCfgconsumes, so it is ported as a first-classargument rather than an artefact.
Gated against upstream on BOTH branches, and the behavioural cases are chosen to
separate near-misses rather than merely to be non-vacuous: under the
unconditional branch, perturbing x must move the output while perturbing prompt,
cond or style must NOT; under the conditional branch all four must move it; and
style is proven BROADCAST rather than read per frame by shifting the whole
vector and requiring every frame to move by the same delta.
5 cases / 128 assertions. Six mutations, six caught, all compile_err=0, file
verified byte-identical: raw cond used instead of the projection, the concat
order swapped, x read frame-major instead of channel-major, the unconditional
branch zeroing x as well, the unconditional branch keeping style, and style
indexed by frame instead of broadcast.
Still no render: the DiT BLOCKS between this front end and the ported tail are
not wired, and correctness against vLLM-Omni needs the oracle pin (#633).
Issue: #634.
🤖 Generated with Claude Code