Skip to content

L9B: the LTX-2.5 render driver, --video-family, and the first shipped-weights render - #548

Merged
localai-bot merged 2 commits into
row/MODEL-DIFFUSION-LTX25from
row/LTX25-L9B-RENDER
Aug 13, 2026
Merged

L9B: the LTX-2.5 render driver, --video-family, and the first shipped-weights render#548
localai-bot merged 2 commits into
row/MODEL-DIFFUSION-LTX25from
row/LTX25-L9B-RENDER

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Row MODEL-DIFFUSION-LTX25, phase L9B. Issue #435. Spec .agents/specs/ltx-2-5.md.
Base is row/MODEL-DIFFUSION-LTX25 (the campaign branch), not main.

What lands

examples/ltx2_gen (ltx2-gen) — a thin client of the public C ABI and
nothing else. It exists rather than a flag on minimax-h3-gen because LTX-2.5
cannot load without three of the v18 family-specific extras (the audio-stream
prompt embeds, the DiT config the shipped FP8 checkpoint does not carry, and the
latent spatial upsampler the distilled recipe's second phase needs), and that
example drives neither family nor the extras arrays. It declares the
family rather than letting detection infer it, because an FP8-vs-NVFP4
comparison is only a statement about the two files if the family is declared.

--video-family on the server. The code said the flag was deliberately not
invented "until a second family exists to disambiguate"; a second family now
exists. Detection stays the default when the flag is absent, so every
pre-existing invocation is byte-identical. --video-extra KEY=VALUE lands with
it, because pinning a family is useless if that family's required load knobs are
unreachable. The declared name is validated in ParseArgs against
RegisteredVideoFamilies(), before the text model loads, mirroring the
--tool-call-parser precedent; the registry is the authority, never a literal
list.

Evidence

RED first, black-box, because the flag table lives in an anonymous namespace
inside server_main.cpp and the binary is the surface under test. Three new
ctests; all three failed with unknown argument '--video-family' before the
change and pass after.

The render, on dgx.casa GB10 under one flock $HOME/gpu.lock hold. The
shipped vonkaiser FP8 DiT (21.00B, 6124 tensors) under LTX-2.5's declared
config (extracted from the first-party NVFP4 sibling's __metadata__:
frequencies_precision=float64, av_ca_timestep_scale_multiplier=1000) renders
end to end at 128x128 / 9 frames through both distilled phases: 9 PPM
frames, a 0.33 s 48 kHz stereo WAV that is not silent, and a valid h264+AAC MP4.

The frames are not a scene. They are smooth low-frequency colour fields.
Measured: neighbour |dx| / whole-image sd is 0.03-0.06 where white noise gives
~1.13, so they are not noise either, and the block-mean aligned/offset ratio sits
at 0.95-1.19, so they are not MiniMax-H3's patch grid. That is the expected
outcome rather than a defect in the DiT: the Gemma-4 tower is unported and so
is the embeddings connector
between it and cross-attention, so the conditioning
never passes through two of its links.

The size ceiling is memory. 320x192 / 25 frames staged fine (~44 GB) and then
grew device usage by a further ~58 GB during the denoise loop. GB10's memory is
unified and this box reboots instead of OOM-killing — it did, twice, before the
run went behind a watchdog that stops it at a floor we choose. docs/USAGE.md
documents the command with both facts.

Not in this PR

  • The NVFP4 arm. row/LTX25-L9A-NVFP4-LINEAR retracted its premise rather
    than implementing (see f400413ef), so the first-party NVFP4 DiT still does
    not load and the second arm is blocked upstream of this work.
  • Any speed number. Spec §0: vLLM-Omni carries no native 2.5 path, so there
    is no production-configuration denominator. Wall clocks in the logs are sizing
    only.

🤖 Generated with Claude Code

…t shipped-weights render

FOLLOWING_AGENTS_PROTOCOL

A way to drive an LTX-2.5 render, the server flag a second video family makes
necessary, and the plain record of what the shipped weights actually produced.

`examples/ltx2_gen` (ltx2-gen) is a THIN CLIENT of the public C ABI and nothing
else. It exists rather than a flag on minimax-h3-gen because LTX-2.5 cannot LOAD
without three of the v18 family-specific extras -- the audio-stream prompt
embeds, the DiT config the shipped FP8 checkpoint does not carry, and the latent
spatial upsampler the distilled recipe's second phase needs -- and that example
drives neither `family` nor the extras arrays. It DECLARES the family rather
than letting detection infer it, because an FP8-vs-NVFP4 comparison is only a
statement about the two FILES if the family is declared.

`--video-family` on the server. The code there said the flag was deliberately
not invented "until a second family exists to disambiguate"; a second family now
exists. Detection stays the default when the flag is absent, so every
pre-existing invocation is byte-identical. `--video-extra KEY=VALUE` lands with
it, because pinning a family is useless if that family's required load knobs are
unreachable: LTX-2.5 needs `dit_config_path` and `audio_prompt_embeds_path`, and
neither has -- or should have -- a dedicated flag on a family-generic server. A
bare KEY is refused rather than read as KEY="", and a `--video-extra partition=X`
contradicting `--video-partition Y` is refused rather than resolved by assignment
order. The declared name is validated in ParseArgs against
RegisteredVideoFamilies(), before the TEXT model loads, for the same reason
--tool-call-parser is; the REGISTRY is the authority, never a literal list, so a
family added in its own file needs no edit here.

RED first, black-box, because the flag table lives in an anonymous namespace
inside server_main.cpp and the binary IS the surface under test. Three new
ctests, all three failing with "unknown argument '--video-family'" before the
change and passing after. The registered-name case proves the check consults the
REGISTRY by getting PAST it to the next validation, so a hardcoded list omitting
ltx-2.5 REDs.

WHAT THE RENDER PRODUCED, stated as measured. The shipped 21.00B FP8 DiT under
LTX-2.5's DECLARED config renders end to end on the GB10 at 128x128 / 9 frames
through BOTH distilled phases: 9 PPM frames, a 0.33 s 48 kHz stereo WAV that is
not silent, and a valid h264+AAC MP4. The frames are NOT a scene. They are
smooth low-frequency colour fields -- neighbour |dx| / whole-image sd is 0.03 to
0.06 where white noise gives ~1.13, so they are not noise either, and the
block-mean ratio near 1.0 says they are not H3's patch grid. That is the
expected outcome, not a defect in the DiT: the Gemma-4 tower is unported AND so
is the embeddings connector between it and cross-attention, so the conditioning
never passes through two of its links.

AND THE SIZE CEILING IS MEMORY. 320x192 / 25 frames staged fine and then grew
device usage by a further ~58 GB during the loop; unified memory makes those
host bytes and this box reboots instead of OOM-killing -- it did, twice, before
the run went behind a watchdog that stops it at a floor we choose. USAGE
documents the command WITH both facts, because a recipe printed without them
reads as working.

Issue: #435

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
@localai-bot

Copy link
Copy Markdown
Collaborator Author

CI note: the device-leakage job is red on inherited debt, not on this diff.

src/vllm/multimodal/ltx2_video.cpp:404: [kcuda] vt::Backend* backend = vt::TryGetBackend(vt::DeviceType::kCUDA);
src/vllm/multimodal/ltx2_video.cpp:421: [kcuda] im.device = vt::Device{vt::DeviceType::kCUDA, index};
ERROR: DSR REGRESSION in bucket 'kcuda': 2 > baseline 0.

This PR touches neither file. Both lines are present verbatim at f400413ef,
the base commit, so the gate fires for any PR opened against
row/MODEL-DIFFUSION-LTX25:

$ git show f400413ef:src/vllm/multimodal/ltx2_video.cpp | grep -n 'DeviceType::kCUDA'
404:    vt::Backend* backend = vt::TryGetBackend(vt::DeviceType::kCUDA);
421:    im.device = vt::Device{vt::DeviceType::kCUDA, index};

Filed as #553 rather than fixed here: it is a phase-L8 device-wiring question
(route the device choice through the platform/provider table, or allowlist the
platform leg with a reason), and AGENTS.md keeps a change a reviewer would not
expect in this PR on the normal row/spec/review path.

…out one model

Only conflict was docs/FEATURES.md and docs/USAGE.md, and neither was a union:
both sides describe the SAME keyed row with facts gathered at different points
in the campaign, so an append would have left the table asserting a model both
renders and does not.

FEATURES: the branch knew the render facts (128x128/9f e2e, valid MP4+WAV,
frames NOT a scene, 320x192 exhausts unified memory) and the campaign knew the
coverage facts (both VAE encoders + mel front-end, the conditioning items, the
6.0e-07 encode bound, FP8 + torchao-NVFP4 loading). Each was accurate about
itself and wrong about the other. Resolved to the facts true of the MERGED
tree, keeping both halves and the "first-party NVFP4 does NOT load" limit,
which L9A has not landed yet.

USAGE: the campaign text said no CLI flag, server endpoint or C ABI call
reaches LTX-2.5 and that there is "nothing to run here yet". That stopped
being true on this branch, which added ltx2-gen and the family flag. Took the
branch's reachable-path opening and kept the campaign's encoder-conditioning
caveat, which is still exactly true: the encoders existing does not make image,
keyframe, reference-video or reference-audio conditioning usable, because the
request-side work is unported and the engine refuses each by name.

Worth recording that this conflict was structural rather than accidental.
doc-checkpoint REQUIRES a FEATURES.md edit from every phase, so every phase
collides there -- which is the lock AGENTS.md warns about, created by the gate
rather than by the people touching it. Filed separately.

Issue: #435

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
@localai-bot
localai-bot merged commit 2c1dda5 into row/MODEL-DIFFUSION-LTX25 Aug 13, 2026
0 of 22 checks passed
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Landed on row/MODEL-DIFFUSION-LTX25 at aa6aa0ecd38cd1fb9f is an ancestor (verified with git merge-base --is-ancestor), folded in at 25bfd6640.

This branch had gone CONFLICTING, and the conflict set was only docs/FEATURES.md and docs/USAGE.md — no source, no test, no record file. The code merged clean.

Resolving it needed judgement rather than a union: both sides described the SAME keyed row with facts gathered at different points, so appending them would have produced a table asserting the model both renders end-to-end and has no reachable entry point. Same for docs/USAGE.md, where the campaign's "there is nothing to run here yet" stopped being true the moment this branch added ltx2-gen and the family flag.

That contention was structural, not carelessness — doc-checkpoint requires every phase of a multi-phase port to write docs/FEATURES.md, so N concurrent phases collide on one table row by construction. Filed as #595, which is the lock AGENTS.md explicitly warns about: "if a checker requires every change to touch a shared file, that is the defect, not the discipline of the people touching it."

Closing rather than merging; it reaches main through #437.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants