feat(LTX25-RETAKE): regenerate a time window of an existing clip, on upstream's frame-folder arm (#924, #987)

[localai-bot]

RetakePipeline keeps the source clip outside [start_time, end_time) and regenerates what is inside it from the prompt (Lightricks/LTX-2 packages/ltx-pipelines/src/ltx_pipelines/retake.py:53,151-329 @ fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca, verified with git rev-parse HEAD on a clean local checkout before any anchor here was taken). It is reachable from vllm_video_generate and from ltx2-gen through --pipeline-kind retake, --ref-video and three window flags.

Issue #924. Spec .agents/specs/ltx25-retake.md, committed before any implementation. Also closes #987, filed and fixed in this flow.

The source is a frame directory, and that is upstream's own second arm

Upstream opens a container with PyAV (media_io/decode.py:226) and no demuxer is vendored here — src/vllm/entrypoints/openai/video_api.cpp:115-121 says so where it defines the reference-video field — so a .mp4 is refused by name. Upstream also carries a folder arm for exactly the no-container case, and three consequences follow from its own lines rather than from a local simplification:

• the frame rate must be supplied, because a folder has none (decode.py:213-215);
• a folder carries no audio stream at all (utils/helpers.py:261-262 returns None before opening anything);
• both of retake's audio predicates are conjunctions with initial_audio_latent is not None (retake.py:279,282), so regenerate_audio has no observable effect on this arm and the soundtrack is generated fresh.

The last one is surprising. It is upstream's, it is mirrored rather than repaired, and test_ltx2_retake pins it so the next reader finds it asserted instead of re-deriving it.

What the port had to get right

TemporalRegionMask (ltx-core/conditioning/types/noise_mask_cond.py:9-45) lands as a replacement of the all-ones denoise mask create_initial_state writes (ltx-core/tools.py:158-161), which is what state.denoise_mask.copy_ does at :44. Nothing else in the phase loop changes, and that is the design rather than a coincidence: every consumer of that mask already broadcasts it, so seeding video_initial with the encoded clip and zeroing the mask outside the window is the whole mechanism. video.clean = video.latent is what makes the blend restore the source rather than zeros (tools.py:156).

Four things in it fail silently, and each has a case:

• causal_fix defaults the other way at the call site. get_pixel_coords declares False (patchifiers.py:140); TemporalRegionMask calls it with getattr(latent_tools, "causal_fix", True) (noise_mask_cond.py:33). Reading the function's default moves every video boundary by time - 1 = seven pixel frames and still renders.
• The two sides use different coordinates. Video bounds are latent frames pushed through get_pixel_coords and divided by fps (:31-35); audio bounds are already seconds (:27-29, via patchifiers.py:216-249) and must not be scaled again.
• The test is overlap, not containment (:39), so a token straddling either edge is inside. Containment drops the boundary tokens and leaves a seam at each end.
• The write replaces, it does not conjoin (:44).

Ltx2ConformLatentLength truncates or zero-pads (utils/helpers.py:149-162) and is deliberately a second function rather than a flag on the audio-to-video one, which truncates only (a2vid_two_stage.py:202) and where a short take is an error because create_initial_state asserts the shape (ltx-core/tools.py:146-148). Both polarities are upstream's, on two callers; one shared helper would have to pick one and be wrong for the other.

The retake recipe is its own row in ResolveLtx2PipelineRecipe rather than distilled_two_stage with a phase removed: that recipe renders stage 1 at spatial_downscale = 2, and a full-resolution source latent does not fit that grid, so a retake on it is refused by name. One phase, DISTILLED_SIGMAS (retake.py:287, distilled defaults True at :85 and the CLI hard-codes it at :359), plain Euler (DiffusionStage.__call__ defaults to EulerDiffusionStep(), utils/blocks.py:526-527, and retake overrides neither) and no negative prompt (SimpleDenoiser at :290-294, [prompt] alone at :259).

Arms

arm	state
bf16 / FP8 / NVFP4	served — retake adds no weight-bearing module, so whatever Ltx2VideoEngine::Load already resolves serves it unchanged
GGUF	not applicable, and the reason is upstream's rather than a preference: its inference quantization kinds are exhaustive and assert_never-closed at ltx-pipelines/utils/quantization_factory.py:23-26 (assert_never at :50) — fp8-cast, fp8-scaled-mm, nvfp4-cast, nvfp4-prequant. There is no GGUF arm to port
container source (.mp4, …)	refused by name — no demuxer
EXR-frame folder and any HDR colour space	refused by name — no OpenImageIO, no scene-linear path; out of scope per .agents/specs/ltx25-retire-dead-arms.md:167
retake on a (kind, version) pair the table does not carry	refused by name
retake alongside audio_path	refused by name — retake's audio comes from the source file only (retake.py:250-256)
a clip off the 8k+1 grid or off the 32-pixel grid	refused, naming the value that would have worked (retake.py:347-353)

Two stale refusal sentences repaired in the same flow (#987)

ltx2_video.cpp claimed "nothing reads ref_video_dir at all". That was false about the tree when it was written: MiniMax-H3 consumes the directory in full (ReadReferenceClipChw, src/vllm/multimodal/minimax_h3_video.cpp:135, called at :650), and #975 inherited the wider claim from this message. It is doubly false now. It also claimed "there is no AUDIO_VAE_ENCODER key filter", which c2019b0e3 landed.

Both are rewritten in the WHAT IS *NOT* THE REASON shape so the ruled-out cause stays visible, and the guarding case asserts the local fact that the LTX side now reads the directory — a case asserting only upstream symbol names is exactly how the first sentence survived.

The reference-conditioning arm itself stays refused and #975 stays open. The !wants_retake guard narrows the refusal rather than weakening it: retake encodes the clip at its own resolution and seeds the initial latent, while the reference item is downscaled by the adapter's factor, temporally subsampled and appended as extra tokens to a stage-1-only adapter (iclora_utils.py:112-117,87-89,144-148). Serving the first says nothing about the second, and mutation M8 (dropping the guard) is red.

Upstream ships no tests

find /home/mudler/_git/LTX-2 -name 'test_*.py' returns 0 lines at this pin, and so does find . -iname '*test*' -not -path './.git/*'. "Port the upstream tests in the same change" therefore has nothing to port, and the obligation became pinning upstream's behaviours against file:line anchors, plus one assertion tied to a local fact so the suite can see a refusal going stale. This is stated rather than quietly skipped, because a reader who sees no ported test file should be able to tell "upstream had none" from "the porter did not look".

Reachability

Production entry point. vllm_video_generate (include/vllm.h:969) → src/capi/vllm_c.cpp → Ltx2VideoEngine::Generate, where the knobs are parsed, the clip read through Ltx2ReadFrameDirectory, encoded, conformed, and the mask assigned into video.mask in the phase loop. The clip rides vllm_video_params::ref_video (include/vllm.h:912), an ABI field that already existed and that this engine previously only tested for emptiness — no struct grows a field, so sibling rows editing that header concurrently are unaffected. ltx2-gen reaches the same path on its default configuration.

Test entry. The five test_ltx2_video cases start at LoadVideoEngine and Generate, not at Ltx2TemporalRegionMaskVideo. M1 deletes the production call site: that suite goes RED (exit 1) while test_ltx2_retake stays green (exit 0), which is the split .agents/reachability.md predicts and the reason the unit suite is not the proof.

/v1/videos forwards no engine extras today (#928), so the CLI and the C ABI are what this row claims and the HTTP surface is not.

Mutations — three facts each

Restores verified by sha256sum of both product files, not assumed. Exit codes captured directly, never through a pipe. Filters are comma-free and the case count is printed beside every exit code, so a filter that matched nothing cannot read as a pass.

#	mutation	git diff --stat	BUILT	exit (video / retake)	verdict
M1	delete the production call site of the mask	ltx2_video.cpp | 2 --	YES, compile_err=0	1 / 0	DETECTED
M2	never seed video_initial with the encoded clip	ltx2_video.cpp | 2 +-	YES, compile_err=0	0 / 0, then 1 / 0	SURVIVED, then DETECTED
M3	take get_pixel_coords' own causal_fix=false	ltx2_video.cpp | 2 +-	YES, compile_err=0	0 / 0, then 1 / 0	SURVIVED, then DETECTED
M4	leave the frozen stream at the schedule's scalar sigma	ltx2_video.cpp | 2 +-	YES, compile_err=0	1 / 0	DETECTED
M5	conform truncates only, never pads	ltx2_retake.cpp | 1 +	YES, compile_err=0	0 / 1	DETECTED
M6	containment instead of overlap	ltx2_retake.cpp | 2 +-	YES, compile_err=0	1 / 1	DETECTED
M7	ignore initial_audio_latent is not None	ltx2_retake.cpp | 4 ++--	NO, compile_err=1, then YES compile_err=0	— , then 0 / 1	NOT ESTABLISHED, then DETECTED
M8	drop the !wants_retake guard	ltx2_video.cpp | 2 +-	YES, compile_err=0	1 / 0	DETECTED
M9	the 8k+1 refusal rounds the snapped value UP	ltx2_retake.cpp | 2 +-	YES, compile_err=0	1 / 1	DETECTED
M10	conform pads once instead of per channel	ltx2_retake.cpp | 4 ++++	YES, compile_err=0	0 / 1	DETECTED

M2 and M3 survived on the first run. Both were claims made in a comment and observed by nothing. M2 is the sharper: the suite recorded retake_latent_absmax, which observes the encode, and inferred the latent reached the phase — a build that reads the clip, encodes it, records the digest and starts from zeros passed everything and rendered a clip of the right length with the right mask. The repair renders two different sources at the same seed and window and requires the pixels to differ. M3's unit case was already red under the flip, but it calls the function directly and says nothing about what the production call site passes; the end-to-end window [0.05, 0.10) selects the same latent frame under both polarities, so a second window [0.30, 0.40) was added where they disagree. M7 failed to build (has_audio_latent unused under -Werror) and is recorded as establishing nothing, then re-run with the parameter referenced.

Merge hazard

ltx2_video.cpp:363-364 carries the READER ANCHORS list, derived and string-compared by test_ltx2_video. This branch shifted it by one (779 … → 780 …) and re-derived it at the final tree. A clean git merge will not warn if another branch shifts it again — re-derive after any merge.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]