LTX chain hardening: pinned-memory allocation, onload idempotency, connector_embeds, pinned Gemma3 - #41
Merged
Merged
Conversation
pin_staging defaulting on (#37) made every pipeline load print two libtorch deprecation warnings per pinned tensor - thousands of lines - because this torch build's Tensor$pin_memory() binding requires the deprecated device argument (omitting it errors with 'Expected a torch_device'). torch_empty_strided is the one creation op exposing pin_memory, so .ltx23_pin_host allocates pinned storage with contiguous strides and copies in; the noisy device-arg call remains as a suppressWarnings fallback for builds where that path fails. GPU validation (is-it-actually-pinned via transfer-rate check) pending - the card is occupied by a live render session; code-only change, nothing installed.
…r-call connectors phase
TroyHernandez
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four related fixes/seams that came out of chained-generation benchmarking at 960x960. Together they take a 3-chunk chained track from ~490 s to ~418 s (~15%) and make resident-transformer configurations actually work.
1. Pin host memory via
torch_empty_strided, not the deprecated overloadThis torch build's
Tensor$pin_memory()binding requires the deprecateddeviceargument (omitting it errors; passing it prints two libtorch warnings per tensor — thousands of lines per pipeline load)..ltx23_pin_host()now allocates page-locked buffers withtorch_empty_strided(pin_memory = TRUE)and copies in; the noisy path survives only as a fallback. Validated at 25.1 GB/s H2D (the card's DMA ceiling) with clean stderr; 931 GPU-visible suite results green.2. Onload idempotency probes the staging pair
The
resident=onload check readmodule$parameters[[1]]inside a tryCatch whose failure mode was a silent re-onload — for module classes that don't expose$parameters(the NF4 transformer), every "resident" call re-transferred 11 GB over itself and fragmented the allocator pool. The probe now reads the staging pair's live tensor, which exists for every pinned component.3.
connector_embeds=ontxt2vid_ltx2()The per-call connectors phase moves the raw Gemma3 hidden-state stack to the GPU — designed for a card the transformer hasn't boarded yet, and the cause of a geometry-independent OOM (identical 14.97 GiB signature at 928 and 960) in every resident-chain attempt. The prompt is constant across a chained track, so precomputed connector outputs (~9 MB) can now be passed once and reused; benchmarked, this cuts the denoise-phase peak by ~2.5 GiB at constant resolution.
4. Pinned Gemma3 staging
The Gemma3 loaders gain
pin=(default: thediffuseR.pin_stagingoption). A CPU-resident encoder is page-locked once, andencode_with_gemma3()swaps it to the compute device per encode (~0.3 s on at DMA rate, pointer-swap off) instead of holding ~8 GB of VRAM or reloading (~15 s). GPU-validated: staging attaches, weights return to CPU after each encode.Benchmark context
Four instrumented runs (two per chain variant, alternating order, all phases fenced with
cuda_synchronize): resident and cycled transformer configurations are within noise of each other once pinned re-onload (0.5 s for 11 GB) is accounted for — the win comes from in-memory conditioning,trim_frames, and the connector bypass, not residency.