Add IC-LoRA Ingredients remix mode (2-8 reference stills) to local LTX video gen#3151
Merged
Conversation
…tch (#3112) Ingredients recomposes a scene from 2-8 reference stills, so it needs both a new input surface and a weight-provisioning path the existing modes didn't. The weight is the hard part. Its first-party repo (Lightricks) is gated, and the un-gated mirror that carries it is DeepBeepMeep/LTX-2 — a ~708 GB aggregate of every LTX weight. The existing download helper only enumerated and pulled whole repos, and resolveIcLoraWeight's bare-repo-id fallback hands the id to ICLoraPipeline._resolve_lora_path, which snapshot_downloads it. So: - hf_download_repo.py gains `--only <filename>` (repeatable), which skips repo enumeration entirely rather than filtering an enumerated list — the absence of a list means nothing downstream can widen it into a snapshot. - startHfDownloadStream gains `fallbacks` (first-success-wins, official then mirror) plus a `cachedFile` predicate, because the repo-wide cache verdict is meaningless for an aggregate mirror: it reports cached off any unrelated resident weight, which would skip the file the user actually asked for. - A `requiresPreDownload` spec suppresses the repo-id fallback outright, so an un-fetched Ingredients render 400s with "download the weight first" instead of starting an unbounded pull mid-render. Its status/repair paths are scoped to the one file rather than walking the mirror's whole snapshot. referenceKind: 'image' selects the second input surface. References resolve from the gallery like `keyframes` do, on their own `icReferenceImageFiles` field — the video fields are left alone, and cross-kind requests are rejected rather than dropped (an image-kind weight fed a clip returns plausible garbage, not an error). The pipeline's reference channel is a video encoder end to end (ffprobe + a VAE needing 1+8k frames), so each still is materialized into a 9-frame clip at render resolution first. The reference count is a weight contract enforced at all three layers from one registry bound, and the metadata behind the entry was measured, not assumed: reference_downscale_factor=1 and the 1,308,778,338-byte size come from a Range read of the weight's safetensors header and HF's x-linked-size.
…ld still be bypassed - findCachedRepoFile: resolve one known file in a snapshot WITHOUT walking it. icLoraWeights and sseDownload were both calling inspectModelCache on the aggregate mirror, which recursively stats every weight in the snapshot — so status, render-time resolution, download and repair could each enumerate hundreds of GB of unrelated weights. The single-file path now never asks a repo-wide question at all: sseDownload skips the walk entirely and defers to the caller's per-file predicate (defaulting to "not cached", so a caller that forgets it re-fetches instead of inheriting a wrong `cached: true`). - Still-materialization no longer leaks temp clips on partial failure. Promise.all rejected at the first bad encode while siblings were still in flight, so their files landed after cleanup had already run. Every target path is now registered before any encode starts and all encodes settle before we decide. - generate_ltx2.py's --ic-min/max-references lose their 1/1 defaults and are required for ic mode. The default let a direct `--ic-mode ingredients --ic-reference one.mp4` pass validation and render garbage — confirmed against the real weight: that invocation now exits with the contract message instead. - The Zod ceiling for both IC reference arrays derives from the registry (MAX_IC_REFERENCES) rather than a second hardcoded 8, which would have 422'd a raised per-weight maximum before the registry assertion could speak.
atomantic
force-pushed
the
claim/issue-3112
branch
from
July 27, 2026 09:20
52575cd to
7d654bb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 3 of #3100. Ingredients recomposes a scene from 2-8 reference stills, which needs both a new input surface and a weight-provisioning path the existing single-clip modes didn't have.
Weight provisioning (the hard part). The first-party
Lightricks/LTX-2.3-22b-IC-LoRA-Ingredientsrepo is gated, and the un-gated mirror that carries the same file isDeepBeepMeep/LTX-2— a ~708 GB aggregate of every LTX weight. The existing helper only enumerated-and-pulled whole repos, andresolveIcLoraWeight's bare-repo-id fallback hands the id toICLoraPipeline._resolve_lora_path, whichsnapshot_downloads it.scripts/hf_download_repo.pygains--only <filename>(repeatable). It skips repo enumeration entirely rather than filtering an enumerated list — the absence of a list is what makes it impossible for anything downstream to widen the pull into a snapshot. Mutually exclusive with--ignore.startHfDownloadStreamgainsfallbacks(ordered{repo, only}, first success wins) plus acachedFilepredicate. A single-file pull never asks a repo-wide question at all:inspectModelCacherecursively stats every weight in the snapshot, and an aggregate mirror reportscachedoff any unrelated resident file — so it would both cost a hundreds-of-GB walk and then skip the file the user actually asked for. A retried failure is downgraded to a log so the UI doesn't flash a red "gated repo" banner for something the mirror then delivers; a user cancel stops rather than rolling to the next source.findCachedRepoFile(repo, filename)inhfCache.jsresolves one known file in the newest snapshot without walking it, andrepairCachedFile(path)is the matching single-file repair (unlinking the blob behind the symlink too, sincehf_hub_downloadtrusts the cached etag rather than the content). Both/models/statusandrepairroute through them for a mirrored spec, and mirror repos are excluded fromicLoraRepos()so an unscoped integrity scan can't hash hundreds of GB of unrelated weights.requiresPreDownloadspec suppresses the repo-id fallback outright — an un-fetched Ingredients render 400s with "download the weight first" instead of starting an unbounded pull mid-render.Image references.
referenceKind: 'image'selects the second input surface. References resolve from the gallery exactly likekeyframesdo, on their ownicReferenceImageFilesfield — thevideo/*upload andicReferenceVideoIdsare left alone, and cross-kind requests are rejected rather than dropped (an image-kind weight fed a clip returns plausible-looking garbage, not an error). The pipeline's reference channel is a video encoder end to end (ffprobe + a VAE needing a 1+8k frame count), so each still is materialized into a 9-frame clip at render resolution first; every target path is registered before any encode starts so a partial ffmpeg failure can't leak clips that land after cleanup ran.Client. The panel branches on
referenceKind: a 2-8 row gallery list modeled onKeyframePanel(add/remove, inline previews, liven/8counter) for image-kind, the unchanged single upload/history pair otherwise. Bounds are read off the spec so a future weight with different limits needs no component change. Submit is blocked on filled-row count, and unlike a clip an image reference is re-derivable from its basename, so an/activeresume repopulates the picker instead of demanding a re-pick.Registry metadata was measured, not assumed:
reference_downscale_factor=1came from an HTTP Range read of the weight's safetensors header (no 1.3 GB download needed to learn it), andsizeBytes: 1_308_778_338from HF'sx-linked-size. Copying Control's factor of 2 would have rejected valid odd-multiple resolutions for nothing.The 2-8 count is a weight contract (a wrong count yields plausible garbage, not an error) enforced at all three layers — route,
icLoraArgs, and the Python helper — from one registry bound. The Zod ceiling derives from the registry rather than a second literal, and--ic-min/max-referencescarry no defaults and are required for--mode ic: a 1/1 default let a direct--ic-mode ingredients --ic-reference one.mp4pass validation and render garbage.Test plan
Validated on a real render (M5 Max, ltx-2-mlx 0.14.8): two stills →
--mode ic --ic-mode ingredientswith the real 1.3 GB weight → a coherent 24-frame clip at the expected half-res 352x224 for--ic-skip-stage-2. Content follows the prompt (a waving mascot in front of a storefront), not noise. Re-ran after the review fixes; still renders. The bypass those fixes closed was confirmed against the real weight too: omitting the bound flags now exits with the contract message instead of rendering.Weight provisioning verified against live HF, no snapshot at any point:
--only README.mdagainstDeepBeepMeep/LTX-2pulled 1069 bytes — proving the 708 GB repo is never enumerated or snapshotted.USER_ERROR:gated_repoand downloaded nothing; the mirror then delivered exactly one file, 1.2 GB resident, byte size matching the registry's1_308_778_338.findCachedIcLoraWeight/resolveIcLoraWeightresolve it out of the real HF cache layout and reportresolvedRepo: DeepBeepMeep/LTX-2;icLoraRepos()confirmed to exclude the mirror.Automated:
cd server && npm test— full suite. The IC-touching suites are green:lib/icLoraWeights(24),lib/hfCache(24, 8 new),lib/sseDownload(18, 12 new),routes/videoGen(99, 11 new),services/videoGen/local(88, 8 new). 58 files fail on this branch, but 56 fail identically on unmodifiedmain(local Postgres not provisioned for the file-backend suites); the 2 extra (lib/dataRoot,services/installState) were confirmed to fail with the branch stashed — they're git-worktree path artifacts, not this change.cd client && npm test— 470 files / 5233 tests, all passing.python3 scripts/generate_ltx2_ic_lora_test.py— 10 new tests, all passing: the 2-8 bounds, the required-bounds contract, the "exactly 1" phrasing for single-reference weights, rejection of nonsense bounds, and that the IC weight rideslora_pathswhile user LoRAs ride_pending_loras(the assertion that an Ingredients x Character stack composes rather than one displacing the other).Reviewed with
codexover two rounds: round 1 raised four correctness findings (the mirror-snapshot walk, the temp-clip leak, the Python bounds default, and the hardcoded schema ceiling), all fixed in52575cdwith tests; round 2 reported no remaining issues.Closes #3112