Skip to content

Add IC-LoRA Ingredients remix mode (2-8 reference stills) to local LTX video gen#3151

Merged
atomantic merged 2 commits into
mainfrom
claim/issue-3112
Jul 27, 2026
Merged

Add IC-LoRA Ingredients remix mode (2-8 reference stills) to local LTX video gen#3151
atomantic merged 2 commits into
mainfrom
claim/issue-3112

Conversation

@atomantic

@atomantic atomantic commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Summary

Phase 3 of #3100. Ingredients recomposes a scene from 2-8 reference stills, which needs both a new input surface and a weight-provisioning path the existing single-clip modes didn't have.

Weight provisioning (the hard part). The first-party Lightricks/LTX-2.3-22b-IC-LoRA-Ingredients repo is gated, and the un-gated mirror that carries the same file is DeepBeepMeep/LTX-2 — a ~708 GB aggregate of every LTX weight. The existing helper only enumerated-and-pulled whole repos, and resolveIcLoraWeight's bare-repo-id fallback hands the id to ICLoraPipeline._resolve_lora_path, which snapshot_downloads it.

  • scripts/hf_download_repo.py gains --only <filename> (repeatable). It skips repo enumeration entirely rather than filtering an enumerated list — the absence of a list is what makes it impossible for anything downstream to widen the pull into a snapshot. Mutually exclusive with --ignore.
  • startHfDownloadStream gains fallbacks (ordered {repo, only}, first success wins) plus a cachedFile predicate. A single-file pull never asks a repo-wide question at all: inspectModelCache recursively stats every weight in the snapshot, and an aggregate mirror reports cached off any unrelated resident file — so it would both cost a hundreds-of-GB walk and then skip the file the user actually asked for. A retried failure is downgraded to a log so the UI doesn't flash a red "gated repo" banner for something the mirror then delivers; a user cancel stops rather than rolling to the next source.
  • findCachedRepoFile(repo, filename) in hfCache.js resolves one known file in the newest snapshot without walking it, and repairCachedFile(path) is the matching single-file repair (unlinking the blob behind the symlink too, since hf_hub_download trusts the cached etag rather than the content). Both /models/status and repair route through them for a mirrored spec, and mirror repos are excluded from icLoraRepos() so an unscoped integrity scan can't hash hundreds of GB of unrelated weights.
  • A requiresPreDownload spec suppresses the repo-id fallback outright — an un-fetched Ingredients render 400s with "download the weight first" instead of starting an unbounded pull mid-render.

Image references. referenceKind: 'image' selects the second input surface. References resolve from the gallery exactly like keyframes do, on their own icReferenceImageFiles field — the video/* upload and icReferenceVideoIds are left alone, and cross-kind requests are rejected rather than dropped (an image-kind weight fed a clip returns plausible-looking garbage, not an error). The pipeline's reference channel is a video encoder end to end (ffprobe + a VAE needing a 1+8k frame count), so each still is materialized into a 9-frame clip at render resolution first; every target path is registered before any encode starts so a partial ffmpeg failure can't leak clips that land after cleanup ran.

Client. The panel branches on referenceKind: a 2-8 row gallery list modeled on KeyframePanel (add/remove, inline previews, live n/8 counter) for image-kind, the unchanged single upload/history pair otherwise. Bounds are read off the spec so a future weight with different limits needs no component change. Submit is blocked on filled-row count, and unlike a clip an image reference is re-derivable from its basename, so an /active resume repopulates the picker instead of demanding a re-pick.

Registry metadata was measured, not assumed: reference_downscale_factor=1 came from an HTTP Range read of the weight's safetensors header (no 1.3 GB download needed to learn it), and sizeBytes: 1_308_778_338 from HF's x-linked-size. Copying Control's factor of 2 would have rejected valid odd-multiple resolutions for nothing.

The 2-8 count is a weight contract (a wrong count yields plausible garbage, not an error) enforced at all three layers — route, icLoraArgs, and the Python helper — from one registry bound. The Zod ceiling derives from the registry rather than a second literal, and --ic-min/max-references carry no defaults and are required for --mode ic: a 1/1 default let a direct --ic-mode ingredients --ic-reference one.mp4 pass validation and render garbage.

Test plan

Validated on a real render (M5 Max, ltx-2-mlx 0.14.8): two stills → --mode ic --ic-mode ingredients with the real 1.3 GB weight → a coherent 24-frame clip at the expected half-res 352x224 for --ic-skip-stage-2. Content follows the prompt (a waving mascot in front of a storefront), not noise. Re-ran after the review fixes; still renders. The bypass those fixes closed was confirmed against the real weight too: omitting the bound flags now exits with the contract message instead of rendering.

Weight provisioning verified against live HF, no snapshot at any point:

  • --only README.md against DeepBeepMeep/LTX-2 pulled 1069 bytes — proving the 708 GB repo is never enumerated or snapshotted.
  • The official repo attempt failed with the expected typed USER_ERROR:gated_repo and downloaded nothing; the mirror then delivered exactly one file, 1.2 GB resident, byte size matching the registry's 1_308_778_338.
  • findCachedIcLoraWeight / resolveIcLoraWeight resolve it out of the real HF cache layout and report resolvedRepo: DeepBeepMeep/LTX-2; icLoraRepos() confirmed to exclude the mirror.

Automated:

  • cd server && npm test — full suite. The IC-touching suites are green: lib/icLoraWeights (24), lib/hfCache (24, 8 new), lib/sseDownload (18, 12 new), routes/videoGen (99, 11 new), services/videoGen/local (88, 8 new). 58 files fail on this branch, but 56 fail identically on unmodified main (local Postgres not provisioned for the file-backend suites); the 2 extra (lib/dataRoot, services/installState) were confirmed to fail with the branch stashed — they're git-worktree path artifacts, not this change.
  • cd client && npm test470 files / 5233 tests, all passing.
  • python3 scripts/generate_ltx2_ic_lora_test.py — 10 new tests, all passing: the 2-8 bounds, the required-bounds contract, the "exactly 1" phrasing for single-reference weights, rejection of nonsense bounds, and that the IC weight rides lora_paths while user LoRAs ride _pending_loras (the assertion that an Ingredients x Character stack composes rather than one displacing the other).

Reviewed with codex over two rounds: round 1 raised four correctness findings (the mirror-snapshot walk, the temp-clip leak, the Python bounds default, and the hardcoded schema ceiling), all fixed in 52575cd with tests; round 2 reported no remaining issues.

Closes #3112

…tch (#3112)

Ingredients recomposes a scene from 2-8 reference stills, so it needs both a
new input surface and a weight-provisioning path the existing modes didn't.

The weight is the hard part. Its first-party repo (Lightricks) is gated, and
the un-gated mirror that carries it is DeepBeepMeep/LTX-2 — a ~708 GB
aggregate of every LTX weight. The existing download helper only enumerated
and pulled whole repos, and resolveIcLoraWeight's bare-repo-id fallback hands
the id to ICLoraPipeline._resolve_lora_path, which snapshot_downloads it. So:

- hf_download_repo.py gains `--only <filename>` (repeatable), which skips
  repo enumeration entirely rather than filtering an enumerated list — the
  absence of a list means nothing downstream can widen it into a snapshot.
- startHfDownloadStream gains `fallbacks` (first-success-wins, official then
  mirror) plus a `cachedFile` predicate, because the repo-wide cache verdict
  is meaningless for an aggregate mirror: it reports cached off any unrelated
  resident weight, which would skip the file the user actually asked for.
- A `requiresPreDownload` spec suppresses the repo-id fallback outright, so an
  un-fetched Ingredients render 400s with "download the weight first" instead
  of starting an unbounded pull mid-render. Its status/repair paths are scoped
  to the one file rather than walking the mirror's whole snapshot.

referenceKind: 'image' selects the second input surface. References resolve
from the gallery like `keyframes` do, on their own `icReferenceImageFiles`
field — the video fields are left alone, and cross-kind requests are rejected
rather than dropped (an image-kind weight fed a clip returns plausible
garbage, not an error). The pipeline's reference channel is a video encoder
end to end (ffprobe + a VAE needing 1+8k frames), so each still is
materialized into a 9-frame clip at render resolution first.

The reference count is a weight contract enforced at all three layers from one
registry bound, and the metadata behind the entry was measured, not assumed:
reference_downscale_factor=1 and the 1,308,778,338-byte size come from a Range
read of the weight's safetensors header and HF's x-linked-size.
…ld still be bypassed

- findCachedRepoFile: resolve one known file in a snapshot WITHOUT walking it.
  icLoraWeights and sseDownload were both calling inspectModelCache on the
  aggregate mirror, which recursively stats every weight in the snapshot — so
  status, render-time resolution, download and repair could each enumerate
  hundreds of GB of unrelated weights. The single-file path now never asks a
  repo-wide question at all: sseDownload skips the walk entirely and defers to
  the caller's per-file predicate (defaulting to "not cached", so a caller that
  forgets it re-fetches instead of inheriting a wrong `cached: true`).

- Still-materialization no longer leaks temp clips on partial failure. Promise.all
  rejected at the first bad encode while siblings were still in flight, so their
  files landed after cleanup had already run. Every target path is now registered
  before any encode starts and all encodes settle before we decide.

- generate_ltx2.py's --ic-min/max-references lose their 1/1 defaults and are
  required for ic mode. The default let a direct `--ic-mode ingredients
  --ic-reference one.mp4` pass validation and render garbage — confirmed against
  the real weight: that invocation now exits with the contract message instead.

- The Zod ceiling for both IC reference arrays derives from the registry
  (MAX_IC_REFERENCES) rather than a second hardcoded 8, which would have 422'd a
  raised per-weight maximum before the registry assertion could speak.
@atomantic
atomantic merged commit 138603a into main Jul 27, 2026
6 checks passed
@atomantic
atomantic deleted the claim/issue-3112 branch July 27, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add IC-LoRA Ingredients remix mode (2-8 reference images) to local LTX video gen

1 participant