diff --git a/README.md b/README.md index b6978ae..af26c96 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ Inline Studio is a free, open-source app for **AI filmmaking on a node canvas**, - **Non-destructive by default** - every render is kept as a versioned take; generating again adds one, nothing is overwritten. - **Local diffusion generation engine** - the built-in Inline Core engine runs popular diffusion models locally, on your own GPU, from a single model file, no external server. Currently supported: **Z-Image Turbo**, **Krea 2** (RAW + Turbo), **FLUX.2**, and **MiniMax H3** for video with sound. -- **Train LoRAs locally** - the Trainer canvas fine-tunes Z-Image, Krea 2, FLUX.2 or MiniMax H3 on your own images, on your own GPU. With a 4-bit base, Krea 2 trains at 512px inside about 12GB, so a 16GB card can train a LoRA for a 26GB model. See [LoRA training](#lora-training). +- **Train LoRAs locally** - the Trainer canvas fine-tunes Z-Image, Krea 2, FLUX.2 or MiniMax H3 on your own images, on your own GPU. H3 also trains on short video clips, so a LoRA can learn motion and not just look. With a 4-bit base, Krea 2 trains at 512px inside about 12GB, so a 16GB card can train a LoRA for a 26GB model. See [LoRA training](#lora-training). - **Hosted models via API Nodes** - reach for closed models with no GPU and no setup for instant creative range; see [API Nodes](#api-nodes). - **Mix both in the same film** - Inline Studio handles everything around the render: exploring options, keeping what works, and shaping a repeatable process you can iterate on and share. diff --git a/TRAINING.md b/TRAINING.md index 9564096..b487d98 100644 --- a/TRAINING.md +++ b/TRAINING.md @@ -12,7 +12,7 @@ resolution. **Contents:** [The graph](#the-graph) · [Datasets and outputs](#datasets-and-outputs) · [Stop and resume](#stop-and-resume) · [Trigger words](#trigger-words) · [Architecture and base model modes](#architecture-and-base-model-modes) · [Install](#install) · -[**Benchmark results**](#benchmark-results) · +[Training on clips](#training-on-clips) · [**Benchmark results**](#benchmark-results) · [Dataset and adapter options](#dataset-and-adapter-options) · [Base precision](#base-precision) ## The graph @@ -57,7 +57,7 @@ The Trainer's Adjust panel picks the **architecture** first (Z-Image, Krea 2, FL **MiniMax H3** is the video model, and it trains on **still images**: - **FL2VA** is the only base, and it is undistilled, so there is no adapter and nothing to drift. Put `minimax_h3_fl2va_bf16.safetensors` in `models/diffusion_models/`, train on stills, then wire the LoRA into any of the four H3 nodes. It loads on the Reference to Video node too, which uses a different checkpoint file: the two partitions are the same architecture. -- **What it learns** is appearance - look, style, character, lighting. It does not learn motion or sound, because it never sees any. This is how image LoRAs for video models are normally trained, and it is the same thing every other H3 trainer does today. +- **Stills or short clips.** Drop images and it learns appearance: look, style, character, lighting. Drop video and it learns motion too. Sound is never learned either way, because the audio rows are empty. See [Training on clips](#training-on-clips). - **The base is 4-bit, always.** H3 is 40GB after the AdaLN factorisation and 11.7GB after quantisation, so full precision is refused rather than offered and then failing. There is no base-precision control for H3 for the same reason. - **A 24GB card is comfortable and a 16GB card works, slowly.** The run encodes latents and captions in two passes that never overlap, because H3's fp32 video VAE and its 32B conditioner cannot be resident together. On a card that holds the conditioner it peaks at 20.6GB; on one that does not, the conditioner runs on the CPU and the peak drops to 12.7GB while a step goes from 0.6s to 16s. Either way there is about seven minutes of startup, and 64GB of system RAM for the smaller card. See [Benchmark results](#benchmark-results) for the split. The download is about 124GB before any of that. @@ -66,6 +66,42 @@ The Trainer's Adjust panel picks the **architecture** first (Z-Image, Krea 2, FL - **Turbo + training adapter** fuses a de-distillation adapter into the base for the duration of training and drops it when the LoRA is saved, which preserves the 8-step speed. Put [ostris/zimage_turbo_training_adapter](https://huggingface.co/ostris/zimage_turbo_training_adapter) in `models/loras/`; any filename containing `adapter` is detected automatically, or point `INLINE_ZIMAGE_TRAIN_ADAPTER` at a specific file. Keep runs short, since the adapter slows the breakdown rather than preventing it. - **De-Turbo** trains without an adapter and needs no extra download. +## Training on clips + +The H3 trainer takes video as well as stills. Drop clips into a dataset the same way, set **Clip +length** in the Adjust panel, and each clip trains as a short piece of motion rather than a frame. +Mixed datasets are fine: a still is simply a one-frame clip. + +**It costs no extra VRAM.** Measured on an L4, every clip length peaks at the same 20.4GB as a +still, because the high-water mark is the caption pass rather than the training: + +| Clip length | Frames | Latent frames | Packed rows at 512px | Peak VRAM | +| ----------- | ------ | ------------- | -------------------- | --------- | +| still | 1 | 1 | 293 | 20.55GB | +| 0.92s | 22 | 7 | 1,832 | 20.4GB | +| 1.6s | 39 | 12 | 3,112 | 20.4GB | +| 4.5s | 107 | 32 | 8,232 | 20.4GB | + +Rows are what a longer clip actually buys you, and they cost time rather than memory. That only +holds while the conditioner is resident; on a card too small for it the peak is the training phase +instead, and a long clip will push that up. + +**Lengths snap to H3's frame grid.** The VAE encodes `17n + 5` frames at 24fps, so a request lands +on the nearest grid point at or below it. The floor is a whole chunk plus the five-frame head: 22 +frames, **0.92 seconds**. Asking for less rounds up rather than being refused, because the VAE has +no way to encode a shorter clip. + +**Each clip is trimmed from its start, once.** The window is fixed at precache time so every clip is +encoded exactly once. Sampling a different window each step would mean re-encoding through the VAE +every step, which is the thing the precache exists to avoid. A clip shorter than the grid floor is +refused by name rather than silently padded. + +**Captions work the same.** A clip is auto-captioned from its middle frame, which describes the shot +better than the first frame usually does. Write them by hand if you would rather. + +Audio is not trained. H3 generates video and its soundtrack jointly, but the trainer packs zero +audio rows, so an adapter changes what a clip looks like and never what it sounds like. + ## Install If you installed with `--extra all` from [Get Started](README.md#get-started), the trainer is already set up - nothing more to do. To add it to a leaner install, its dependencies (PEFT, 8-bit Adam, the captioner) sit behind the `training` extra: @@ -109,6 +145,7 @@ The LoRA a run produces lands in `models/loras/` and shows up in the LoRA loader | MiniMax H3 | FL2VA | 512 | **4-bit** | **20.6GB** | **12.7GB** | | MiniMax H3 | FL2VA | 768 | **4-bit** | **20.6GB** | not measured | | MiniMax H3 | FL2VA | 1024 | **4-bit** | **20.6GB** | not measured | +| MiniMax H3 | FL2VA, clips | 512 | **4-bit** | **20.4GB** | not measured | **MiniMax H3 costs less on a smaller card, which is not a typo.** The run has three phases that never overlap, and the tallest is not the one doing the learning: @@ -124,13 +161,14 @@ On a card too small for the conditioner it never goes there at all, so the peak **The bill arrives as time instead.** The conditioner runs on the CPU, and bitsandbytes only quantises on the move to CUDA, so it runs unquantised: -| | L40S (46GB) | T4 (16GB, 64GB RAM) | -| ----------------------- | ----------- | ------------------- | -| Peak VRAM | 20.6GB | 12.7GB | -| Seconds per step | 0.63 | 16.2 | -| Caption pass, 26 images | 1 min | 19 min | +| | L40S (46GB) | L4 (24GB) | T4 (16GB, 64GB RAM) | +| ----------------------- | ----------- | --------- | ------------------- | +| Peak VRAM, 512px | 20.6GB | 20.55GB | 12.7GB | +| Seconds per step, 512px | 0.63 | 1.81 | 16.2 | +| Seconds per step, 768px | 0.77 | 2.73 | not measured | +| Caption pass, 26 images | 1 min | 1 min | 19 min | -A 1500-step run is about 16 minutes on the L40S and closer to seven hours on the T4. Some of that is the T4 being a T4, and some is the caption pass being on the wrong processor. +A 1500-step run at 512px is about 16 minutes on the L40S, 45 on the L4, and closer to seven hours on the T4. The L4 holds the conditioner, so it looks like a slower L40S rather than a faster T4: the 9x gap to the T4 is mostly the caption pass being on the wrong processor, not the cards themselves. **It also wants a lot of system RAM.** The unquantised conditioner pages roughly 63GB through the page cache, and on a 64GB machine that sits at 59GB resident, close enough to the edge that the caption pass is the riskiest part of the run. A T4 with only 16GB of RAM has room in neither VRAM nor RAM and is refused before anything loads, because a host-RAM overrun is killed by the kernel rather than raising. diff --git a/core/pyproject.toml b/core/pyproject.toml index 09739dd..649434f 100644 --- a/core/pyproject.toml +++ b/core/pyproject.toml @@ -1,7 +1,7 @@ [project] # PyPI name; the import package is `inline_core` (src/inline_core). name = "inline-core" -version = "1.2.63" +version = "1.2.64" description = "The generation engine behind Inline Studio." readme = "README.md" license = "GPL-3.0-or-later" @@ -82,6 +82,8 @@ all = [ "accelerate>=0.30", "safetensors>=0.4", "torchao>=0.14", + # Clip decode for MiniMax H3 LoRA training, and H3's reference node. + "av>=12", "scipy>=1.11", "huggingface_hub>=0.23", "controlnet-aux>=0.0.7", diff --git a/core/src/inline_core/server/__main__.py b/core/src/inline_core/server/__main__.py index 2260a8c..6d35791 100644 --- a/core/src/inline_core/server/__main__.py +++ b/core/src/inline_core/server/__main__.py @@ -79,6 +79,9 @@ def main() -> None: studio_config.workspace_dir(), default_core_url=studio_config.DEFAULT_CORE_URL, ) + # Reopen whatever was open before the restart, so a browser tab left open across it keeps + # working instead of failing every call with "No project is open." + store.restore_last_project() app = create_app( registry=registry, cache=InMemoryCache(), diff --git a/core/src/inline_core/studio/handlers.py b/core/src/inline_core/studio/handlers.py index c8c861b..0be876d 100644 --- a/core/src/inline_core/studio/handlers.py +++ b/core/src/inline_core/studio/handlers.py @@ -88,6 +88,7 @@ def fn(*_args: Any) -> Any: reg("project:openZip", lambda: None) reg("project:listRecent", store.list_recent) reg("project:current", store.current_project) + reg("project:close", store.close_project) reg("project:mediaDirs", store.media_dirs) reg("project:export", lambda _path: None) # zip export: pending (see plan) reg("dialog:pickDirectory", lambda *_: str(cfg.workspace_dir())) @@ -264,6 +265,7 @@ def cancel_generation(frame_id: str | None = None) -> None: reg("training:createDataset", lambda inp: training.create_dataset(inp)) reg("training:listItems", lambda did: training.list_items(did)) reg("training:addItems", lambda did, aids: training.add_items(did, aids)) + reg("training:addFromPath", lambda did, path: training.add_from_path(did, path)) reg("training:removeItem", lambda iid: training.remove_item(iid)) reg("training:setCaption", lambda iid, cap: training.set_caption(iid, cap)) reg("training:autoCaption", diff --git a/core/src/inline_core/studio/store.py b/core/src/inline_core/studio/store.py index b03d66b..60c01e2 100644 --- a/core/src/inline_core/studio/store.py +++ b/core/src/inline_core/studio/store.py @@ -131,6 +131,7 @@ def create_project(self, name: str, parent_dir: str | None = None) -> dict[str, project = {"id": pid, "name": name, "path": str(folder), "createdAt": now, "updatedAt": now} self._current = project self.record_recent(name, str(folder)) + self._remember_last_project(str(folder)) return project def open_project(self, selected: str) -> dict[str, Any]: @@ -143,6 +144,7 @@ def open_project(self, selected: str) -> dict[str, Any]: project = self._load_project_row(folder) self._current = project self.record_recent(project["name"], str(folder)) + self._remember_last_project(str(folder)) return project def _load_project_row(self, folder: Path) -> dict[str, Any]: @@ -160,7 +162,44 @@ def _load_project_row(self, folder: Path) -> dict[str, Any]: } def current_project(self) -> dict[str, Any] | None: - return self._current + return self._current or self.restore_last_project() + + def close_project(self) -> None: + self.close() + self._current = None + self._remember_last_project(None) + + # --- last opened project ---------------------------------------------------------------------- + # The open project is otherwise only in memory, so restarting Core left a still-open browser tab + # failing every call with "No project is open." Kept in its own file rather than settings.json, + # because _save_settings rewrites that from get_settings() and would drop any key it omits. + + def _last_project_file(self) -> Path: + return self._app_data / "last_project" + + def _remember_last_project(self, path: str | None) -> None: + file = self._last_project_file() + try: + if path: + file.write_text(path, encoding="utf-8") + elif file.exists(): + file.unlink() + except OSError: + pass # never fail an open just because the marker could not be written + + def restore_last_project(self) -> dict[str, Any] | None: + """Reopen the project left open at shutdown. Best-effort: a moved or deleted one is + forgotten and the launcher shows instead.""" + if self._conn is not None: + return self._current + file = self._last_project_file() + if not file.exists(): + return None + try: + return self.open_project(file.read_text(encoding="utf-8").strip()) + except (OSError, ValueError, sqlite3.Error): + self._remember_last_project(None) + return None def media_dirs(self) -> dict[str, str]: if self._folder is None: diff --git a/core/src/inline_core/studio/training.py b/core/src/inline_core/studio/training.py index 45607b0..7a79aaa 100644 --- a/core/src/inline_core/studio/training.py +++ b/core/src/inline_core/studio/training.py @@ -71,6 +71,42 @@ def list_items(self, dataset_id: str) -> list[dict[str, Any]]: def add_items(self, dataset_id: str, asset_ids: list[str]) -> list[dict[str, Any]]: return ts.add_items(self._conn(), dataset_id, asset_ids) + def add_from_path(self, dataset_id: str, path: str) -> list[dict[str, Any]]: + """Import a folder of images and clips into the dataset, captions included. + + The browser cannot hand over a folder, and uploading a clip dataset through it means + pushing gigabytes over HTTP to a server that can already see the disk. Paths come from the + client here the same way ``assets:importPaths`` already accepts them. + """ + from . import assets as ax + + folder = Path(path).expanduser() + if not folder.is_dir(): + raise ValueError(f"Not a folder: {path}") + conn, project = self._conn(), self._store.folder() + media = [ + p + for p in sorted(folder.iterdir()) + if p.is_file() and ax.kind_for_file(str(p)) in ("image", "video") + ] + if not media: + raise ValueError(f"No images or clips in {path}") + + imported = [(p, ax.import_file(conn, project, str(p), None)) for p in media] + added = ts.add_items(conn, dataset_id, [a["id"] for _p, a in imported if a]) + + # `NNNN.txt` beside `NNNN.png` is the caption, the convention the drag-drop path already + # follows. Only newly added items are touched, so re-importing cannot clobber an edit. + by_asset = {item["assetId"]: item for item in added} + for source, asset in imported: + item = by_asset.get(asset["id"]) if asset else None + sidecar = source.with_suffix(".txt") + if item and sidecar.is_file(): + caption = sidecar.read_text(encoding="utf-8").strip() + if caption: + ts.set_caption(conn, item["id"], caption) + return ts.list_items(conn, dataset_id) + def remove_item(self, item_id: str) -> None: ts.remove_item(self._conn(), item_id) diff --git a/core/src/inline_core/training/arch.py b/core/src/inline_core/training/arch.py index 28b056f..b38b06f 100644 --- a/core/src/inline_core/training/arch.py +++ b/core/src/inline_core/training/arch.py @@ -333,6 +333,22 @@ def _h3_forward(transformer: Any, noisy: Any, timestep: Any, item: dict[str, Any } +def clip_frames(arch: TrainingArch, seconds: Any) -> int: + """How many frames of a clip to train on, snapped to the arch's frame grid. + + 1 for an arch with no clip support, which is what a still costs. For H3 the floor is a whole + 17-frame chunk plus the 5-frame head, so a shorter request rounds up to 0.92s rather than being + refused; the VAE has no way to encode less. + """ + if arch.key != MINIMAX_H3: + return 1 + from ..models.minimaxh3.vendor.packing import MINIMAX_H3_FPS + from ..models.minimaxh3.vendor.packing_ref2va import trim_reference_num_frames + + wanted = round(float(seconds) * MINIMAX_H3_FPS) if seconds else 1 + return trim_reference_num_frames(max(1, wanted)) + + def get(key: str | None) -> TrainingArch: """The arch to train. Defaults to Z-Image so a run predating Krea 2 still resumes.""" arch = ARCHS.get(key or Z_IMAGE) diff --git a/core/src/inline_core/training/cache.py b/core/src/inline_core/training/cache.py index 230c95b..a314ece 100644 --- a/core/src/inline_core/training/cache.py +++ b/core/src/inline_core/training/cache.py @@ -7,6 +7,7 @@ from __future__ import annotations +from collections.abc import Callable from typing import Any from . import arch as archs @@ -28,13 +29,21 @@ def build( *, flip: bool = False, dropout: float = 0.0, + clip_frames: int = 1, + on_status: Callable[[str], None] | None = None, ) -> tuple[list[dict[str, Any]], dict[str, Any] | None, float]: - """Return ``(items, unconditional, shift)``, all as CPU tensors, with the encoders freed.""" + """Return ``(items, unconditional, shift)``, all as CPU tensors, with the encoders freed. + + ``on_status`` reports phase progress to the caller, which forwards it over the JSON protocol. + Precaching a large dataset takes minutes, and a logger call would be dropped here: the trainer + subprocess configures no logging handler, so anything below WARNING goes nowhere. + """ if arch == archs.MINIMAX_H3: from . import h3 items, unconditional = h3.precache( - dataset_dir, models_dir, device, dtype, resolution, flip, dropout > 0 + dataset_dir, models_dir, device, dtype, resolution, flip, dropout > 0, clip_frames, + on_status=on_status, ) return items, unconditional, _H3_SHIFT diff --git a/core/src/inline_core/training/caption.py b/core/src/inline_core/training/caption.py index 419dfdb..7827997 100644 --- a/core/src/inline_core/training/caption.py +++ b/core/src/inline_core/training/caption.py @@ -105,13 +105,30 @@ def _load_with_fallback(model_id: str) -> tuple[Any, Any, Any]: raise first from None +def _open(path: str) -> Any: + """The frame to caption. A clip is captioned from its middle frame, which is more + representative than the first and stops PIL raising on a container it cannot read.""" + from pathlib import Path + + from PIL import Image + + from . import dataset as ds + + if not ds.is_video(Path(path)): + return Image.open(path).convert("RGB") + + from ..models.minimaxh3.vendor.packing_ref2va import decode_reference_video + + frames, _fps, _audio = decode_reference_video(path) + return Image.fromarray(frames[len(frames) // 2]).convert("RGB") + + def _caption_one(model: Any, processor: Any, device: str, path: str) -> str: """One caption. Handles both shapes: task-token models (Florence-2, which post-processes a tagged string) and plain image-captioning models (BLIP), which just decode the output.""" import torch - from PIL import Image - image = Image.open(path).convert("RGB") + image = _open(path) task_style = hasattr(processor, "post_process_generation") inputs = ( processor(text=_TASK, images=image, return_tensors="pt") diff --git a/core/src/inline_core/training/dataset.py b/core/src/inline_core/training/dataset.py index 4e74615..0c7c5a4 100644 --- a/core/src/inline_core/training/dataset.py +++ b/core/src/inline_core/training/dataset.py @@ -17,15 +17,25 @@ _IMAGE_SUFFIXES = (".png", ".jpg", ".jpeg", ".webp", ".bmp") +#: Only the video archs pass these to ``_pairs``. An image arch handed a clip would reach PIL and +#: raise, so the default stays images and each caller opts in. +_VIDEO_SUFFIXES = (".mp4", ".mov", ".webm", ".mkv", ".avi") -def _pairs(dataset_dir: Path) -> list[tuple[Path, str]]: + +def is_video(path: Path) -> bool: + return path.suffix.lower() in _VIDEO_SUFFIXES + + +def _pairs( + dataset_dir: Path, suffixes: tuple[str, ...] = _IMAGE_SUFFIXES +) -> list[tuple[Path, str]]: out: list[tuple[Path, str]] = [] - for img in sorted(dataset_dir.iterdir()): - if img.suffix.lower() not in _IMAGE_SUFFIXES: + for media in sorted(dataset_dir.iterdir()): + if media.suffix.lower() not in suffixes: continue - caption_file = img.with_suffix(".txt") + caption_file = media.with_suffix(".txt") caption = caption_file.read_text(encoding="utf-8").strip() if caption_file.exists() else "" - out.append((img, caption)) + out.append((media, caption)) return out diff --git a/core/src/inline_core/training/h3.py b/core/src/inline_core/training/h3.py index 3a53b39..a32c55c 100644 --- a/core/src/inline_core/training/h3.py +++ b/core/src/inline_core/training/h3.py @@ -14,6 +14,7 @@ import gc import logging +from collections.abc import Callable from pathlib import Path from typing import Any @@ -29,6 +30,10 @@ #: caption dropout and an image whose ``.txt`` is missing. _EMPTY_CAPTION = " " +#: H3's fixed frame rate, and the shortest clip its video VAE encodes (the first ``17n + 5``). +_H3_FPS = 24 +_MIN_CLIP_FRAMES = 22 + def precache( dataset_dir: str, @@ -38,38 +43,55 @@ def precache( resolution: int, flip: bool, want_unconditional: bool, + clip_frames: int = 1, + on_status: Callable[[str], None] | None = None, ) -> tuple[list[dict[str, Any]], dict[str, Any] | None]: """Every image as a latent and every caption as conditioning, as CPU tensors.""" from . import dataset as ds - pairs = ds._pairs(Path(dataset_dir)) + say = on_status or (lambda _text: None) + pairs = ds._pairs(Path(dataset_dir), ds._IMAGE_SUFFIXES + ds._VIDEO_SUFFIXES) if not pairs: raise RuntimeError("The exported dataset is empty.") root = Path(models_dir) - latents = _encode_pixels(root, pairs, device, resolution, flip) - captions = [caption for _img, caption in pairs for _ in ((False, True) if flip else (False,))] + # Only the clips that survived encoding carry captions, or every caption after the first skip + # would be paired with the wrong latent. + latents, kept = _encode_pixels(root, pairs, device, resolution, flip, clip_frames, say) + if not kept: + raise RuntimeError( + f"None of the {len(pairs)} dataset items could be encoded. For clips, each must be at " + f"least {_MIN_CLIP_FRAMES} frames at {_H3_FPS}fps " + f"({_MIN_CLIP_FRAMES / _H3_FPS:.2f}s)." + ) + captions = [caption for _img, caption in kept for _ in ((False, True) if flip else (False,))] if want_unconditional: captions.append("") - embeds = _encode_captions(root, captions, device, dtype) + embeds = _encode_captions(root, captions, device, dtype, say) + say(f"cached {len(latents)} latents and {len(embeds)} captions") items = [ {"latent": latent, **_conditioning(embed, tags, latent)} for latent, (embed, tags) in zip(latents, embeds, strict=False) ] - unconditional = None if want_unconditional: + # Dropout swaps a different text length in, which moves every row after it, so the whole + # layout travels with the embedding. It also depends on the latent grid, and a dataset + # mixing stills with clips has more than one, so each item carries its own rather than + # sharing the first item's and mis-sizing every clip. embed, tags = embeds[-1] - # Caption dropout swaps in a different text length, which moves every row after it, so the - # whole layout travels with the embedding rather than just the embedding. - unconditional = _conditioning(embed, tags, latents[0]) - return items, unconditional + for item in items: + item["uncond"] = _conditioning(embed, tags, item["latent"]) + # The per-item copies are what dropout uses; the loop keeps the global slot for the image archs. + return items, None def _encode_pixels( - root: Path, pairs: list[tuple[Path, str]], device: str, resolution: int, flip: bool -) -> list[Any]: - """Pass one: the video VAE, then dropped.""" + root: Path, pairs: list[tuple[Path, str]], device: str, resolution: int, flip: bool, + clip_frames: int = 1, say: Callable[[str], None] = lambda _text: None, +) -> tuple[list[Any], list[tuple[Path, str]]]: + """Pass one: the video VAE, then dropped. Returns the latents and the pairs they came from.""" + import numpy import torch from PIL import Image @@ -84,45 +106,98 @@ def _encode_pixels( from . import dataset as ds out: list[Any] = [] + kept: list[tuple[Path, str]] = [] + skipped: list[str] = [] + total = len(pairs) + say(f"encoding {total} items at {resolution}px through the video VAE") try: - for img_path, _caption in pairs: + for index, (path, _caption) in enumerate(pairs, start=1): + clip = ds.is_video(path) + try: + frames = _clip_frames(path, clip_frames) if clip else [Image.open(path)] + except ShortClipError as exc: + skipped.append(path.name) + say(f"skipped {exc}") + continue + # A long precache is otherwise silent for many minutes, so report often enough that it + # reads as progress rather than a hang. + if index == 1 or index % 5 == 0 or index == total: + say(f"caching latents {index}/{total}") for mirrored in (False, True) if flip else (False,): - square = ds._square(Image.open(img_path), resolution, mirrored) - # H3 normalises with ImageNet statistics, not to [-1, 1] like the image archs, and - # a still is one frame: (1, 3, 1, H, W). - raw = torch.from_numpy(_as_array(square)).to(device) - pixels = raw.permute(2, 0, 1)[None, :, None] + stack = [_as_array(ds._square(f, resolution, mirrored)) for f in frames] + # ImageNet statistics, not the [-1, 1] the image archs use, and always 5D: + # (1, 3, F, H, W). + raw = torch.from_numpy(numpy.stack(stack)).to(device) + pixels = raw.permute(3, 0, 1, 2)[None] pixels = (pixels.to(torch.float32).div(255.0) - pixel_mean) / pixel_std with torch.no_grad(): - # The spatial encoder alone, the path inference uses for a single frame; the - # temporal chunking is for 17n+5 clips. - latent = _sample(vae._encode_clip(pixels)) + # A single frame takes the spatial encoder; a 17n+5 clip takes the temporal + # chunking. Mirrors the split the vendored reference encoder makes. + moments = vae._encode(pixels) if clip else vae._encode_clip(pixels) + latent = _sample(moments) out.append(((latent.cpu() - mean) / std)[0]) + kept.append((path, _caption)) finally: del vae _reclaim() - logger.info("MiniMax H3: cached %d latents, video VAE released", len(out)) - return out + if skipped: + say(f"skipped {len(skipped)} of {total} items as too short: {', '.join(skipped)}") + say(f"cached {len(out)} latents from {len(kept)} items, video VAE released") + return out, kept + + +class ShortClipError(RuntimeError): + """A clip below H3's frame floor. Skipped, never fatal: one bad file in a large dataset must + not throw away a precache that takes many minutes.""" + + +def _clip_frames(path: Path, clip_frames: int) -> list[Any]: + """A clip as PIL frames on H3's 24fps, 17n+5 grid, taken from the start. + + Trimmed rather than sampled: a fixed window keeps the precache to one encode per clip, and + re-encoding a different window every step would defeat caching the latents at all. + """ + from PIL import Image + + from ..models.minimaxh3.vendor.packing_ref2va import ( + decode_reference_video, + resample_reference_frames, + trim_reference_num_frames, + ) + + decoded, fps, _audio = decode_reference_video(str(path)) + frames = resample_reference_frames(decoded, fps) + keep = trim_reference_num_frames(min(frames.shape[0], clip_frames)) + if keep > frames.shape[0]: + raise ShortClipError( + f"{path.name} is {frames.shape[0]} frames once resampled to {_H3_FPS}fps, below H3's " + f"{keep}-frame minimum ({keep / _H3_FPS:.2f}s). Skipped." + ) + return [Image.fromarray(frame) for frame in frames[:keep]] def _encode_captions( - root: Path, captions: list[str], device: str, dtype: Any + root: Path, captions: list[str], device: str, dtype: Any, + say: Callable[[str], None] = lambda _text: None, ) -> list[tuple[Any, Any]]: """Pass two: the 4-bit conditioner, then dropped.""" import torch from ..models.minimaxh3.vendor.encoders import MiniMaxH3TextEncoderStep + say("loading the 4-bit text conditioner (20.5GB)") pipeline = _load_conditioner(root, device, dtype) # Encode wherever it landed. It spills to host RAM on a card too small for 20.5GB, and the # vendored step builds its input ids on the device it is handed, so CUDA ids against a # CPU-resident encoder fail in `index_select`. where = next(pipeline.text_encoder.parameters()).device if where.type != torch.device(device).type: - logger.info("MiniMax H3: conditioner is on %s, encoding captions there", where) + say(f"conditioner spilled to {where}, encoding captions there (slower)") out: list[tuple[Any, Any]] = [] try: - for caption in captions: + for index, caption in enumerate(captions, start=1): + if index == 1 or index % 10 == 0 or index == len(captions): + say(f"encoding captions {index}/{len(captions)}") caption = caption or _EMPTY_CAPTION with torch.no_grad(): # The staticmethod rather than the block, so nothing needs a PipelineState. `dtype` @@ -134,7 +209,7 @@ def _encode_captions( out.append((embeds[0].cpu(), tags.cpu())) finally: _drop_conditioner(pipeline) - logger.info("MiniMax H3: cached %d captions, conditioner released", len(out)) + say(f"cached {len(out)} captions, conditioner released") return out diff --git a/core/src/inline_core/training/models.py b/core/src/inline_core/training/models.py index 3eef6f3..0b24bd7 100644 --- a/core/src/inline_core/training/models.py +++ b/core/src/inline_core/training/models.py @@ -367,6 +367,66 @@ def _base_size(models_dir: str, arch: str, base_mode: str) -> int: return 0 +def _proc_int(path: str) -> int | None: + try: + return int(Path(path).read_text().strip()) + except (OSError, ValueError): + return None + + +def _memory_totals() -> tuple[int, int]: + """``(RAM, swap)`` in bytes, or ``(0, 0)`` where /proc/meminfo is not readable.""" + found: dict[str, int] = {} + try: + for line in Path("/proc/meminfo").read_text().splitlines(): + key, _, rest = line.partition(":") + if key in ("MemTotal", "SwapTotal"): + found[key] = int(rest.split()[0]) * 1024 + except (OSError, ValueError, IndexError): + return 0, 0 + return found.get("MemTotal", 0), found.get("SwapTotal", 0) + + +def check_base_mappable(models_dir: str, arch: str, base_mode: str) -> None: + """Refuse a run the kernel will not let mmap the base, before the precache rather than after. + + safetensors maps the checkpoint in one call, so a 62GB file asks for a 62GB mapping. Under + ``vm.overcommit_memory=0`` the kernel rejects any single mapping larger than RAM plus swap, and + the error it raises names the checkpoint, so it reads like a corrupt download. The mapping is + virtual and the loader streams through it at roughly one tensor of resident memory, so the + limit is bookkeeping rather than a real shortage. Checked here because precaching a large + dataset costs twenty minutes and runs first: the cheap failure has to come before the dear one. + """ + mode = _proc_int("/proc/sys/vm/overcommit_memory") + # 1 is unrestricted, and a missing knob means this is not Linux. + if mode is None or mode == 1: + return + size = _base_size(models_dir, arch, base_mode) + ram, swap = _memory_totals() + if size <= 0 or ram <= 0: + return + if mode == 2: + ratio = _proc_int("/proc/sys/vm/overcommit_ratio") or 50 + allowed = ram * ratio // 100 + swap + else: + allowed = ram + swap + if size <= allowed: + return + + gib = 1024**3 + name = Path(_base_file(Path(models_dir), arch, base_mode)).name + raise RuntimeError( + f"{name} needs a single {size / gib:.1f}GiB memory mapping, but this machine caps one at " + f"{allowed / gib:.1f}GiB (RAM {ram / gib:.0f}GiB plus swap {swap / gib:.0f}GiB) while " + f"vm.overcommit_memory={mode}. The mapping is virtual and the weights stream through it, " + f"so the memory is never all used at once, but the kernel refuses the request up front. " + f"Allow it with:\n" + f" sudo sysctl -w vm.overcommit_memory=1\n" + f"and to keep it across reboots:\n" + f" echo 'vm.overcommit_memory = 1' | sudo tee /etc/sysctl.d/99-inline-studio.conf" + ) + + def load_transformer( models_dir: str, arch: str, base_mode: str, device: str, dtype: Any, quant: Any = None ) -> Any: diff --git a/core/src/inline_core/training/trainer.py b/core/src/inline_core/training/trainer.py index 848802e..a0352f6 100644 --- a/core/src/inline_core/training/trainer.py +++ b/core/src/inline_core/training/trainer.py @@ -139,10 +139,14 @@ def _activation_offload(enabled: bool) -> Any: def _to_device(item: dict[str, Any], device: Any, dtype: Any) -> dict[str, Any]: - """A cached item on the training device, casting only its activations.""" + """A cached item on the training device, casting only its activations. + + Anything that is not a tensor is dropped: an arch may stash its own bookkeeping on the item + (H3 keeps a per-item unconditional layout there) and the model never sees it.""" return { key: value.to(device, dtype) if key in _ACTIVATION_KEYS else value.to(device) for key, value in item.items() + if hasattr(value, "to") } @@ -167,11 +171,20 @@ def train(manifest: dict[str, Any]) -> str | None: # Two phases, never overlapping: encoders -> precache -> free, THEN the transformer. Held # together they add the text encoder's several GB to the base; apart, peak is just the base. + # Before the precache, never after: this costs milliseconds and precaching costs twenty + # minutes, and the failure it catches only surfaces once the base finally loads. + models.check_base_mappable(manifest["modelsDir"], arch.key, manifest["baseMode"]) + protocol.progress(0, steps, status="caching latents") dropout = max(0.0, min(1.0, float(hp.get("captionDropout") or 0.0))) + # Precache is minutes of silence on a large dataset, so its phases are reported as progress + # statuses. The orchestrator turns each new status into a log line, which is the only channel + # that reaches the UI: this subprocess installs no logging handler. data, unconditional, shift = cache.build( manifest["datasetDir"], manifest["modelsDir"], arch.key, str(device), dtype, resolution, flip=bool(hp.get("flipAugment")), dropout=dropout, + clip_frames=archs.clip_frames(arch, hp.get("clipSeconds")), + on_status=lambda text: protocol.progress(0, steps, status=text), ) quant = models.resolve_quant( @@ -227,10 +240,14 @@ def train(manifest: dict[str, Any]) -> str | None: if stop.flagged: break source = data[step % len(data)] - if unconditional is not None and random.random() < dropout: - source = {**source, **unconditional} + if dropout and random.random() < dropout: + # An arch whose layout depends on the item carries its own unconditional; the rest + # share one. H3 needs the per-item form because a clip and a still pack differently. + swap = source.get("uncond") or unconditional + if swap is not None: + source = {**source, **swap} item = _to_device(source, device, dtype) - clean = item["latent"] # (C, H, W) + clean = item["latent"] # (C, H, W) for the image archs, (C, F, H, W) for H3 noise = torch.randn_like(clean) sigma = arch.sigma(device, shift) # scalar noise fraction in (0, 1) noisy = (1 - sigma) * clean + sigma * noise diff --git a/core/tests/test_minimaxh3_training.py b/core/tests/test_minimaxh3_training.py index 22dc36e..8093155 100644 --- a/core/tests/test_minimaxh3_training.py +++ b/core/tests/test_minimaxh3_training.py @@ -359,3 +359,96 @@ def test_h3_forward_packs_and_unpacks_back_to_the_latent_grid() -> None: assert tuple(transformer.seen["hidden_states"].shape) == (1, 16, 96) assert tuple(transformer.seen["audio_hidden_states"].shape) == (1, 0, 32) assert tuple(transformer.seen["timestep"].shape) == (1,) + + +def test_clip_length_snaps_to_the_frame_grid() -> None: + """H3's VAE encodes 17n+5 frames, so a request lands on the grid or not at all.""" + h3 = archs.get(archs.MINIMAX_H3) + + # Its floor is a whole 17-frame chunk plus the 5-frame head: 22 frames, 0.92s. Anything + # shorter rounds up rather than being refused, because the VAE cannot encode less. + assert archs.clip_frames(h3, 0.1) == 22 + assert archs.clip_frames(h3, 1.0) == 22 + assert archs.clip_frames(h3, 2.0) == 39 + assert archs.clip_frames(h3, 5.0) == 107 + for frames in (22, 39, 107): + assert (frames - 5) % 17 == 0 + + +def test_an_arch_without_clips_always_reports_one_frame() -> None: + for key in (archs.Z_IMAGE, archs.KREA2, archs.FLUX2): + assert archs.clip_frames(archs.get(key), 5.0) == 1 + + +def test_unset_clip_length_still_gives_an_encodable_clip() -> None: + """A dataset can hold a clip with no clip length set, and 1 frame is not encodable.""" + assert archs.clip_frames(archs.get(archs.MINIMAX_H3), None) == 22 + + +def test_a_clip_packs_more_rows_than_a_still_at_the_same_resolution() -> None: + """The reason clip training costs what it does: rows scale with latent frames.""" + still = _layout(text_tokens=5, latent=8) + clip = packing.build_packed_sequence( + text_token_tags=torch.full((5,), packing.MINIMAX_H3_TEXT_TAG, dtype=torch.long), + num_latent_frames=packing.video_latent_num_frames(22), + latent_height=8, + latent_width=8, + num_audio_latents=0, + patch_size=PATCH, + keyframe_anchors=(), + ) + + assert packing.video_latent_num_frames(22) == 7 + assert clip.video_indices.numel() == 7 * still.video_indices.numel() + assert clip.audio_indices.numel() == 0 + + +def test_only_the_video_archs_see_clips_in_a_dataset(tmp_path: object) -> None: + """An image arch handed an mp4 would reach PIL and raise, so the filter is opt-in.""" + from pathlib import Path + + from inline_core.training import dataset as ds + + root = Path(str(tmp_path)) + (root / "0000.jpg").write_bytes(b"") + (root / "0001.mp4").write_bytes(b"") + + assert [p.name for p, _c in ds._pairs(root)] == ["0000.jpg"] + both = ds._pairs(root, ds._IMAGE_SUFFIXES + ds._VIDEO_SUFFIXES) + assert [p.name for p, _c in both] == ["0000.jpg", "0001.mp4"] + assert ds.is_video(root / "0001.mp4") and not ds.is_video(root / "0000.jpg") + + +def _write_clip(path: Path, frames: int, fps: int = 24) -> Path: + """A tiny valid mp4 with an exact frame count.""" + av = pytest.importorskip("av") + numpy = pytest.importorskip("numpy") + container = av.open(str(path), mode="w") + stream = container.add_stream("libx264", rate=fps) + stream.width, stream.height, stream.pix_fmt = 64, 64, "yuv420p" + for i in range(frames): + arr = numpy.full((64, 64, 3), i * 4 % 256, dtype=numpy.uint8) + container.mux(stream.encode(av.VideoFrame.from_ndarray(arr, format="rgb24"))) + container.mux(stream.encode()) + container.close() + return path + + +def test_short_clip_raises_the_skippable_error(tmp_path) -> None: + """One clip under H3's 22-frame floor must be skippable, not fatal: a hard failure threw away a + precache that can take many minutes on a large dataset.""" + from inline_core.training import h3 + + clip = _write_clip(tmp_path / "tooshort.mp4", frames=6) + with pytest.raises(h3.ShortClipError) as caught: + h3._clip_frames(clip, clip_frames=24) + assert "tooshort.mp4" in str(caught.value) + + +def test_long_enough_clip_encodes_on_the_frame_grid(tmp_path) -> None: + from inline_core.training import h3 + + clip = _write_clip(tmp_path / "ok.mp4", frames=40) + frames = h3._clip_frames(clip, clip_frames=24) + # Snapped down onto H3's 17n+5 grid rather than taking all 40. + assert len(frames) == 22 diff --git a/core/tests/test_studio_store.py b/core/tests/test_studio_store.py index dea6848..83809c7 100644 --- a/core/tests/test_studio_store.py +++ b/core/tests/test_studio_store.py @@ -3,6 +3,7 @@ from __future__ import annotations import sqlite3 +from pathlib import Path import pytest @@ -104,3 +105,34 @@ def test_settings_defaults_and_overrides(tmp_path) -> None: assert store.get_settings()["coreUrl"] == "http://127.0.0.1:9999" # Blank falls back to the default. assert store.set_core_url(" ")["coreUrl"] == "http://core" + + +def test_restart_reopens_the_last_project(tmp_path) -> None: + """Core holds the open project in memory, so without this a restart left a still-open browser + tab failing every call with "No project is open.".""" + created = _store(tmp_path).create_project("My Film") + + restarted = _store(tmp_path) + assert restarted.current_project()["id"] == created["id"] + assert restarted.conn() is not None + + +def test_closing_a_project_stops_the_reopen(tmp_path) -> None: + store = _store(tmp_path) + store.create_project("My Film") + store.close_project() + + assert store.current_project() is None + assert _store(tmp_path).current_project() is None + + +def test_restore_forgets_a_project_that_moved(tmp_path) -> None: + store = _store(tmp_path) + project = store.create_project("My Film") + store.close() + Path(project["path"]).rename(tmp_path / "workspace" / "elsewhere.inlinestudio") + + restarted = _store(tmp_path) + assert restarted.current_project() is None + # Forgotten, not retried on every call. + assert not (tmp_path / "appdata" / "last_project").exists() diff --git a/core/tests/test_studio_training.py b/core/tests/test_studio_training.py index fc8af43..d1421e9 100644 --- a/core/tests/test_studio_training.py +++ b/core/tests/test_studio_training.py @@ -7,6 +7,7 @@ import sqlite3 import pytest + from inline_core.models import lora from inline_core.studio import training_store as ts from inline_core.studio.schema import apply_schema @@ -79,3 +80,96 @@ def test_peft_adapter_keys_are_fuser_compatible() -> None: # The fuser strips the PEFT `base_model.model.` prefix, yielding the real module path. assert "transformer_blocks.0.attn.to_q" in lora._candidates(stem) + + +class _Store: + """The two things `Training` asks a store for.""" + + def __init__(self, conn: sqlite3.Connection, folder: object) -> None: + self._conn, self._folder = conn, folder + + def conn(self) -> sqlite3.Connection: + return self._conn + + def folder(self) -> object: + return self._folder + + +def test_add_from_path_imports_a_folder_with_its_sidecar_captions( + conn: sqlite3.Connection, tmp_path: object +) -> None: + """The clip case: pointing at a folder beats pushing gigabytes through the browser.""" + from pathlib import Path + + from inline_core.studio.training import Training + + src = Path(str(tmp_path)) / "src" + src.mkdir() + (src / "0000.png").write_bytes(b"x") + (src / "0000.txt").write_text("a red car") + (src / "0001.mp4").write_bytes(b"x") # a clip, no caption + (src / "notes.md").write_text("ignored") # not media + + project = Path(str(tmp_path)) / "project" + project.mkdir() + dataset = ts.create_dataset(conn, "d", "") + items = Training(_Store(conn, project), events=None).add_from_path(dataset["id"], str(src)) + + assert len(items) == 2, "the .md is not media and must not be imported" + captions = {i["caption"] for i in items} + assert captions == {"a red car", ""} + # The files are copied into the project rather than referenced in place. + assert len(list((project / "assets").iterdir())) == 2 + + +def test_add_from_path_rejects_a_folder_that_is_not_one( + conn: sqlite3.Connection, tmp_path: object +) -> None: + from pathlib import Path + + from inline_core.studio.training import Training + + dataset = ts.create_dataset(conn, "d", "") + training = Training(_Store(conn, Path(str(tmp_path))), events=None) + with pytest.raises(ValueError, match="Not a folder"): + training.add_from_path(dataset["id"], str(Path(str(tmp_path)) / "nope")) + + +def test_add_from_path_says_so_when_a_folder_holds_no_media( + conn: sqlite3.Connection, tmp_path: object +) -> None: + from pathlib import Path + + from inline_core.studio.training import Training + + empty = Path(str(tmp_path)) / "empty" + empty.mkdir() + (empty / "readme.txt").write_text("just captions, no images") + dataset = ts.create_dataset(conn, "d", "") + training = Training(_Store(conn, Path(str(tmp_path))), events=None) + with pytest.raises(ValueError, match="No images or clips"): + training.add_from_path(dataset["id"], str(empty)) + + +def test_each_new_precache_status_becomes_a_log_line() -> None: + """Precache progress reaches the UI only as a changing progress status. The trainer subprocess + installs no logging handler, so a logger.info there is dropped and the pane stays silent for + the minutes a large dataset takes.""" + from inline_core.studio.training import _progress_log_line + + last = "" + lines = [] + for status in ("caching latents 5/173", "caching latents 10/173", "caching latents 10/173"): + line = _progress_log_line({"status": status}, last) + if line is not None: + lines.append(line) + last = status + # The first two are new phases and get a line; the repeat is suppressed. + assert lines == ["caching latents 5/173", "caching latents 10/173"] + + +def test_a_step_with_a_loss_still_wins_over_the_status() -> None: + from inline_core.studio.training import _progress_log_line + + line = _progress_log_line({"status": "training", "step": 7, "total": 500, "loss": 0.1234}, "") + assert line == "step 7/500 · loss 0.1234" diff --git a/core/tests/test_training_models.py b/core/tests/test_training_models.py index fecf3a9..b7623dc 100644 --- a/core/tests/test_training_models.py +++ b/core/tests/test_training_models.py @@ -133,3 +133,68 @@ def test_zimage_has_no_four_bit_path_and_says_so(tmp_path) -> None: assert models.resolve_quant("auto", str(tmp_path), archs.Z_IMAGE, "deturbo", 1024) is ( Quantization.NONE ) + + +# --- mmap preflight ------------------------------------------------------------------------- +# +# safetensors maps the whole checkpoint in one call, and vm.overcommit_memory=0 refuses a single +# mapping larger than RAM+swap. The raw kernel error names the checkpoint, so it reads as a corrupt +# download, and it only fires after the precache has already cost twenty minutes. + + +def _fake_env(monkeypatch, *, mode, ram_gib, swap_gib=0, size_gib=62, ratio=50): + values = { + "/proc/sys/vm/overcommit_memory": mode, + "/proc/sys/vm/overcommit_ratio": ratio, + } + monkeypatch.setattr(models, "_proc_int", lambda p: values.get(p)) + monkeypatch.setattr( + models, "_memory_totals", lambda: (ram_gib * 1024**3, swap_gib * 1024**3) + ) + monkeypatch.setattr(models, "_base_size", lambda *a: int(size_gib * 1024**3)) + monkeypatch.setattr(models, "_base_file", lambda *a: "/m/minimax_h3_fl2va_bf16.safetensors") + + +def test_refuses_a_checkpoint_bigger_than_ram_plus_swap(monkeypatch) -> None: + _fake_env(monkeypatch, mode=0, ram_gib=30, swap_gib=0, size_gib=62) + with pytest.raises(RuntimeError) as caught: + models.check_base_mappable("/m", "minimax-h3", "raw") + message = str(caught.value) + assert "minimax_h3_fl2va_bf16.safetensors" in message + # The fix has to be in the message, since this is the whole point of checking early. + assert "vm.overcommit_memory=1" in message + + +def test_swap_counts_toward_the_ceiling(monkeypatch) -> None: + _fake_env(monkeypatch, mode=0, ram_gib=30, swap_gib=64, size_gib=62) + models.check_base_mappable("/m", "minimax-h3", "raw") + + +def test_overcommit_always_is_never_refused(monkeypatch) -> None: + """Mode 1 lets any mapping through, however large.""" + _fake_env(monkeypatch, mode=1, ram_gib=8, swap_gib=0, size_gib=62) + models.check_base_mappable("/m", "minimax-h3", "raw") + + +def test_strict_mode_uses_the_overcommit_ratio(monkeypatch) -> None: + """Mode 2 allows only ratio% of RAM plus swap, so 50% of 200GB does not fit 62GB... it does, + but 50% of 100GB with no swap does not.""" + _fake_env(monkeypatch, mode=2, ram_gib=100, swap_gib=0, size_gib=62, ratio=50) + with pytest.raises(RuntimeError): + models.check_base_mappable("/m", "minimax-h3", "raw") + _fake_env(monkeypatch, mode=2, ram_gib=100, swap_gib=0, size_gib=40, ratio=50) + models.check_base_mappable("/m", "minimax-h3", "raw") + + +def test_fails_open_when_the_machine_cannot_be_read(monkeypatch) -> None: + """No /proc (not Linux) or an unmeasurable checkpoint must never block a run that would work.""" + monkeypatch.setattr(models, "_proc_int", lambda _p: None) + models.check_base_mappable("/m", "minimax-h3", "raw") + + _fake_env(monkeypatch, mode=0, ram_gib=30, size_gib=62) + monkeypatch.setattr(models, "_memory_totals", lambda: (0, 0)) + models.check_base_mappable("/m", "minimax-h3", "raw") + + _fake_env(monkeypatch, mode=0, ram_gib=30, size_gib=62) + monkeypatch.setattr(models, "_base_size", lambda *a: 0) + models.check_base_mappable("/m", "minimax-h3", "raw") diff --git a/package.json b/package.json index 6029931..56b231e 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "inline-studio", - "version": "1.2.63", + "version": "1.2.64", "description": "AI filmmaking on a node canvas. Generate locally on your own GPU and train your own LoRAs on the same canvas, with the built-in Inline Core engine and hosted models. Every render is kept as a versioned take.", "keywords": [ "ai-filmmaking", diff --git a/packages/frontend/pyproject.toml b/packages/frontend/pyproject.toml index 013939c..c32cf61 100644 --- a/packages/frontend/pyproject.toml +++ b/packages/frontend/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "inline-studio-frontend" -version = "1.2.63" +version = "1.2.64" description = "Prebuilt Inline Studio web UI (SPA), served by Inline Core. Mirrors comfyui-frontend-package." requires-python = ">=3.9" readme = "README.md" diff --git a/src/renderer/App.tsx b/src/renderer/App.tsx index bea9daa..a015a9a 100644 --- a/src/renderer/App.tsx +++ b/src/renderer/App.tsx @@ -8,9 +8,15 @@ import { UpdateBanner } from './components/UpdateBanner' export function App(): React.JSX.Element { const current = useProjectStore((s) => s.current) + const restoring = useProjectStore((s) => s.restoring) + const restore = useProjectStore((s) => s.restore) const loadRecents = useProjectStore((s) => s.loadRecents) const subscribeToUpdates = useUpdateStore((s) => s.subscribeToEvents) + useEffect(() => { + void restore() + }, [restore]) + useEffect(() => { void loadRecents() }, [loadRecents]) @@ -19,6 +25,9 @@ export function App(): React.JSX.Element { useEffect(() => subscribeToLibraryChanges(), []) + // Hold the first paint until Core has answered, so a restored project does not flash the launcher. + if (restoring) return
+ return ( <> diff --git a/src/renderer/components/MediaLightbox.tsx b/src/renderer/components/MediaLightbox.tsx index 4e9678e..11b4907 100644 --- a/src/renderer/components/MediaLightbox.tsx +++ b/src/renderer/components/MediaLightbox.tsx @@ -4,9 +4,10 @@ import { useLightboxStore } from '../store/lightboxStore' import { CloseIcon } from './icons' /** - * Fullscreen media viewer, opened by double-clicking a node's image/video. Portaled to - * document.body so it sits above the canvas and all panels. Click the backdrop or press Escape - * to close. + * Fullscreen media viewer, opened by double-clicking a node's or dataset tile's image/video. + * Portaled to document.body so it sits above the canvas and all panels. Its z-index must stay + * above every modal (the Captioning modal is z-200), since it can be opened from inside one. + * Click the backdrop or press Escape to close. */ export function MediaLightbox(): React.JSX.Element | null { const media = useLightboxStore((s) => s.media) @@ -25,7 +26,7 @@ export function MediaLightbox(): React.JSX.Element | null { return createPortal(
{media.kind === 'video' ? ( diff --git a/src/renderer/store/projectStore.ts b/src/renderer/store/projectStore.ts index bd3ef5f..56ab64d 100644 --- a/src/renderer/store/projectStore.ts +++ b/src/renderer/store/projectStore.ts @@ -11,12 +11,16 @@ interface ProjectState { current: Project | null recents: RecentProject[] loading: boolean + /** True until Core has been asked what is open, so boot shows no launcher flash. */ + restoring: boolean error: string | null /** A transient success message (e.g. "Exported to …"). */ notice: string | null /** Path of the project currently being exported, if any. */ exportingPath: string | null + /** Adopt whatever project Core has open, so a refresh lands back in the workspace. */ + restore: () => Promise loadRecents: () => Promise createProject: (name: string) => Promise openFromDialog: () => Promise @@ -31,10 +35,17 @@ export const useProjectStore = create((set, get) => ({ current: null, recents: [], loading: false, + restoring: true, error: null, notice: null, exportingPath: null, + restore: async () => { + const res = await studio().project.current() + // A failure here just means the launcher shows; it must never block boot behind an error. + set({ current: res.ok ? res.value : null, restoring: false }) + }, + loadRecents: async () => { const res = await studio().project.listRecent() if (res.ok) set({ recents: res.value }) @@ -94,5 +105,9 @@ export const useProjectStore = create((set, get) => ({ } }, - closeProject: () => set({ current: null, error: null, notice: null }), + // Tell Core too, or it reopens this project on its next start. + closeProject: () => { + void studio().project.close() + set({ current: null, error: null, notice: null }) + }, })) diff --git a/src/renderer/store/trainingStore.ts b/src/renderer/store/trainingStore.ts index e2b598a..7cc2c06 100644 --- a/src/renderer/store/trainingStore.ts +++ b/src/renderer/store/trainingStore.ts @@ -54,6 +54,8 @@ interface TrainingState { loadItems: (datasetId: string) => Promise /** Returns the created items, so a caller can pair captions to the assets it just added. */ addItems: (datasetId: string, assetIds: string[]) => Promise + /** Import a folder on the Core machine, sidecar captions included. Returns an error string. */ + addFromPath: (datasetId: string, path: string) => Promise removeItem: (datasetId: string, itemId: string) => Promise /** Remove every image from a dataset in one go, refetching once when done. */ removeAll: (datasetId: string) => Promise @@ -131,6 +133,18 @@ export const useTrainingStore = create((set, get) => ({ return res.value }, + addFromPath: async (datasetId, path) => { + const res = await studio().training.addFromPath(datasetId, path) + if (!res.ok) { + // Returned rather than only stored: a bad path is the common case here and the field wants + // to show it inline, next to what the reader typed. + set({ error: res.error }) + return ipcErrorMessage(res.error) + } + await get().loadItems(datasetId) + return null + }, + removeItem: async (datasetId, itemId) => { const res = await studio().training.removeItem(itemId) if (!res.ok) return set({ error: res.error }) @@ -233,10 +247,12 @@ export const useTrainingStore = create((set, get) => ({ applyLog: (e) => set((s) => ({ - // Capped so a long run can't grow the buffer without bound; the node shows the tail. + // Capped so a long run can't grow the buffer without bound; the node shows the tail. Deep + // enough to still hold the setup and precache lines once a few hundred steps have streamed + // in, since those are what a "Copy logs" for a failed run needs to include. logsByRun: { ...s.logsByRun, - [e.runId]: [...(s.logsByRun[e.runId] ?? []), e.line].slice(-400), + [e.runId]: [...(s.logsByRun[e.runId] ?? []), e.line].slice(-3000), }, })), diff --git a/src/renderer/views/Moodboard/nodes/ThumbStrip.tsx b/src/renderer/views/Moodboard/nodes/ThumbStrip.tsx index e0c6c72..dfb7ca9 100644 --- a/src/renderer/views/Moodboard/nodes/ThumbStrip.tsx +++ b/src/renderer/views/Moodboard/nodes/ThumbStrip.tsx @@ -41,9 +41,12 @@ export function ThumbStrip({ edge === 'top' ? 'top-0 bg-gradient-to-b from-black/80 via-black/40 to-transparent pt-1.5 pb-5' : 'bottom-0 bg-gradient-to-t from-black/80 via-black/40 to-transparent pb-1.5 pt-5' + // Wraps rather than scrolling sideways: a single row fits about six thumbnails at the default node + // width, which hid the rest of a large input set behind a scrollbar nobody found. Rows grow as the + // node is resized, capped so the strip never swallows the preview it floats over. return (
{items.map((it, i) => (
@@ -68,7 +71,14 @@ export function ThumbStrip({ ) : it.kind === 'video' && !it.poster ? ( -
+ {fromPath && ( +
+
+ { + setPath(e.target.value) + setPathError(null) + }} + onKeyDown={(e) => { + if (e.key === 'Enter') void loadPath() + }} + placeholder="/path/to/a/folder of images or clips" + spellCheck={false} + className="flex-1 rounded-md border border-border bg-black/30 px-2 py-1.5 text-sm text-zinc-100 outline-none focus:border-zinc-500" + /> + +
+ + {pathError ?? + 'A folder on the machine running Inline Studio, not your browser. Copies every image and clip in it, and reads any matching .txt as the caption.'} + +
+ )} + {items.length === 0 ? (
Drop images here, or use Add files. A .txt next to an image (1.png + 1.txt) is read as its @@ -381,7 +499,7 @@ export function DatasetItemsGrid({ datasetId }: { datasetId: string }): React.JS className="group flex flex-col rounded-md border border-border bg-surface/60" >
- {src && } + {src && }
-
- {thumbs.length === 0 ? ( +
+ {items.length === 0 ? (
- {datasetId ? 'No images yet' : 'Pick a dataset'} + {datasetId ? 'No items yet' : 'Pick a dataset'}
) : ( -
+
{thumbs.map((it) => { const asset = byId.get(it.assetId) const src = asset ? resolveMedia(asset.thumbPath ?? asset.filePath) : '' @@ -84,7 +109,19 @@ export function TrainDatasetNode({ id, selected }: NodeProps): React.JSX.Element key={it.id} className="aspect-square overflow-hidden rounded-sm bg-zinc-900" > - {src && } + {src && + (asset?.kind === 'video' ? ( + // Poster generation is deferred, so a clip has no thumbPath and an + // pointed at an mp4 renders broken. `#t=` seeks the browser to a frame. +
) })} @@ -93,7 +130,7 @@ export function TrainDatasetNode({ id, selected }: NodeProps): React.JSX.Element
- {items.length} images · {captioned} captioned + {items.length} {noun} · {captioned} captioned
diff --git a/src/renderer/views/Trainer/nodes/TrainerNode.tsx b/src/renderer/views/Trainer/nodes/TrainerNode.tsx index 5bb5a65..670036a 100644 --- a/src/renderer/views/Trainer/nodes/TrainerNode.tsx +++ b/src/renderer/views/Trainer/nodes/TrainerNode.tsx @@ -6,10 +6,12 @@ * `runId` is persisted in the node's data, so the node rebinds to its run after a reload and can * still offer Resume. Hyperparameters live in the Adjust sidebar, never on the node face. */ -import { useEffect, useRef } from 'react' +import { useEffect, useRef, useState } from 'react' import { Handle, Position, type NodeProps } from '@xyflow/react' import type { TrainingRun } from '@shared/types' import { useTrainingStore } from '../../../store/trainingStore' +import { copyText } from '../../../lib/clipboard' +import { CopyIcon } from '../../../components/icons' import { useTrainerBoardStore } from '../../../store/trainerBoardStore' import { useCoreNodesStore } from '../../../store/coreNodesStore' import { activeDownload, useModelRequirementsStore } from '../../../store/modelRequirementsStore' @@ -93,6 +95,15 @@ export function TrainerNode({ id, selected }: NodeProps): React.JSX.Element { if (el && stickToBottom.current) el.scrollTop = el.scrollHeight }, [logs.length]) + // Which copy just fired ('all' or 'line-N'), so the button/row can flash confirmation. + const [copied, setCopied] = useState(null) + const copy = async (text: string, key: string): Promise => { + if (await copyText(text)) { + setCopied(key) + setTimeout(() => setCopied((k) => (k === key ? null : k)), 1200) + } + } + // Another node holding the GPU blocks this one (the backend allows one run at a time). const otherRunning = runs.some( (r) => r.id !== runId && (r.status === 'training' || r.status === 'queued'), @@ -186,7 +197,7 @@ export function TrainerNode({ id, selected }: NodeProps): React.JSX.Element { const el = e.currentTarget stickToBottom.current = el.scrollHeight - el.scrollTop - el.clientHeight < 24 }} - className={`nowheel nodrag h-full overflow-auto px-2 pb-2 font-mono text-[10px] leading-snug text-zinc-300 ${ + className={`nowheel nodrag h-full cursor-text select-text overflow-auto px-2 pb-2 font-mono text-[10px] leading-snug text-zinc-300 ${ busy ? 'pt-8' : 'pt-2' }`} > @@ -196,12 +207,29 @@ export function TrainerNode({ id, selected }: NodeProps): React.JSX.Element { ) : ( logs.slice(-200).map((line, i) => ( -
+
void copy(line, `line-${i}`)} + title="Double-click to copy this line" + className={`whitespace-pre rounded-sm px-1 -mx-1 hover:bg-white/5 ${ + copied === `line-${i}` ? 'bg-emerald-500/25' : '' + }`} + > {line}
)) )}
+ {logs.length > 0 && ( + + )} {busy && (
> listRecent(): Promise> current(): Promise> + /** Close the open project, so Core does not reopen it on its next start. */ + close(): Promise> /** Absolute input/output dirs of the open project, for sharing with ComfyUI. */ mediaDirs(): Promise> /** Zip a project folder (by path) into a portable .zip; null if the save dialog is cancelled. */ @@ -438,6 +442,11 @@ export interface InlineStudioApi { listItems(datasetId: string): Promise> /** Append library assets as dataset items (skips duplicates). */ addItems(datasetId: string, assetIds: string[]): Promise> + /** + * Import every image and clip in a folder on the machine running Core, with any + * `NNNN.txt` sidecar as that item's caption. Returns the dataset's full item list. + */ + addFromPath(datasetId: string, path: string): Promise> removeItem(itemId: string): Promise> setCaption(itemId: string, caption: string): Promise> /** diff --git a/src/shared/types.ts b/src/shared/types.ts index f78cd2b..f4a3f67 100644 --- a/src/shared/types.ts +++ b/src/shared/types.ts @@ -519,6 +519,11 @@ export interface TrainingHyperparams { batchSize: number /** Square training resolution in px (e.g. 1024). */ resolution: number + /** + * Seconds of each video clip to train on, for archs that accept clips. Snapped to the model's + * frame grid, and ignored entirely for stills. MiniMax H3's floor is 0.92s. + */ + clipSeconds?: number /** Checkpoint every N steps. */ saveEvery: number /** GPU indices to train on; `[]` = auto (first available). */