Minimax H3 lora training with clip dataset

README.md

@@ -27,7 +27,7 @@ Inline Studio is a free, open-source app for **AI filmmaking on a node canvas**,
 
 - **Non-destructive by default** - every render is kept as a versioned take; generating again adds one, nothing is overwritten.
 - **Local diffusion generation engine** - the built-in Inline Core engine runs popular diffusion models locally, on your own GPU, from a single model file, no external server. Currently supported: **Z-Image Turbo**, **Krea 2** (RAW + Turbo), **FLUX.2**, and **MiniMax H3** for video with sound.
-- **Train LoRAs locally** - the Trainer canvas fine-tunes Z-Image, Krea 2, FLUX.2 or MiniMax H3 on your own images, on your own GPU. With a 4-bit base, Krea 2 trains at 512px inside about 12GB, so a 16GB card can train a LoRA for a 26GB model. See [LoRA training](#lora-training).
+- **Train LoRAs locally** - the Trainer canvas fine-tunes Z-Image, Krea 2, FLUX.2 or MiniMax H3 on your own images, on your own GPU. H3 also trains on short video clips, so a LoRA can learn motion and not just look. With a 4-bit base, Krea 2 trains at 512px inside about 12GB, so a 16GB card can train a LoRA for a 26GB model. See [LoRA training](#lora-training).
 - **Hosted models via API Nodes** - reach for closed models with no GPU and no setup for instant creative range; see [API Nodes](#api-nodes).
 - **Mix both in the same film** - Inline Studio handles everything around the render: exploring options, keeping what works, and shaping a repeatable process you can iterate on and share.
 

TRAINING.md

@@ -12,7 +12,7 @@ resolution.
 **Contents:** [The graph](#the-graph) · [Datasets and outputs](#datasets-and-outputs) ·
 [Stop and resume](#stop-and-resume) · [Trigger words](#trigger-words) ·
 [Architecture and base model modes](#architecture-and-base-model-modes) · [Install](#install) ·
-[**Benchmark results**](#benchmark-results) ·
+[Training on clips](#training-on-clips) · [**Benchmark results**](#benchmark-results) ·
 [Dataset and adapter options](#dataset-and-adapter-options) · [Base precision](#base-precision)
 
 ## The graph
@@ -57,7 +57,7 @@ The Trainer's Adjust panel picks the **architecture** first (Z-Image, Krea 2, FL
 **MiniMax H3** is the video model, and it trains on **still images**:
 
 - **FL2VA** is the only base, and it is undistilled, so there is no adapter and nothing to drift. Put `minimax_h3_fl2va_bf16.safetensors` in `models/diffusion_models/`, train on stills, then wire the LoRA into any of the four H3 nodes. It loads on the Reference to Video node too, which uses a different checkpoint file: the two partitions are the same architecture.
-- **What it learns** is appearance - look, style, character, lighting. It does not learn motion or sound, because it never sees any. This is how image LoRAs for video models are normally trained, and it is the same thing every other H3 trainer does today.
+- **Stills or short clips.** Drop images and it learns appearance: look, style, character, lighting. Drop video and it learns motion too. Sound is never learned either way, because the audio rows are empty. See [Training on clips](#training-on-clips).
 - **The base is 4-bit, always.** H3 is 40GB after the AdaLN factorisation and 11.7GB after quantisation, so full precision is refused rather than offered and then failing. There is no base-precision control for H3 for the same reason.
 - **A 24GB card is comfortable and a 16GB card works, slowly.** The run encodes latents and captions in two passes that never overlap, because H3's fp32 video VAE and its 32B conditioner cannot be resident together. On a card that holds the conditioner it peaks at 20.6GB; on one that does not, the conditioner runs on the CPU and the peak drops to 12.7GB while a step goes from 0.6s to 16s. Either way there is about seven minutes of startup, and 64GB of system RAM for the smaller card. See [Benchmark results](#benchmark-results) for the split. The download is about 124GB before any of that.
 
@@ -66,6 +66,42 @@ The Trainer's Adjust panel picks the **architecture** first (Z-Image, Krea 2, FL
 - **Turbo + training adapter** fuses a de-distillation adapter into the base for the duration of training and drops it when the LoRA is saved, which preserves the 8-step speed. Put [ostris/zimage_turbo_training_adapter](https://huggingface.co/ostris/zimage_turbo_training_adapter) in `models/loras/`; any filename containing `adapter` is detected automatically, or point `INLINE_ZIMAGE_TRAIN_ADAPTER` at a specific file. Keep runs short, since the adapter slows the breakdown rather than preventing it.
 - **De-Turbo** trains without an adapter and needs no extra download.
 
+## Training on clips
+
+The H3 trainer takes video as well as stills. Drop clips into a dataset the same way, set **Clip
+length** in the Adjust panel, and each clip trains as a short piece of motion rather than a frame.
+Mixed datasets are fine: a still is simply a one-frame clip.
+
+**It costs no extra VRAM.** Measured on an L4, every clip length peaks at the same 20.4GB as a
+still, because the high-water mark is the caption pass rather than the training:
+
+| Clip length | Frames | Latent frames | Packed rows at 512px | Peak VRAM |
+| ----------- | ------ | ------------- | -------------------- | --------- |
+| still       | 1      | 1             | 293                  | 20.55GB   |
+| 0.92s       | 22     | 7             | 1,832                | 20.4GB    |
+| 1.6s        | 39     | 12            | 3,112                | 20.4GB    |
+| 4.5s        | 107    | 32            | 8,232                | 20.4GB    |
+
+Rows are what a longer clip actually buys you, and they cost time rather than memory. That only
+holds while the conditioner is resident; on a card too small for it the peak is the training phase
+instead, and a long clip will push that up.
+
+**Lengths snap to H3's frame grid.** The VAE encodes `17n + 5` frames at 24fps, so a request lands
+on the nearest grid point at or below it. The floor is a whole chunk plus the five-frame head: 22
+frames, **0.92 seconds**. Asking for less rounds up rather than being refused, because the VAE has
+no way to encode a shorter clip.
+
+**Each clip is trimmed from its start, once.** The window is fixed at precache time so every clip is
+encoded exactly once. Sampling a different window each step would mean re-encoding through the VAE
+every step, which is the thing the precache exists to avoid. A clip shorter than the grid floor is
+refused by name rather than silently padded.
+
+**Captions work the same.** A clip is auto-captioned from its middle frame, which describes the shot
+better than the first frame usually does. Write them by hand if you would rather.
+
+Audio is not trained. H3 generates video and its soundtrack jointly, but the trainer packs zero
+audio rows, so an adapter changes what a clip looks like and never what it sounds like.
+
 ## Install
 
 If you installed with `--extra all` from [Get Started](README.md#get-started), the trainer is already set up - nothing more to do. To add it to a leaner install, its dependencies (PEFT, 8-bit Adam, the captioner) sit behind the `training` extra:
@@ -109,6 +145,7 @@ The LoRA a run produces lands in `models/loras/` and shows up in the LoRA loader
 | MiniMax H3 | FL2VA           | 512  | **4-bit**      | **20.6GB**    | **12.7GB**    |
 | MiniMax H3 | FL2VA           | 768  | **4-bit**      | **20.6GB**    | not measured  |
 | MiniMax H3 | FL2VA           | 1024 | **4-bit**      | **20.6GB**    | not measured  |
+| MiniMax H3 | FL2VA, clips    | 512  | **4-bit**      | **20.4GB**    | not measured  |
 
 **MiniMax H3 costs less on a smaller card, which is not a typo.** The run has three phases that never overlap, and the tallest is not the one doing the learning:
 
@@ -124,13 +161,14 @@ On a card too small for the conditioner it never goes there at all, so the peak
 
 **The bill arrives as time instead.** The conditioner runs on the CPU, and bitsandbytes only quantises on the move to CUDA, so it runs unquantised:
 
-|                         | L40S (46GB) | T4 (16GB, 64GB RAM) |
-| ----------------------- | ----------- | ------------------- |
-| Peak VRAM               | 20.6GB      | 12.7GB              |
-| Seconds per step        | 0.63        | 16.2                |
-| Caption pass, 26 images | 1 min       | 19 min              |
+|                         | L40S (46GB) | L4 (24GB) | T4 (16GB, 64GB RAM) |
+| ----------------------- | ----------- | --------- | ------------------- |
+| Peak VRAM, 512px        | 20.6GB      | 20.55GB   | 12.7GB              |
+| Seconds per step, 512px | 0.63        | 1.81      | 16.2                |
+| Seconds per step, 768px | 0.77        | 2.73      | not measured        |
+| Caption pass, 26 images | 1 min       | 1 min     | 19 min              |
 
-A 1500-step run is about 16 minutes on the L40S and closer to seven hours on the T4. Some of that is the T4 being a T4, and some is the caption pass being on the wrong processor.
+A 1500-step run at 512px is about 16 minutes on the L40S, 45 on the L4, and closer to seven hours on the T4. The L4 holds the conditioner, so it looks like a slower L40S rather than a faster T4: the 9x gap to the T4 is mostly the caption pass being on the wrong processor, not the cards themselves.
 
 **It also wants a lot of system RAM.** The unquantised conditioner pages roughly 63GB through the page cache, and on a 64GB machine that sits at 59GB resident, close enough to the edge that the caption pass is the riskiest part of the run. A T4 with only 16GB of RAM has room in neither VRAM nor RAM and is refused before anything loads, because a host-RAM overrun is killed by the kernel rather than raising.
 

core/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 # PyPI name; the import package is `inline_core` (src/inline_core).
 name = "inline-core"
-version = "1.2.63"
+version = "1.2.64"
 description = "The generation engine behind Inline Studio."
 readme = "README.md"
 license = "GPL-3.0-or-later"
@@ -82,6 +82,8 @@ all = [
   "accelerate>=0.30",
   "safetensors>=0.4",
   "torchao>=0.14",
+  # Clip decode for MiniMax H3 LoRA training, and H3's reference node.
+  "av>=12",
   "scipy>=1.11",
   "huggingface_hub>=0.23",
   "controlnet-aux>=0.0.7",

core/src/inline_core/server/__main__.py

@@ -79,6 +79,9 @@ def main() -> None:
         studio_config.workspace_dir(),
         default_core_url=studio_config.DEFAULT_CORE_URL,
     )
+    # Reopen whatever was open before the restart, so a browser tab left open across it keeps
+    # working instead of failing every call with "No project is open."
+    store.restore_last_project()
     app = create_app(
         registry=registry,
         cache=InMemoryCache(),

core/src/inline_core/studio/handlers.py

@@ -88,6 +88,7 @@ def fn(*_args: Any) -> Any:
     reg("project:openZip", lambda: None)
     reg("project:listRecent", store.list_recent)
     reg("project:current", store.current_project)
+    reg("project:close", store.close_project)
     reg("project:mediaDirs", store.media_dirs)
     reg("project:export", lambda _path: None)  # zip export: pending (see plan)
     reg("dialog:pickDirectory", lambda *_: str(cfg.workspace_dir()))
@@ -264,6 +265,7 @@ def cancel_generation(frame_id: str | None = None) -> None:
         reg("training:createDataset", lambda inp: training.create_dataset(inp))
         reg("training:listItems", lambda did: training.list_items(did))
         reg("training:addItems", lambda did, aids: training.add_items(did, aids))
+        reg("training:addFromPath", lambda did, path: training.add_from_path(did, path))
         reg("training:removeItem", lambda iid: training.remove_item(iid))
         reg("training:setCaption", lambda iid, cap: training.set_caption(iid, cap))
         reg("training:autoCaption",

core/src/inline_core/studio/store.py

@@ -131,6 +131,7 @@ def create_project(self, name: str, parent_dir: str | None = None) -> dict[str,
         project = {"id": pid, "name": name, "path": str(folder), "createdAt": now, "updatedAt": now}
         self._current = project
         self.record_recent(name, str(folder))
+        self._remember_last_project(str(folder))
         return project
 
     def open_project(self, selected: str) -> dict[str, Any]:
@@ -143,6 +144,7 @@ def open_project(self, selected: str) -> dict[str, Any]:
         project = self._load_project_row(folder)
         self._current = project
         self.record_recent(project["name"], str(folder))
+        self._remember_last_project(str(folder))
         return project
 
     def _load_project_row(self, folder: Path) -> dict[str, Any]:
@@ -160,7 +162,44 @@ def _load_project_row(self, folder: Path) -> dict[str, Any]:
         }
 
     def current_project(self) -> dict[str, Any] | None:
-        return self._current
+        return self._current or self.restore_last_project()
+
+    def close_project(self) -> None:
+        self.close()
+        self._current = None
+        self._remember_last_project(None)
+
+    # --- last opened project ----------------------------------------------------------------------
+    # The open project is otherwise only in memory, so restarting Core left a still-open browser tab
+    # failing every call with "No project is open." Kept in its own file rather than settings.json,
+    # because _save_settings rewrites that from get_settings() and would drop any key it omits.
+
+    def _last_project_file(self) -> Path:
+        return self._app_data / "last_project"
+
+    def _remember_last_project(self, path: str | None) -> None:
+        file = self._last_project_file()
+        try:
+            if path:
+                file.write_text(path, encoding="utf-8")
+            elif file.exists():
+                file.unlink()
+        except OSError:
+            pass  # never fail an open just because the marker could not be written
+
+    def restore_last_project(self) -> dict[str, Any] | None:
+        """Reopen the project left open at shutdown. Best-effort: a moved or deleted one is
+        forgotten and the launcher shows instead."""
+        if self._conn is not None:
+            return self._current
+        file = self._last_project_file()
+        if not file.exists():
+            return None
+        try:
+            return self.open_project(file.read_text(encoding="utf-8").strip())
+        except (OSError, ValueError, sqlite3.Error):
+            self._remember_last_project(None)
+            return None
 
     def media_dirs(self) -> dict[str, str]:
         if self._folder is None:

core/src/inline_core/studio/training.py

@@ -71,6 +71,42 @@ def list_items(self, dataset_id: str) -> list[dict[str, Any]]:
     def add_items(self, dataset_id: str, asset_ids: list[str]) -> list[dict[str, Any]]:
         return ts.add_items(self._conn(), dataset_id, asset_ids)
 
+    def add_from_path(self, dataset_id: str, path: str) -> list[dict[str, Any]]:
+        """Import a folder of images and clips into the dataset, captions included.
+
+        The browser cannot hand over a folder, and uploading a clip dataset through it means
+        pushing gigabytes over HTTP to a server that can already see the disk. Paths come from the
+        client here the same way ``assets:importPaths`` already accepts them.
+        """
+        from . import assets as ax
+
+        folder = Path(path).expanduser()
+        if not folder.is_dir():
+            raise ValueError(f"Not a folder: {path}")
+        conn, project = self._conn(), self._store.folder()
+        media = [
+            p
+            for p in sorted(folder.iterdir())
+            if p.is_file() and ax.kind_for_file(str(p)) in ("image", "video")
+        ]
+        if not media:
+            raise ValueError(f"No images or clips in {path}")
+
+        imported = [(p, ax.import_file(conn, project, str(p), None)) for p in media]
+        added = ts.add_items(conn, dataset_id, [a["id"] for _p, a in imported if a])
+
+        # `NNNN.txt` beside `NNNN.png` is the caption, the convention the drag-drop path already
+        # follows. Only newly added items are touched, so re-importing cannot clobber an edit.
+        by_asset = {item["assetId"]: item for item in added}
+        for source, asset in imported:
+            item = by_asset.get(asset["id"]) if asset else None
+            sidecar = source.with_suffix(".txt")
+            if item and sidecar.is_file():
+                caption = sidecar.read_text(encoding="utf-8").strip()
+                if caption:
+                    ts.set_caption(conn, item["id"], caption)
+        return ts.list_items(conn, dataset_id)
+
     def remove_item(self, item_id: str) -> None:
         ts.remove_item(self._conn(), item_id)
 

core/src/inline_core/training/arch.py

@@ -333,6 +333,22 @@ def _h3_forward(transformer: Any, noisy: Any, timestep: Any, item: dict[str, Any
 }
 
 
+def clip_frames(arch: TrainingArch, seconds: Any) -> int:
+    """How many frames of a clip to train on, snapped to the arch's frame grid.
+
+    1 for an arch with no clip support, which is what a still costs. For H3 the floor is a whole
+    17-frame chunk plus the 5-frame head, so a shorter request rounds up to 0.92s rather than being
+    refused; the VAE has no way to encode less.
+    """
+    if arch.key != MINIMAX_H3:
+        return 1
+    from ..models.minimaxh3.vendor.packing import MINIMAX_H3_FPS
+    from ..models.minimaxh3.vendor.packing_ref2va import trim_reference_num_frames
+
+    wanted = round(float(seconds) * MINIMAX_H3_FPS) if seconds else 1
+    return trim_reference_num_frames(max(1, wanted))
+
+
 def get(key: str | None) -> TrainingArch:
     """The arch to train. Defaults to Z-Image so a run predating Krea 2 still resumes."""
     arch = ARCHS.get(key or Z_IMAGE)

core/src/inline_core/training/cache.py

@@ -7,6 +7,7 @@
 
 from __future__ import annotations
 
+from collections.abc import Callable
 from typing import Any
 
 from . import arch as archs
@@ -28,13 +29,21 @@ def build(
     *,
     flip: bool = False,
     dropout: float = 0.0,
+    clip_frames: int = 1,
+    on_status: Callable[[str], None] | None = None,
 ) -> tuple[list[dict[str, Any]], dict[str, Any] | None, float]:
-    """Return ``(items, unconditional, shift)``, all as CPU tensors, with the encoders freed."""
+    """Return ``(items, unconditional, shift)``, all as CPU tensors, with the encoders freed.
+
+    ``on_status`` reports phase progress to the caller, which forwards it over the JSON protocol.
+    Precaching a large dataset takes minutes, and a logger call would be dropped here: the trainer
+    subprocess configures no logging handler, so anything below WARNING goes nowhere.
+    """
     if arch == archs.MINIMAX_H3:
         from . import h3
 
         items, unconditional = h3.precache(
-            dataset_dir, models_dir, device, dtype, resolution, flip, dropout > 0
+            dataset_dir, models_dir, device, dtype, resolution, flip, dropout > 0, clip_frames,
+            on_status=on_status,
         )
         return items, unconditional, _H3_SHIFT
 

core/src/inline_core/training/caption.py

@@ -105,13 +105,30 @@ def _load_with_fallback(model_id: str) -> tuple[Any, Any, Any]:
             raise first from None
 
 
+def _open(path: str) -> Any:
+    """The frame to caption. A clip is captioned from its middle frame, which is more
+    representative than the first and stops PIL raising on a container it cannot read."""
+    from pathlib import Path
+
+    from PIL import Image
+
+    from . import dataset as ds
+
+    if not ds.is_video(Path(path)):
+        return Image.open(path).convert("RGB")
+
+    from ..models.minimaxh3.vendor.packing_ref2va import decode_reference_video
+
+    frames, _fps, _audio = decode_reference_video(path)
+    return Image.fromarray(frames[len(frames) // 2]).convert("RGB")
+
+
 def _caption_one(model: Any, processor: Any, device: str, path: str) -> str:
     """One caption. Handles both shapes: task-token models (Florence-2, which post-processes a
     tagged string) and plain image-captioning models (BLIP), which just decode the output."""
     import torch
-    from PIL import Image
 
-    image = Image.open(path).convert("RGB")
+    image = _open(path)
     task_style = hasattr(processor, "post_process_generation")
     inputs = (
         processor(text=_TASK, images=image, return_tensors="pt")