extract ~/-expanding expandHome to fileUtils + add HF download --local-dir / --ignore flags

PLAN.md

@@ -67,6 +67,7 @@ Items here need a research / design pass, an explicit decision, or a preconditio
 - [ ] [video-resume-restore-keyframes] **VideoGen resume: restore keyframes from the active job.** `ACTIVE_JOB_PARAM_FIELDS` in `server/routes/videoGen.js` intentionally omits `keyframes` (gallery filename + frame-index pairs for multi-keyframe FFLF) because the v1 resume effect doesn't repopulate the picker UI for them. A user who reloads mid-render on a keyframe job loses the keyframe-picker state even though every other form field is restored. Whitelist `keyframes` in the route AND wire a `setKeyframes()`-style setter in `client/src/pages/VideoGen.jsx`'s resume effect. Deferred from video-reload PR — keyframes UI restoration is independent of the SSE re-attach.
 - [ ] [unify-media-job-sse-hook-imagegen-videogen] **Unify ImageGen + VideoGen SSE attach into `useMediaJobSse` hook.** Both pages now have near-duplicate EventSource handlers (ImageGen `startLocalGeneration` lines 653-708 + resume block lines 317-328; VideoGen `attachJobEvents`). Build a hook on top of `useSseProgress` (`client/src/hooks/useSseProgress.js`) that handles the queued / started / status / progress / complete / error / canceled SSE types and consolidate both sites. Deferred from the video-reload PR — scope was just the bug fix.
 - [ ] [whole-episode-audio-generation-strategy-stop] **Whole-episode audio generation strategy.** Stop relying on per-clip audio; drive audio gen from episode-level prose/script arc. Generator candidates: Suno (commercial, duration control), MusicGen-MLX (local, bounded ~30s), AudioLDM2. New `audioMode: 'per-clip' | 'silent' | 'generated' | 'uploaded-track'`. Treat as a new sub-brainstorm when picked up — investigation first.
+- [ ] [ltx2-fflf-skips-last-image-resize-when-both-frames-set] **ltx2 FFLF skips last-image resize when both start and end frames are provided.** `server/services/videoGen/local.js:571` gates `lastImageWillBeUsed` on `mode === 'fflf' && !sourceImagePath` so for the ltx2 runtime's true-FFLF flow (both `sourceImagePath` AND `lastImagePath` set — accepted at `buildLtx2Args` line 233 and passed via `--last-image` at line 337) the end-frame image bypasses the `resizeImage` ffmpeg pass at line 601. Either size-pad the last image (matching the start-image resize), or confirm `scripts/generate_ltx2.py` itself resizes/letterboxes its `--last-image` input before conditioning. Surfaced by gemini's `/do:review` pass against PR #526 (extract-expandhome-helper, 2026-05-29) — UNVERIFIED that this actually causes generation failures; the ltx2 helper script may already handle non-matching dimensions. Out-of-scope from PR #526 (that PR only touches the `expandHome` helper + HF download flags).
 
 ### Large product / UX redesigns
 

scripts/hf_download_repo.py

@@ -24,6 +24,7 @@
 """
 
 import argparse
+import inspect
 import os
 import sys
 from pathlib import Path
@@ -41,11 +42,24 @@
     sys.exit(2)
 
 
+# `hf_hub_download(..., local_dir=...)` defaulted to populating `local_dir`
+# via symlinks into the HF cache on huggingface_hub < 0.23, which would
+# break BYOV installers (HiDream-O1) that need real on-disk files. Force
+# real copies with `local_dir_use_symlinks=False`. Newer huggingface_hub
+# (>= 0.23) deprecated the kwarg and always copies, eventually removing
+# it — probe the signature so we only pass it where it's still accepted.
+_HF_DOWNLOAD_ACCEPTS_SYMLINK_KWARG = (
+    "local_dir_use_symlinks" in inspect.signature(hf_hub_download).parameters
+)
+
+
 def main() -> int:
     parser = argparse.ArgumentParser(description="Pre-fetch a HuggingFace repo snapshot.")
     parser.add_argument("--repo", required=True, help="HF repo id, e.g. 'org/name'.")
     parser.add_argument("--revision", default=None, help="Optional revision (branch / tag / sha).")
     parser.add_argument("--token-env", default=None, help="Env var name to read the HF token from (e.g. HF_TOKEN).")
+    parser.add_argument("--local-dir", default=None, help="If set, materialize the repo as a flat copy at this dir instead of relying on the standard HF cache symlinks (used by BYOV installers like HiDream-O1 that need a real on-disk repo).")
+    parser.add_argument("--ignore", action="append", default=[], help="Glob pattern (fnmatch) to skip from the file list. Repeat for multiple patterns. e.g. --ignore 'scripts/**' --ignore 'docs/**' to skip non-weight subdirs.")
     args = parser.parse_args()
 
     token = None
@@ -81,6 +95,9 @@ def main() -> int:
     # as part of a snapshot (`.gitattributes` is, but `LICENSE` and similar
     # are — we keep them; the only true skip is the `.huggingface` folder).
     files = [f for f in files if not f.startswith(".huggingface/")]
+    if args.ignore:
+        import fnmatch
+        files = [f for f in files if not any(fnmatch.fnmatch(f, pat) for pat in args.ignore)]
     total = len(files)
     if total == 0:
         print(f"USER_ERROR:repo_empty:{args.repo}", file=sys.stderr, flush=True)
@@ -94,13 +111,20 @@ def main() -> int:
         # drives progress in either pipeline).
         print(f"STAGE:download:{i}/{total}:{filename}", file=sys.stderr, flush=True)
         print(f"DOWNLOAD:{i}/{total}:{filename}", file=sys.stderr, flush=True)
+        download_kwargs = {
+            "repo_id": args.repo,
+            "filename": filename,
+            "revision": args.revision,
+            "token": token,
+            "local_dir": args.local_dir,
+        }
+        # Only force copies when the caller actually asked for a flat
+        # on-disk layout — without `--local-dir`, hf_hub_download writes
+        # into the standard HF cache where symlinks are the contract.
+        if args.local_dir and _HF_DOWNLOAD_ACCEPTS_SYMLINK_KWARG:
+            download_kwargs["local_dir_use_symlinks"] = False
         try:
-            resolved = hf_hub_download(
-                repo_id=args.repo,
-                filename=filename,
-                revision=args.revision,
-                token=token,
-            )
+            resolved = hf_hub_download(**download_kwargs)
         except GatedRepoError:
             print(f"USER_ERROR:gated_repo:{args.repo}", file=sys.stderr, flush=True)
             print(f"❌ {args.repo} is gated. Accept the license + paste your HF token.", file=sys.stderr, flush=True)

server/lib/README.md

@@ -90,7 +90,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
 | `collectionStore.js` | Per-type, per-record JSON storage with explicit type-level `schemaVersion` stamping. Use for collections that have outgrown a monolithic JSON file. `createCollectionStore({ dir, type, schemaVersion, sanitizeRecord })` returns `loadOne` / `saveOne` / `saveOneNow` / `listIds` / `loadAll` / `deleteOne` / `loadTypeIndex` / `saveTypeIndex` / `verifySchemaVersion`. Per-id write queue means writes to different records don't serialize; `saveOneNow` is for callers already inside a collection write queue. Boot-time `verifyCollectionVersions([store, ...])` logs schema-version mismatches. |
 | `conflictJournal.js` | Non-blocking edit-conflict journal for cross-install LWW merges. `maybeJournalBeforeOverwrite({kind,id,local,remote,source})` (call right before a merge overwrite) archives the losing local version when a true 3-way divergence is detected (`detectConflict` via per-record `syncBaseHash` + `contentHashForRecord`), then advances the base hash; `flushBaseHashes()` persists the batched base-hash side store. `deleteSyncBaseHash(kind,id)` evicts a record's base hash when its tombstone is hard-pruned (called from `pruneTombstonedUniverses`/`pruneTombstonedSeries`) so the side store doesn't grow without bound. `conflictJournalStore()` is the `pending`/`resolved` entry store (discard resolves an entry; DELETE hard-removes it — there is no `dismissed` status). Local-only — never crosses the wire. |
 | `schemaVersions.js` | Cross-instance sync version contract. `PORTOS_SCHEMA_VERSIONS` (frozen map of `{ category: layoutVersion }`), `RECORD_KIND_SCHEMA_CATEGORIES` (frozen map of federated record kind → the schema categories it writes), `buildPortosMeta()` (envelope for every outbound sync payload), `compareSchemaVersions(sender, receiver)` returning `{ ahead, behind, compatible }`, `scopeVersionDiff(diff, categories)` (restrict that diff to the categories a specific transfer touches), and `formatVersionGap()` for UI/log lines. Receivers gate `applyIncomingPush` / share-bucket import / snapshot apply per-category on the scoped comparator result so an upgraded sender can't corrupt a downstream peer — and a bump to one category doesn't sever sync of the others. |
-| `fileUtils.js` | `PATHS` constants, `atomicWrite`, `tryReadFile`, `safeJSONParse`, JSONL append/read/write helpers, dir scans, hashes, JSON helpers. Most paths/file work goes through here. |
+| `fileUtils.js` | `PATHS` constants, `atomicWrite`, `tryReadFile`, `safeJSONParse`, `expandHome` (`~/foo` → absolute), JSONL append/read/write helpers, dir scans, hashes, JSON helpers. Most paths/file work goes through here. |
 | `fileWriteQueue.js` | Single-tail promise chain for serializing writes to a file. |
 | `imageClean.js` | `cleanImageBuffer` (sharp-based denoise + C2PA strip) + `autoCleanGeneratedImage` (in-place clean for post-generation hook). HTTP route in `routes/imageClean.js` wraps `cleanImageBuffer`. |
 | `multipart.js` | Streaming multipart/form-data parser. |

server/lib/fileUtils.js

@@ -182,6 +182,17 @@ export function dataPath(...segments) {
   return join(PATHS.data, ...segments);
 }
 
+// `path.join(homedir(), '/.foo')` discards the homedir prefix because of
+// the leading slash, so strip the leading `~/` (or `~\` on Windows) before
+// joining. Only expands a leading `~` — embedded `~` chars in path segments
+// (e.g. `iCloud~md~obsidian`) are preserved.
+export function expandHome(p) {
+  if (typeof p !== 'string' || !p) return p;
+  if (p === '~') return homedir();
+  if (p.startsWith('~/') || p.startsWith('~\\')) return join(homedir(), p.slice(2));
+  return p;
+}
+
 /**
  * Check if a string is potentially valid JSON.
  * Performs quick structural validation before parsing.

server/lib/fileUtils.test.js

@@ -2,10 +2,11 @@ import { describe, it, expect, vi, beforeEach, afterEach, afterAll } from 'vites
 import { readFile, writeFile, rm, mkdir } from 'fs/promises';
 import { mkdtempSync, rmSync, writeFileSync } from 'fs';
 import { join } from 'path';
-import { tmpdir } from 'os';
+import { tmpdir, homedir } from 'os';
 import { createHash } from 'crypto';
 import {
   assertSafeFilename,
+  expandHome,
   isValidJSON,
   listDirectoryByExtension,
   safeJSONParse,
@@ -697,4 +698,39 @@ describe('fileUtils', () => {
       expect(await sha256File(a)).not.toBe(await sha256File(b));
     });
   });
+
+  describe('expandHome', () => {
+    it('expands a bare `~` to the homedir', () => {
+      expect(expandHome('~')).toBe(homedir());
+    });
+
+    it('expands `~/foo` to homedir + foo', () => {
+      const out = expandHome('~/foo');
+      expect(out.startsWith(homedir())).toBe(true);
+      expect(out.endsWith('foo')).toBe(true);
+    });
+
+    it('expands the Windows form `~\\foo` so `lib/fileUtils.js` stays cross-platform', () => {
+      const out = expandHome('~\\foo');
+      expect(out.startsWith(homedir())).toBe(true);
+      expect(out.endsWith('foo')).toBe(true);
+    });
+
+    it('preserves absolute, relative, and empty inputs', () => {
+      expect(expandHome('/abs/path')).toBe('/abs/path');
+      expect(expandHome('relative/path')).toBe('relative/path');
+      expect(expandHome('')).toBe('');
+    });
+
+    it('preserves non-string inputs (null / undefined / number) without throwing', () => {
+      expect(expandHome(null)).toBe(null);
+      expect(expandHome(undefined)).toBe(undefined);
+      expect(expandHome(42)).toBe(42);
+    });
+
+    it('only expands a leading `~` — embedded `~` chars are preserved (iCloud~md~obsidian)', () => {
+      expect(expandHome('iCloud~md~obsidian')).toBe('iCloud~md~obsidian');
+      expect(expandHome('foo/~bar')).toBe('foo/~bar');
+    });
+  });
 });

server/lib/mediaModels.js

@@ -16,9 +16,8 @@
  */
 
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'fs';
-import { homedir } from 'os';
 import { join, dirname } from 'path';
-import { PATHS } from './fileUtils.js';
+import { PATHS, expandHome } from './fileUtils.js';
 import { isPlainObject } from './objects.js';
 import { RUNNER_FAMILIES } from './runners.js';
 // fileUtils.ensureDir is async/Promise-returning; this module needs a
@@ -276,15 +275,6 @@ const DEFAULT_REGISTRY = {
   selectedTextEncoder: 'gemma-bf16',
 };
 
-// `path.join(homedir(), '/.foo')` discards the homedir because of the
-// leading slash, so we have to strip the `~/` prefix (or `~`) before joining.
-const expandHome = (p) => {
-  if (!p || !p.startsWith('~')) return p;
-  if (p === '~') return homedir();
-  if (p.startsWith('~/')) return join(homedir(), p.slice(2));
-  return p;
-};
-
 let cached = null;
 
 const ensureDir = (file) => {

server/lib/runners.test.js

@@ -3,17 +3,19 @@ import { readFileSync } from 'fs';
 import { dirname, join } from 'path';
 import { fileURLToPath } from 'url';
 
-import { RUNNER_FAMILIES, isMflux, isFlux2, isZImage, isErnie } from './runners.js';
+import { RUNNER_FAMILIES, isMflux, isFlux2, isZImage, isErnie, isHiDream, isQwen } from './runners.js';
 
 const __dirname_self = dirname(fileURLToPath(import.meta.url));
 const CLIENT_MIRROR_PATH = join(__dirname_self, '..', '..', 'client', 'src', 'lib', 'runnerFamilies.js');
 
 describe('RUNNER_FAMILIES', () => {
-  it('exports the four canonical runner ids', () => {
+  it('exports the canonical runner ids', () => {
     expect(RUNNER_FAMILIES.MFLUX).toBe('mflux');
     expect(RUNNER_FAMILIES.FLUX2).toBe('flux2');
     expect(RUNNER_FAMILIES.Z_IMAGE).toBe('z-image');
     expect(RUNNER_FAMILIES.ERNIE).toBe('ernie');
+    expect(RUNNER_FAMILIES.HIDREAM).toBe('hidream');
+    expect(RUNNER_FAMILIES.QWEN).toBe('qwen');
   });
 
   it('is frozen so callers can\'t mutate the canonical strings at runtime', () => {
@@ -30,13 +32,17 @@ describe('RUNNER_FAMILIES', () => {
     expect(text).toMatch(/FLUX2:\s*'flux2'/);
     expect(text).toMatch(/Z_IMAGE:\s*'z-image'/);
     expect(text).toMatch(/ERNIE:\s*'ernie'/);
+    expect(text).toMatch(/HIDREAM:\s*'hidream'/);
+    expect(text).toMatch(/QWEN:\s*'qwen'/);
   });
 
   it('predicate helpers match on the canonical runner ids', () => {
     expect(isMflux({ runner: 'mflux' })).toBe(true);
     expect(isFlux2({ runner: 'flux2' })).toBe(true);
     expect(isZImage({ runner: 'z-image' })).toBe(true);
     expect(isErnie({ runner: 'ernie' })).toBe(true);
+    expect(isHiDream({ runner: 'hidream' })).toBe(true);
+    expect(isQwen({ runner: 'qwen' })).toBe(true);
     expect(isFlux2({ runner: 'mflux' })).toBe(false);
     expect(isFlux2(null)).toBe(false);
     expect(isFlux2(undefined)).toBe(false);

server/routes/scaffold.js

@@ -5,10 +5,10 @@ import { join, dirname, resolve, sep } from 'path';
 import { fileURLToPath } from 'url';
 import { exec, spawn } from 'child_process';
 import { promisify } from 'util';
-import { homedir, platform } from 'os';
+import { platform } from 'os';
 import { createApp, getReservedPorts } from '../services/apps.js';
 import { asyncHandler, ServerError } from '../lib/errorHandler.js';
-import { ensureDir } from '../lib/fileUtils.js';
+import { ensureDir, expandHome } from '../lib/fileUtils.js';
 import { scaffoldVite } from './scaffoldVite.js';
 import { scaffoldExpress } from './scaffoldExpress.js';
 import { scaffoldIOS } from './scaffoldIOS.js';
@@ -31,11 +31,10 @@ router.get('/directories', asyncHandler(async (req, res) => {
   let targetPath;
   if (!dirPath) {
     targetPath = defaultPath;
-  } else if (dirPath === '~') {
-    targetPath = homedir();
-  } else if (dirPath.startsWith('~/') || dirPath.startsWith('~\\')) {
-    // Expand leading ~ only; preserve embedded ~ chars (e.g. iCloud~md~obsidian)
-    targetPath = resolve(join(homedir(), dirPath.slice(2)));
+  } else if (dirPath === '~' || dirPath.startsWith('~/') || dirPath.startsWith('~\\')) {
+    // expandHome covers `~`, `~/foo`, and `~\foo` (Windows); embedded `~`
+    // chars (e.g. `iCloud~md~obsidian`) fall through to the else branch.
+    targetPath = resolve(expandHome(dirPath));
   } else {
     targetPath = resolve(dirPath);
   }

server/services/referenceRepos.js

@@ -20,10 +20,9 @@
  */
 
 import { existsSync, statSync } from 'fs';
-import { homedir } from 'os';
 import { join } from 'path';
 import { v4 as uuidv4 } from '../lib/uuid.js';
-import { ensureDir, PATHS } from '../lib/fileUtils.js';
+import { ensureDir, expandHome, PATHS } from '../lib/fileUtils.js';
 import { ServerError } from '../lib/errorHandler.js';
 import { execGit } from '../lib/execGit.js';
 import {
@@ -32,17 +31,6 @@ import {
 } from './apps.js';
 import { DEFAULT_REVIEWER } from '../lib/validation.js';
 
-// `path.join(homedir(), '/.foo')` discards the homedir because of the leading
-// slash, so we strip the `~/` prefix (or `~`) before joining. Same shape as
-// the helper in lib/mediaModels.js — kept inline here to avoid pulling in the
-// rest of that module just for one helper.
-const expandHome = (p) => {
-  if (!p || typeof p !== 'string' || !p.startsWith('~')) return p;
-  if (p === '~') return homedir();
-  if (p.startsWith('~/')) return join(homedir(), p.slice(2));
-  return p;
-};
-
 const REFERENCE_REPOS_ROOT = join(PATHS.data, 'cos', 'reference-repos');
 
 // 40-char hex SHA the same way git outputs it. Used by every callsite that

server/services/referenceRepos.test.js

@@ -3,8 +3,9 @@ import { describe, it, expect, vi, beforeEach } from 'vitest';
 // ─── mocks (must precede the import under test) ──────────────────────────────
 
 vi.mock('../lib/fileUtils.js', () => ({
-tryReadFile: vi.fn().mockResolvedValue(null),
+  tryReadFile: vi.fn().mockResolvedValue(null),
   ensureDir: vi.fn(async () => {}),
+  expandHome: vi.fn((p) => p),
   PATHS: { data: '/mock/data', root: '/mock/root' },
   readJSONFile: vi.fn(async () => ({ apps: {} })),
 }));

server/services/voice/config.js

@@ -6,6 +6,7 @@ import { homedir } from 'os';
 import { join } from 'path';
 import { getSettings, updateSettings } from '../settings.js';
 import { deepMerge } from '../../lib/objects.js';
+import { expandHome } from '../../lib/fileUtils.js';
 
 const VOICE_HOME = join(homedir(), '.portos', 'voice');
 
@@ -110,12 +111,9 @@ export const VOICE_DEFAULTS = Object.freeze({
   vad: { endOfSpeechMs: 700, minUtteranceMs: 250 },
 });
 
-export const expandPath = (p) => {
-  if (typeof p !== 'string') return p;
-  if (p.startsWith('~/')) return join(homedir(), p.slice(2));
-  if (p === '~') return homedir();
-  return p;
-};
+// Alias kept for backward-compat with the 5 callers under server/services/voice/.
+// The canonical implementation now lives in `server/lib/fileUtils.js#expandHome`.
+export const expandPath = expandHome;
 
 // Tiny in-memory cache so hot paths (per-turn enabled check in voice sockets)
 // don't hit disk on every dispatch. Invalidated on updateVoiceConfig and