Skip to content

Address storyboard scenes by durable id, and cancel the orphaned job when a stage write rejects - #3426

Merged
atomantic merged 2 commits into
mainfrom
claim/issue-3413
Aug 3, 2026
Merged

Address storyboard scenes by durable id, and cancel the orphaned job when a stage write rejects#3426
atomantic merged 2 commits into
mainfrom
claim/issue-3413

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

Closes #3413. Follow-up to #3400 / PR #3411.

Storyboard scenes were addressed purely by array index end to end — the route took sceneIndex, the media-job owner encoded scene<idx>, and (post-#3400) the locked write patched scenes[index]. Two consequences remained:

  1. A reorder/removal between the caller's read and the locked write silently retargeted the write onto whatever scene now occupied that slot whenever the index was still in range (only the out-of-range case 404'd), and the completion hook routed the finished render by the same stale index — so the mis-targeting was consistent rather than self-correcting.
  2. The job is enqueued before the persist, so a rejected write left it queued with nothing referencing it: burning GPU time and leaving an untracked artifact. comicPages.js#persistComicPageSlot and covers.js#renderComicCoverLike had the identical shape.

1. Durable scene ids

server/lib/storyboardScenes.js (new): ensureStoryboardIds(scenes) stamps a missing id on each scene and each shots[] entry; resolveStoryboardTarget(list, { id, index }){ index, record, matchedBy, stale }. sanitizeVisualStage's storyboards branch calls the stamp, so every record grows ids on its next write.

Ids are deterministic (scene-01, shot-02, collision-escaped to scene-01-2) rather than random, on purpose: two concurrent readers of an un-migrated record must derive the same id, and every federated peer runs the backfill over its own copy of a shared issue — random ids would make each peer stamp something different and churn conflicts forever.

2. Backfill migrations (both stores)

  • scripts/migrations/222-storyboard-scene-durable-ids.js — the file-backed store (data/pipeline-issues/{id}/index.json): still live under MEMORY_BACKEND=file/tests, and still the source the one-time migrateIssuesToDB import reads on installs not yet on Postgres, so stamping here first carries ids across the import.
  • server/scripts/db-migrations/007-storyboard-scene-durable-ids.js — the pipeline_issues rows. The file runner executes before the DB pool exists, so a row transform cannot live in scripts/migrations/.

Both are idempotent and deliberately do not bump updated_at/updatedAt — a derived normalization must not advance the LWW clock and out-race real remote edits.

3. Resolve by id inside the write region

patchStoryboardScene now takes { id, index } captured during the caller's read and re-resolves inside updateStageWithLatest. A captured id that is gone is 409 PIPELINE_SCENE_STALE_TARGET (shots: PIPELINE_SHOT_STALE_TARGET), distinct from the existing 404 for an index that never existed; the index is used only when no id was captured. The three callers (scene video, shot start-frame, prompt refine) return the index the write landed on, not the stale read index.

4. Completion path

buildStoryboardsShotOwner now emits …:scene<N>:shot<M>:sid<sceneId>:tid<shotId> (ids percent-encoded so a : inside one can't break the parse), plus a buildStoryboardsSceneOwner for the scene-level video render. parseStoryboardsShotOwner parses legacy index-only owners with sceneId: null, shotId: null, and storyboardsFilenameHook.js resolves by id with an index fallback — so jobs already sitting in the queue at upgrade time still attach.

5. Orphan cancel

persistRenderOrCancel(jobId, persist, label) in visualStageHelpers.js wraps the persist step: on reject it cancelJobs the job it just enqueued, logs the cleanup, and rethrows the original error. Applied to the storyboard scene/shot enqueues, persistComicPageSlot, renderComicCoverLike, and renderVolumeCoverLike.

Test plan

  • cd server && NODE_ENV=test npx vitest run — full suite green (1183 files / 24,369 tests; DB suites skip as designed, no writes to the real portos DB).
  • New/extended coverage, one per Acceptance bullet:
    • server/lib/storyboardScenes.test.js — determinism, collision escape, idempotence-by-reference, id-vs-index resolution, duplicate-id tiebreak, absent-index sentinel.
    • server/services/pipeline/visualStages.test.js — driven through the interleaved-write harness from fix: merge storyboard scene writes against fresh state so sibling renders don't clobber job ids (#3400) #3411: a reorder that keeps the index in range does not move the job id onto the wrong scene (scene and shot level); a scene/shot removed between read and write 409s instead of mis-writing; a rejected write leaves no queued job for that owner (asserted via a listJobs({ owner }) stand-in) and the job is recorded canceled; a pre-migration record with no ids still resolves by index; rejected comic-page and cover writes cancel their jobs too.
    • server/services/pipeline/storyboardsFilenameHook.test.js (new) — id routing after a reorder, legacy index-only owner still attaches, stale-jobId and vanished-scene skips.
    • server/services/pipeline/owners.test.js — new owner round-trip, percent-encoding, legacy parse.
    • Migration tests for both 222 and db-007 (backfill, idempotent re-run byte-identical, no LWW bump, empty/unreadable/fresh-install cases).

…write reject (#3413)

Scenes were addressed purely by array index end to end — the route took
sceneIndex, the media-job owner encoded scene<idx>, and the stage write
patched scenes[index]. #3400 closed the clobbering race but left the
mis-targeting one: a reorder or delete landing between the caller's read
and the locked write retargeted the job id onto whatever scene occupied
that slot whenever the index was still in range, and the completion hook
routed the finished render by the same stale index.

Scenes (and shots) now carry a durable, deterministic id stamped by
sanitizeVisualStage and backfilled by migration 222 (file store) plus
db-migration 007 (Postgres). Deterministic rather than random so two
concurrent readers of an un-migrated record — and two federated peers
running the backfill independently — derive identical ids. Every write
region re-resolves the captured id against the fresh array and 409s
(PIPELINE_SCENE_STALE_TARGET / PIPELINE_SHOT_STALE_TARGET) when it is
gone, distinct from the 404 for an index that never existed; the index
stays as the fallback only for pre-migration records. Owners carry the
ids percent-encoded, and the filename hook still attaches jobs queued
under the legacy index-only owner format.

The render surfaces all enqueue the job before persisting it, so a
rejected write left the job queued with nothing referencing it — burning
GPU time for an artifact nothing points at. persistRenderOrCancel wraps
the persist step for storyboard scene/shot renders, comic page slots, and
issue/volume covers: on reject it cancels the job it just enqueued and
rethrows the original error.
…malization (#3413)

Review findings on the first pass:

- The file-store migration rewrote each issue with a truncating writeFile. An
  interrupted rewrite would leave unparseable JSON that the next pass silently
  skips — permanent loss for that issue, since the migration is already marked
  applied. Uses atomicWrite (temp + rename) instead.
- A list that ALREADY carried duplicate ids kept them: the resolver tie-breaks
  on the captured index, so once two same-id entries are reordered the write
  lands on the wrong one — the exact mis-target this change prevents. Later
  copies are now re-stamped; the first occurrence keeps the id, matching
  sanitizeShots' first-wins resolution of continuityFromShotId.
@atomantic
atomantic merged commit 8e8f94e into main Aug 3, 2026
6 checks passed
@atomantic
atomantic deleted the claim/issue-3413 branch August 3, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Address storyboard scenes by durable id, and cancel the orphaned job when a stage write rejects

1 participant