feat(studio): drag assets from the sidebar onto the timeline

[miguel-heygen]

Problem

Studio still broke down in three concrete authoring flows around timeline assets:

• you could import media into Assets, but not drag an already-imported asset from the Assets tab onto the timeline and persist it into source
• dragging a file from outside the app onto the timeline only uploaded it into Assets instead of placing it at the dropped time/track
• once a clip was on the timeline, there was no reliable keyboard delete flow for removing it safely from source

While implementing direct external drops, another real bug showed up:

• valid binary uploads like raycast.mp4 from Downloads were being rejected as unsupported media in Studio dev because the Vite API bridge was corrupting multipart request bodies before they reached the upload route

What this fixes

Timeline asset placement from inside Studio

• asset cards in the Assets tab are draggable
• the timeline accepts asset drops even when it already has clips
• dropping an asset onto the timeline inserts a new clip into the active composition source at the dropped time / track
• asset paths are rewritten relative to the target composition file so drops into sub-compositions resolve correctly
• the new clip is persisted immediately and the preview refreshes

Direct external file drops onto the timeline

• dropping a file from outside the app onto the timeline now uploads it and places it onto the dropped track/time in one shot
• it no longer stops halfway by only adding the file into Assets
• multiple dropped files are placed using the same drop start and successive tracks

Delete key support

• selected timeline clips can now be deleted with Delete / Backspace
• deletion is persisted back to source, not just removed from local state
• the delete path now uses a server-side DOM mutation helper with LinkeDOM for structural safety instead of client-side string surgery

Binary upload fix for media files

• the Studio Vite API bridge now forwards non-GET request bodies as raw bytes instead of decoding them as UTF-8 text
• that preserves multipart uploads for binary media like MP4s
• valid local videos from Downloads no longer get rejected as Unsupported media skipped just because the dev bridge corrupted the request body
• upload validation now probes buffered media through a temp file path that preserves the file extension before saving into the project

Root cause

There were really two separate gaps:

1. Asset placement / deletion workflow gaps

The timeline and asset systems already existed, but they were disconnected:

• AssetsTab only supported copy/import flows
• Timeline only handled raw file import, not positioned placement for existing assets
• there was no utility layer for converting a dropped asset into persisted timeline HTML
• there was no structurally safe deletion path for arbitrary selected timeline clips

2. Binary upload corruption in Studio dev

The Studio Vite API bridge rebuilt non-GET request bodies like this:

• read each request chunk
• call chunk.toString()
• concatenate into a string
• construct the Fetch Request from that string body

That works for text, but it corrupts multipart binary uploads. By the time the upload route wrote the received file and ran ffprobe, otherwise valid MP4s had already been mangled in-flight.

Behavior

• dropping on index.html inserts the asset into the root composition
• dropping while drilled into a composition inserts into that composition file instead
• drop X position maps to data-start
• drop Y position maps to the current visible track row, with a new bottom track created if the drop lands below existing rows
• images default to a short finite duration
• audio/video default to their metadata duration when available, with a fallback duration if metadata cannot be read quickly
• pressing Delete on a selected clip removes that clip from the underlying HTML source and clears selection in Studio
• valid uploaded MP4s now survive the Studio dev API bridge intact instead of being rejected during upload validation

Verification

Local checks

• bunx oxlint packages/core/src/studio-api/helpers/sourceMutation.ts packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/core/src/studio-api/routes/files.ts packages/studio/src/App.tsx packages/studio/src/components/nle/NLELayout.tsx packages/studio/src/components/sidebar/AssetsTab.tsx packages/studio/src/player/components/Timeline.tsx packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.config.ts packages/studio/vite.request-body.ts packages/studio/vite.request-body.test.ts
• bunx oxfmt --check on the touched files
• bun run --filter @hyperframes/core typecheck
• bun run --filter @hyperframes/studio typecheck
• bun test packages/core/src/studio-api/helpers/sourceMutation.test.ts packages/core/src/studio-api/helpers/mediaValidation.test.ts packages/studio/src/player/components/Timeline.test.ts packages/studio/src/utils/timelineAssetDrop.test.ts packages/studio/vite.request-body.test.ts

Browser / live verification

Verified against a live local Studio fixture:

• dragging an existing asset from the Assets tab onto the timeline creates a persisted clip at the dropped position
• dropping a file from outside the app directly onto the timeline uploads it and creates a persisted clip at the dropped position
• selecting a dropped clip and pressing Delete removes it from both the live timeline and the saved source HTML
• valid MP4 uploads like raycast.mp4 now succeed through the live Studio upload route instead of being rejected as unsupported media

Notes

• the local timeline-trio-verify and timeline-overlap-debug projects used for verification are local-only and are not part of this PR
• this PR is about asset placement, upload correctness, and deletion safety; it does not broaden into richer editing workflows beyond placing/removing clips from the timeline

[jrusso1020]

Verdict: approve on code merit — wait on regression-shards before merge

Clean architecture. The split between pure helpers (timelineAssetDrop.ts, sourceMutation.ts) and integration (App.tsx, Timeline.tsx) is the right shape, and the test coverage targets exactly the helpers that define the critical invariants — getTimelineAssetKind / buildTimelineAssetId / resolveTimelineAssetSrc / buildTimelineAssetInsertHtml / insertTimelineAssetIntoSource on the asset-drop side, shouldHandleTimelineDeleteKey / getDefaultDroppedTrack / resolveTimelineAssetDrop on the timeline interaction side, and removeElementFromHtml on the server-side mutation side.

What this closes from the Loom feedback

hf#463 explicitly deferred Jake's #2 (scene expansion) and #3 (drag-drop imports). This PR ships #3 cleanly + adds Delete/Backspace timeline clip removal (a natural companion that falls out of the same source-mutation plumbing). That's a tight, complete follow-up.

Architecture wins worth calling out

• Server-side DOM mutation via LinkeDOM (removeElementFromHtml) — right call over client-side string surgery. Handles full-doc HTML vs fragment correctly (parseSourceDocument sniffs <!doctype|<html and wraps fragments into a synthetic document, then returns body.innerHTML on fragments or document.toString() on full docs). No-op semantics when the target isn't found (returns source unchanged) is the right defensive shape.
• Custom MIME type (TIMELINE_ASSET_MIME = "application/x-hyperframes-asset") on the asset drag payload prevents collision with generic file drops. Timeline.handleAssetDragOver correctly short-circuits on neither-files-nor-asset, then branches dropEffect = "copy" only on asset.
• Delete in-flight guard (deleteInFlightRef) prevents rapid-fire double-delete. suppressClickRef = true after delete prevents a spurious click from reselecting the deleted-now-nonexistent clip.
• resolveTimelineAssetSrc handles sub-composition path rewriting (asset dropped in compositions/scene-a.html gets path rewritten to ../assets/photo.png) — one of those things that silently breaks authoring if wrong.
• z-index rewriting on drop and delete keeps the layer ordering monotonic per-track without holes. O(n-per-track) work per mutation is fine at typical composition sizes.

Non-blocking notes

1. Regex-based collectHtmlIds. \bid="([^"]+)" catches id="..." on elements, but it also catches data-id="..." / aria-describedby-id="..." / any attribute whose name ends in id. That means buildTimelineAssetId's dedupe may treat a non-id attribute value as an "existing id" and suffix unnecessarily. Defensive-wrong (adds an extra _2 suffix that wouldn't have been needed) rather than correctness-wrong. A tiny tightening to \s+id="([^"]+)" or (?:^|\s)id="([^"]+)" would avoid the false-positive class. Minor.
2. Delete-key window listener composes with other global handlers. event.preventDefault() stops the browser's default (Backspace→navigate-back) but doesn't stopPropagation(), so if another window-level keydown listener exists (file-tree row, command palette, etc.) it still fires. Consider adding event.stopPropagation() after preventDefault() once you confirm no downstream handler actually wants to chain. Not blocking since the shouldHandleTimelineDeleteKey input/contenteditable guard handles the most common cases.
3. resolveDroppedAssetDuration 3-second timeout. Reasonable default. On slow network or cold disk, an honest probe could take longer than 3s and fall back to the 5-second video default, producing a clip that doesn't match source duration. User can trim afterwards; the current default is a sensible UX tradeoff.
4. insertTimelineAssetIntoSource throws if no data-composition-id root — good error but an unguarded throw could surface as an unhandled rejection in handleTimelineAssetDrop which would only land in the showToast catch if wrapped. Traced the call site; it IS wrapped in the try/catch, so toast-message path handles it. ✓

Browser-test ask — same capability gap as #424/#463

My session doesn't have Playwright/Puppeteer wired in, so I can't run the interactive drag-drop or Delete-key verification. Your Loom screenshot + webm capture is what we have; the pure-helper test coverage + CI green on everything non-regression-shard is strong code-merit signal. For the final pass (especially the z-index-rewrite under multi-clip tracks, and the path-rewrite under nested sub-compositions), a manual human-at-browser check is still the right gate.

CI state

Format / Typecheck / Lint / Test / Build / Test: runtime contract / Smoke: global install / Tests on windows-latest / Render on windows-latest / CodeQL / player-perf / Perf (fps/scrub/drift/parity/load) — all green. Regression-shards (8 of 10) still in-flight at review time — hold merge until those complete.

No blockers. Ship once regression is green.

---

Review by hyperframes

[jrusso1020]

Follow-up on the 4 new commits since initial approval

Took another pass over 82a70a0 (external drops), 461ed9a (media-validation), 67fceaf (vite body-bridge), and 9754c3b (live child-timeline duration). Verdict: all four are clean, well-scoped, and each is clearly motivated — approval from the prior review carries through on my end. Holding final merge on regression-shards completing.

Per-commit notes

9754c3b — live child timelines for runtime visibility (the regression fix)

Nicely targeted. Parameterizing includeAuthoredTimingAttrs on resolveDurationForElement and flipping it to false at the composition-host visibility site is the minimum-blast-radius shape. Everywhere else (data-composition-id root, captions, etc.) keeps trusting data-hf-authored-duration; only the visibility-cutoff path now prefers the live child timeline's .duration(). The test covers the exact invariant that broke: authored=14, live=8, renderSeek(9) → hidden. ✓

Subtle thing worth being aware of: if a child's registered timeline happens to be mis-sized (e.g. a paused tl with duration 0 because tweens haven't been added yet), visibility will cut earlier than the author expected. The current resolver.resolveDurationForElement implementation presumably handles null/0 via the authored fallback, but a line of comment over this call site noting "we intentionally prefer live over authored here because authored lies when sub-compositions shrink dynamically" would save a future reader the five-minute archaeology. Not blocking.

67fceaf — preserve binary request bodies in the Vite bridge

Correct fix for the right reason. chunk.toString() was silently doing UTF-8 decoding of multipart boundaries mid-binary-payload; Buffer.concat preserves bytes. The readNodeRequestBody helper's typeof chunk === "string" branch is defensive paranoia (Node IncomingMessage yields Buffers unless .setEncoding() is called), but it's harmless and future-proofs against a transform-stream wrapper. Test covers both the binary-preservation and empty-body paths. ✓

One small thing: body = bytes.byteLength > 0 ? bytes : undefined; in vite.config.ts collapses "empty body" to undefined before handing to new Request(...). That matches the old string-based behavior (empty string → undefined) so no behavior change, but if any downstream route expected an empty-but-present body (e.g. a POST with Content-Length: 0 that was still meaningful), this would swallow it. In practice, multipart uploads and JSON POSTs always have bytes, so the tradeoff is fine.

461ed9a — reject invalid media uploads

Good split: validateUploadedMedia (path-based) + validateUploadedMediaBuffer (buffer-based with a tempfile) — the buffer variant's basename(fileName) safely strips path traversal, and the /tmp/hyperframes-upload-* temp dir is cleaned up in finally. The ENOENT-on-ffprobe early-return-OK is the right call for dev boxes without ffmpeg; production should have ffmpeg available and will get the real stream check.

The surfaced-to-UI error text ("Unsupported media skipped: ...") in App.tsx's upload handler matches the PR-description wording. ✓

One nit: on an ffprobe status-nonzero failure you return "ffprobe failed to read the media file" without including the stderr or exit code. For a user who's dropped an unsupported codec (e.g., ProRes RAW), the toast says "ffprobe failed" and they're left guessing. Not blocking for this PR — later we can pass the first line of result.stderr into the reason string.

82a70a0 — external file drops onto timeline

Reasonable shape. buildTimelineFileDropPlacements (pure, testable) plus uploadProjectFiles + resolveDroppedAssetDuration composed in App.tsx. The 3-second metadata-probe timeout keeps the drop interaction snappy; the fallback to DEFAULT_TIMELINE_ASSET_DURATION[kind] on timeout / error / non-finite duration is sensible UX.

One concern worth calling out (carry-over from my initial review, still stands on the new code):

• collectHtmlIds regex \bid="([^"]+)" is the same false-positive source. It still catches data-id, aria-describedby-id, etc. It's used in buildTimelineAssetId's dedupe, so the worst case is defensive-wrong (an unnecessary _2 suffix), not correctness-wrong. A tighten to (?:^|\s|<\w[^>]*\s)id="([^"]+)" or a LinkeDOM pass would kill the false-positive class. Low priority but worth putting on the cleanup list since this regex is now used on both drop-from-assets and drop-from-external paths.

Another small observation:

• *media.src = \/api/projects/${projectId}/preview/${assetPath}`;* in resolveDroppedAssetDuration—assetPathis user-controllable file name (post-upload), and path segments with spaces or# would break URL parsing. The upload route presumably sanitizes names (finalNameinfiles.ts) but a defensive encodeURI(assetPath)` here would be cheap insurance. Very low priority.

CI state (as of this comment)

All non-regression checks green on 9754c3b:

• Format / Typecheck / Lint / Test / Build / Test: runtime contract / Smoke / Tests on windows-latest / Render on windows-latest / CodeQL / player-perf / Perf (fps/scrub/drift/parity/load)

Regression-shards (all 10 shards) are in-progress — triggered at 00:27:47Z on the new head. Not blocking yet; will watch those and confirm when they land.

Tl;dr — code looks good, holding merge on regression-shards completing.

— hyperframes