Skip to content

feat(studio): simplify inspector selection UX#692

Merged
miguel-heygen merged 8 commits into
nextfrom
feat/studio-inspector-selection-ux
May 9, 2026
Merged

feat(studio): simplify inspector selection UX#692
miguel-heygen merged 8 commits into
nextfrom
feat/studio-inspector-selection-ux

Conversation

@miguel-heygen
Copy link
Copy Markdown
Collaborator

@miguel-heygen miguel-heygen commented May 9, 2026

Problem

The Studio inspector UX was exposing too much low-confidence editing surface while direct canvas selection was coupled to manual dragging. Users had to discover elements through timeline controls, and timeline layer inspection could fail for master composition clips whose timeline metadata used preview URLs instead of the source paths kept in the DOM.

The downloaded Notion showcase also exposed three runtime/Studio preview issues: the Play button could flip to Pause without the real preview clock advancing, a failed narration.wav source could keep Studio in the asset-loading/disabled-controls state after Chrome had already rejected the media, and clicking the visible “Your 24/7 AI team” artwork looked like a text-selection miss because that copy is baked into the hero background image.

What this fixes

  • Enables canvas click/hover selection by default whenever inspector panels are enabled.
  • Keeps manual canvas movement disabled by default behind VITE_STUDIO_ENABLE_PREVIEW_MANUAL_DRAGGING.
  • Ranks elementsFromPoint() candidates so direct clicks prefer the deepest visible editable child instead of oversized containers.
  • Keeps explicit container selection available through the timeline layer list.
  • Detects direct clicks on large raster/background selections and opens the Ask Agent dialog at the click location, carrying preview click context into the copied prompt.
  • Normalizes Studio preview composition URLs when resolving timeline elements back to DOM layers.
  • Removes the default per-clip thumbnail toggle from the Studio timeline path and keeps thumbnails visible by default.
  • Simplifies the Design panel to text-first controls plus layout, colors where applicable, radius, and shadow.
  • Treats media tags and background-image layers as image-like controls, hiding color controls for those selections.
  • Lets runtime playback advance duration-only/nested compositions by seeking registered child timelines from their DOM schedule when there is no captured root timeline.
  • Treats failed media sources as failed instead of buffering, so they do not freeze the runtime clock or keep the Studio loading overlay stuck.
  • Makes Studio thumbnail generation failures non-fatal so a Puppeteer thumbnail timeout warns and returns null instead of crashing the dev server.

Root cause

Selection and movement were sharing the same feature gate, so disabling manual dragging also disabled the expected click-to-select behavior. The canvas picker used a single hit target, which favored broad containers in common compositions. The timeline layer resolver also compared absolute Studio preview URLs against relative data-composition-src values, so the inspect button could become active without opening the layer list for master clips.

For the Notion showcase, the main composition registers child GSAP timelines and an empty main timeline. When the runtime did not have a captured root timeline, player.play() returned before starting the transport clock. Separately, the broken narration.wav reported MEDIA_ELEMENT_ERROR: Format error; Studio and the runtime were still treating low-readyState failed media as pending/buffering.

The “Your 24/7 AI team” click case is not a DOM-depth ranking failure: the visible words in that frame are pixels inside assets/og-image.png on .hero-bg. There is no selectable DOM text node at that point, so the best default path is to keep the raster target selected and open Ask Agent with the precise preview click context.

Verification

Local

  • bun run --filter @hyperframes/core test -- src/runtime/init.test.ts
  • bun run --filter @hyperframes/studio test -- src/player/components/Player.test.ts src/components/editor/manualEditingAvailability.test.ts src/components/editor/domEditing.test.ts src/components/editor/PropertyPanel.test.ts
  • bun run --filter @hyperframes/studio test -- src/components/editor/domEditing.test.ts src/components/editor/PropertyPanel.test.ts src/components/editor/manualEditingAvailability.test.ts
  • bun run --filter @hyperframes/studio typecheck
  • bun run --filter @hyperframes/studio build
  • bun run --filter @hyperframes/core build:hyperframes-runtime
  • bunx oxlint packages/studio/src/App.tsx packages/studio/src/components/editor/domEditing.ts packages/studio/src/components/editor/domEditing.test.ts
  • bunx oxfmt --check packages/studio/src/App.tsx packages/studio/src/components/editor/domEditing.ts packages/studio/src/components/editor/domEditing.test.ts
  • git diff --check
  • Pre-commit hook: lint, format, repo typecheck, commitlint

Browser

Verified with agent-browser against apple-presentation-download on local Studio:

  • Direct text click opens Design and shows Text first.
  • Deep text leaf click selects the leaf instead of the broad heading/container.
  • Video selection shows only layout, radius, and shadow controls.
  • Timeline clips show thumbnails by default and no per-clip thumbnail toggle buttons.
  • Manual drag gesture does not move selected text while the manual-drag flag is off.
  • Timeline layer inspect opens the clip layer list for Slide Studio; explicit layer row selection selects Studio Screenshot.

Verified with agent-browser against notion-showcase-download on local Studio:

  • After the failed narration.wav is detected, the loading overlay clears and Play is enabled instead of staying disabled for the 10s asset timeout.
  • Clicking Play in the main <hyperframes-player> shadow iframe advances playback; observed runtime state playing=true, time advancing past 4s, and UI time 0:04/0:25.
  • Direct text and image selection still work on the Notion showcase.
  • Clicking the rasterized “Your 24/7 AI team” region selects Hero Bg and opens the Ask Agent modal at the click location.

Artifacts are committed under qa-artifacts/studio-inspector-selection-ux/, including the original selection-flow.webm and screenshots for text, video, deep leaf selection, manual drag disabled, timeline thumbnails, and layer-list escape hatch. Notion-specific screenshots and recordings are under qa-artifacts/studio-inspector-selection-ux/notion-showcase/, including raster-text-ask-agent-fallback.png and raster-text-ask-agent-fallback.webm.

Notes

  • Motion panel default remains off.
  • packages/studio/vite.studioMotion.ts now imports the core helper sources directly, matching the Studio Vite config pattern and avoiding unresolved package subpath imports during Studio test/dev startup.

miguel-heygen and others added 8 commits May 8, 2026 21:40
Fixes and improvements based on power-user testing feedback:

1. Fix stale selection after style edits — handleDomStyleCommit now
   calls refreshDomEditSelectionFromPreview after persisting, matching
   every other commit handler. Without this, the PropertyPanel showed
   frozen computedStyles after color/radius/shadow edits, making it
   look like editing "didn't work." Also adds error handling around
   the persist call.

2. Add rotation field to the Design panel Layout section — reads the
   current rotation angle from the manual edit manifest and commits
   via the existing handleDomRotationCommit handler.

3. Enable motion panel by default — STUDIO_MOTION_PANEL_ENABLED now
   defaults to true so the Motion tab is discoverable without env vars.

4. Color controls only when element has color — fill color section now
   only shows when the element has an explicit non-transparent
   background-color. Text color shows only when the element has a
   color style. Prevents showing color pickers on elements where
   color edits have no visible effect.

5. Exclude canvas from selection — added "canvas" to
   DOM_LAYER_IGNORED_TAGS so canvas elements are not selectable in the
   preview or listed in the layer panel.

6. Multi-selection feedback — shows "N elements selected" with
   guidance instead of the generic empty state when multiple elements
   are selected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The shared Puppeteer browser pool in getSharedBrowser() could throw a
30s TimeoutError during launch. This error propagated as an uncaught
rejection and killed the vite process, even though generateThumbnail
had its own try/catch — the browser launch promise rejected outside
that scope. Now getSharedBrowser itself catches launch failures and
returns null, so thumbnails degrade gracefully instead of crashing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Motion panel stays opt-in via env var per product direction. Only
the Design panel is enabled by default.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The seek/play/applyAfter wrapper functions in manualEdits.ts crashed
with "Cannot set property X which has only a getter" when the player
or timeline objects define seek/play as getter-only properties. This
prevented ALL manual edits (position, rotation, size) from persisting
to disk — the error thrown during applyCurrentStudioManualEditsToPreview
aborted the save queue.

Wrapped all three property assignments in try/catch so wrapping
gracefully degrades when the target object is non-configurable.

Verified: position edit (X=42px) now persists to
.hyperframes/studio-manual-edits.json and survives page refresh.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@miguel-heygen miguel-heygen merged commit a75b6e6 into next May 9, 2026
4 checks passed
@miguel-heygen miguel-heygen deleted the feat/studio-inspector-selection-ux branch May 9, 2026 20:06
Copy link
Copy Markdown
Collaborator

@vanceingalls vanceingalls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-merge advisory review (this PR is already merged into next). Findings flagged for follow-up, not gating.

Verdict — This is the headline PR. Approve in spirit, but 6 follow-ups to surface before the alpha widens. Two of them are should-be-tickets-today.

This PR delivers the design-thread direction directly: deepest-visible-editable element, raster fallback → ask-agent, canvas excluded, design panel trimmed. The resolveVisualDomEditSelectionTarget scoring function plus isLargeRasterDomEditSelection together implement Miguel's plan from the thread almost line-for-line. That's good.

What I'm flagging below: PR hygiene was rough on this one, the design panel +25/-1272 LOC trim was correct but had side effects, and there are correctness edges that aren't tested.

Follow-ups — should-be-tickets-today

  1. Junk file --width was committed to next. Empty zero-byte file at the repo root, appears in commit a75b6e6c00c231dc55903114ed0411d8a2c3d262. Almost certainly a misplaced bun run dev -- --width=... shell redirect from local testing. Verified it's not currently in next tree — likely fixed in a follow-up commit — but worth confirming with gh api repos/heygen-com/hyperframes/contents/--width?ref=next and adding a pre-commit guard for --* files at repo root.

  2. qa-artifacts/ was committed despite PR body claiming it's local-only. The PR description states "Browser proof artifacts and the symlinked Downloads project are not committed." But the merge commit added ~12 PNG/webm files under qa-artifacts/studio-inspector-selection-ux/, and gh api repos/heygen-com/hyperframes/git/trees/next?recursive=1 confirms they're still on next today. Reason: this is several MB of binary content in the repo that no CI consumes. Either add qa-artifacts/ to .gitignore and delete it, or move it to a git lfs track, or store proofs externally. The discrepancy between PR description and reality is the bigger issue — it eroded the audit trail for what actually merged.

Follow-ups — important but not urgent

  1. Design panel was deleted, not refactored. PropertyPanel.tsx: +235/-1272. That's a net deletion of ~1000 LOC including formatColorToken, collectSelectionColors, and the entire Selection Colors panel. The PR body says "Simplifies the Design panel to text-first controls plus layout, colors where applicable, radius, and shadow" — and that's the right intent. But: the audit-all-sites rule says when you remove a property class (selection colors), verify other property classes honor the same simplification contract. Did position/scale/dimension/rotate/color/text-size all get the same "only show if the element actually has this property" treatment as the new conditional color controls? Worth one ticket to audit consistency across all 6 of Vance's base set properties.

  2. getVisualElementScore ranking is heuristic and untested at the edge. The scoring function combines depth × 10,000 + visual-leaf bonus (2,000) + smaller-element bonus (up to 1,000) − pointer-stack-index. The depth term will always dominate for deeply-nested compositions — which is the intended bias toward "deepest editable" — but two cases worry me:

    • A deeply-nested invisible 1px wrapper (hasRenderedBox returns false → excluded from best, but if every descendant is also invisible, we silently return null and fall back to elementFromPoint). Is the fallback a sensible default for this case?
    • Two equally-deep siblings where one is larger (e.g. a heading wrapping multiple words). The smallerElementBonus favors the smaller — meaning a word wrapped in a <span> is preferred over its parent <h1>. Is that the desired bias? Worth a test asserting the expected behavior for sibling-equal-depth.
  3. Raster fallback opens Ask Agent at the click location — but doesn't preserve text-context for accessibility. buildRasterClickSelectionContext includes pixel coordinates and viewport dimensions, but not OCR of the visible text at the click point. The whole reason a user clicked "Your 24/7 AI team" is because they wanted to edit that copy. Reason: the agent gets the coords but has to re-derive what the user meant. Worth a follow-up to optionally pipe screenshot crop + OCR'd text into the prompt context.

  4. No test for "selection during composition switch". Same gap as PR #683. Add one of: select element → switch composition → assert domEditSelection cleared, or assert it persists across composition equivalency (e.g. if the same element ID exists in both).

Important

  • PR ate 7 sequential commits + a merge-from-next. Some of those commits ("revert motion panel default to false", "prevent read-only property crash in manual edit wrappers", "stale selection") indicate the PR was being iteratively debugged against live testing. That's fine for development, but those commits should have been squashed before merge — the history makes it hard to tell which behavior change goes with which root cause.
  • packages/studio/vite.studioMotion.ts import change in this PR is unrelated to the inspector UX. It's load-bearing for Studio test/dev startup, but it should have been in PR #682 or its own bottom-of-stack PR — folding it into the headline UX PR muddies blame.
  • Multi-element selection feedback (commit 1 of follow-ups). The "N elements selected" empty state is good, but I didn't find a test for the multi-select case in the diff. Group editing was explicitly called out in the design thread as "disabled or whitelisted" — what's the actual gating? If multi-select shows "guidance," is it informational only, or can users actually batch-edit? Worth confirming explicit behavior.

Nits

  • clampNumber is a 3-line utility duplicated in this PR's local scope. The studio likely already has one in utils/.
  • getAgentModalPositionStyle uses hardcoded modalWidth = 480, estimatedModalHeight = 270. Worth deriving from the modal's actual rendered size via useLayoutEffect for robustness across viewport sizes.

Praise

  • The ranking function design (depth-first, leaf-bonus, area-tiebreaker) is the right shape for "deepest visible editable" — and it's exposed as resolveVisualDomEditSelectionTarget for testing, not buried in the click handler. Good separation.
  • Canvas now excluded from DOM_LAYER_IGNORED_TAGS — directly addresses the design-thread rule.
  • The "data-composition-src" normalization fix (normalizeTimelineCompositionSource) for master-clip layer-list resolution is a real bug fix that would've been impossible to spot without browser verification.
  • The raster fallback → Ask Agent at click location is genuinely clever. It turns a "selection failed" UX dead-end into a productive ask-the-AI flow.

— Vai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants