feat(captions): energy-based technique selection and mandatory quality checks by vanceingalls · Pull Request #176 · heygen-com/hyperframes

vanceingalls · 2026-04-01T04:20:50Z

Summary

Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits
Replace tl.call() per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time, no per-frame callbacks needed
Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model)
Add caption word structure lint rule (.caption-group + <span>) for studio editor compatibility
Add multilingual model guidance and decision tree for model selection

Test plan

Skill files render correctly as markdown
Cross-references between SKILL.md, dynamic-techniques.md, and transcript-guide.md resolve correctly
dynamic-techniques.md audio-reactive section uses tl.to()/tl.set() only, no tl.call() loops

🤖 Generated with Claude Code

vanceingalls · 2026-04-01T04:21:05Z

feat(captions): energy-based technique selection and mandatory quality checks #176 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

vanceingalls · 2026-04-02T06:54:54Z

Folded into #175 (vance/whisper-improvements)

…multilingual defaults (#175) ## Summary **Whisper improvements:** - Auto-detect language and switch from `.en` to multilingual model when needed - Detect speech onset in WAV to strip hallucinated words before speech begins - Merge whisper-cpp token fragments: contractions (`didn` + `'t` → `didn't`), split capitals (`C` + `aught` → `Caught`), dropped-g (`shin` + `in'` → `shinin'`) - Interpolate zero-duration word clusters for reliable karaoke timing **Captions skill updates (folded from #176):** - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility **Multilingual defaults (folded from #186):** - Default whisper model changed from `small.en` to `small` to prevent silent translation of non-English audio - Added non-negotiable language rule to captions skill ## Test plan - [ ] `pnpm test` passes (contraction merging, fragment merging, zero-duration interpolation, speech onset) - [ ] Transcribe non-English audio — verify it transcribes in original language, not translates - [ ] Skill files render correctly, cross-references resolve - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Remove custom skill installation from the CLI in favor of vercel-labs/skills. Fixes staleness, deduplication, listing budget, and completeness issues identified by agentic harness pattern review. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

6 tasks: delete install-skills, strip skills from init, front-load SKILL.md descriptions, update CLAUDE.md, clean up my-video copies. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace single "color" with activeColor (spoken) and dimColor (inactive). Runtime hook modifies existing GSAP color tweens by detecting dim vs active states via opacity heuristic. Property panel shows both color pickers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove typography, color, opacity, stroke, and background controls. Keep only Position (X/Y) and Transform (Scale/Rotation) — the controls that work reliably with the wrapper-based override system. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

1px per press, 10px with Shift held. Syncs to store and auto-saves. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Caption overlay and hover states always active on captions compositions - Clicking a word auto-opens the property panel - Remove Edit Captions button — edit mode activates automatically - Reset edit mode when navigating away from captions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Caption edit mode now activates automatically when .caption-group elements are found in the iframe DOM. Works from index.html (master) without requiring drill-down into the captions sub-composition. Finds the captions source file via data-composition-src attribute. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ding Replace polling with a message listener for hf-preview state/timeline events. These fire after loadExternalCompositions completes, guaranteeing .caption-group elements exist in the DOM. Eliminates the race condition on initial page load. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Remove store reset from useEffect cleanup — was wiping state on dep changes and breaking auto-save subscription - Move panel auto-open into useEffect instead of during render - Add fallback 500ms poll alongside postMessage listener for initial load Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Runtime hook's wrapper was missing data-caption-wrapper attribute, so the overlay created a second wrapper on top. Transforms were applied to different wrappers causing values to compound on reload. Also: - Suppress auto-save during loadOverrides to prevent immediate re-write - Only sync non-default transform values to avoid spurious overrides Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Reduce poll interval from 500ms to 200ms for faster detection - Fall back to DOM querySelector for iframe when ref isn't set yet - Runtime wrapper sets data-caption-wrapper for consistent detection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The runtime strips data-composition-src after loading, so the DOM query couldn't find the captions file path. Now uses the compIdToSrc map (built from raw index.html) as the primary lookup. Also adds compIdToSrc to the effect dependency array so it re-runs when the map is populated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Panel rotation/scale/position changes now go through the wrapper span using gsap.set, reading current values first to preserve other properties. Previously set el.style.transform directly on the word span, overwriting all existing transforms. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Delete dead useCaptionDrag.ts (superseded by overlay inline drag) - Self-cancel 200ms poll after caption activation - Remove stale effectiveStyle deps from applyToIframeDom useCallback Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Replace doc!/win!/captionSrcPath! with guards in App.tsx async callback - Move useCallback hooks above early return in CaptionAnimationPanel (React hooks must be called unconditionally in the same order) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…erty panel - Replace filter(Boolean) + ! assertions with typed filter in generator.ts - Hoist repeated seg.id.replace() to const segVar - Remove unused ToggleButton, ColorInput, container/typography derived values, applyContainerToIframeDom, handleContainerChange, buildTextDecoration - Remove CaptionContainerStyle import Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add change detection to 66ms overlay poll (skip setWordBoxes when unchanged) - Add activating guard to prevent duplicate caption source fetches - Remove applyOpacityToColor wrapper (inline hexToRgba) - Rename rawWords back to words in normalize.ts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The single-quote replacement regex ran after key-quoting, causing it to match apostrophes inside already-double-quoted strings like "didn't". Moving it before key-quoting fixes the issue since it only matches actual single-quoted string values at that point. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

vanceingalls · 2026-04-02T07:17:49Z

Work folded into #175 (merged)

…multilingual defaults (heygen-com#175) ## Summary **Whisper improvements:** - Auto-detect language and switch from `.en` to multilingual model when needed - Detect speech onset in WAV to strip hallucinated words before speech begins - Merge whisper-cpp token fragments: contractions (`didn` + `'t` → `didn't`), split capitals (`C` + `aught` → `Caught`), dropped-g (`shin` + `in'` → `shinin'`) - Interpolate zero-duration word clusters for reliable karaoke timing **Captions skill updates (folded from heygen-com#176):** - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility **Multilingual defaults (folded from heygen-com#186):** - Default whisper model changed from `small.en` to `small` to prevent silent translation of non-English audio - Added non-negotiable language rule to captions skill ## Test plan - [ ] `pnpm test` passes (contraction merging, fragment merging, zero-duration interpolation, speech onset) - [ ] Transcribe non-English audio — verify it transcribes in original language, not translates - [ ] Skill files render correctly, cross-references resolve - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code)

…y checks (heygen-com#176) ## Summary - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time, no per-frame callbacks needed - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility - Add multilingual model guidance and decision tree for model selection ## Test plan - [ ] Skill files render correctly as markdown - [ ] Cross-references between SKILL.md, dynamic-techniques.md, and transcript-guide.md resolve correctly - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code)

This was referenced Apr 1, 2026

feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults #175

Merged

feat(skills): add dynamic caption techniques and split captions skill into references #173

Merged

vanceingalls mentioned this pull request Apr 1, 2026

feat(cli): skill install targets + remove custom install in favor of vercel-labs/skills #177

Merged

4 tasks

vanceingalls marked this pull request as ready for review April 1, 2026 04:22

This was referenced Apr 1, 2026

fix(studio): serve runtime locally instead of from CDN #178

Closed

feat(cli): add hyperframes validate command #179

Merged

vanceingalls force-pushed the vance/captions-skill-updates branch from 1cbe901 to fe9a3a2 Compare April 1, 2026 05:38

vanceingalls force-pushed the vance/whisper-improvements branch from 5360ae6 to c47472a Compare April 1, 2026 05:38

vanceingalls mentioned this pull request Apr 1, 2026

feat(studio): caption designer #180

Closed

8 tasks

miguel-heygen approved these changes Apr 1, 2026

View reviewed changes

vanceingalls force-pushed the vance/captions-skill-updates branch from 44abb54 to 847ba6e Compare April 1, 2026 22:05

vanceingalls force-pushed the vance/whisper-improvements branch from c47472a to 28e707c Compare April 1, 2026 22:05

vanceingalls mentioned this pull request Apr 1, 2026

fix(skills): default to non-.en whisper models to prevent silent translation #186

Closed

2 tasks

vanceingalls force-pushed the vance/captions-skill-updates branch from 847ba6e to efb09e3 Compare April 2, 2026 05:30

vanceingalls force-pushed the vance/whisper-improvements branch from 28e707c to 4b0ce61 Compare April 2, 2026 05:30

vanceingalls mentioned this pull request Apr 2, 2026

refactor(cli): remove custom skill install, delegate to vercel-labs/skills #189

Closed

4 tasks

vanceingalls force-pushed the vance/whisper-improvements branch from 4b0ce61 to 436cd55 Compare April 2, 2026 05:57

vanceingalls force-pushed the vance/captions-skill-updates branch from efb09e3 to 777a00a Compare April 2, 2026 05:57

vanceingalls force-pushed the vance/whisper-improvements branch 2 times, most recently from 7085bdc to 4e51632 Compare April 2, 2026 06:25

vanceingalls force-pushed the vance/captions-skill-updates branch from 71bc159 to 0686408 Compare April 2, 2026 06:25

vanceingalls changed the base branch from vance/whisper-improvements to graphite-base/176 April 2, 2026 06:38

vanceingalls closed this Apr 2, 2026

vanceingalls reopened this Apr 2, 2026

vanceingalls and others added 3 commits April 2, 2026 00:12

docs: add skill system cleanup implementation plan

1a3fbe9

6 tasks: delete install-skills, strip skills from init, front-load SKILL.md descriptions, update CLAUDE.md, clean up my-video copies. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

refactor(cli): remove skills subcommand and install-skills module

813ae07

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

vanceingalls and others added 17 commits April 2, 2026 00:13

feat: arrow key nudge for selected caption words

be953aa

1px per press, 10px with Shift held. Syncs to store and auto-saves. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: close edit panel when caption word is deselected

50afa18

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

vanceingalls closed this Apr 2, 2026

vanceingalls reopened this Apr 2, 2026

vanceingalls closed this Apr 2, 2026

vanceingalls reopened this Apr 2, 2026

vanceingalls force-pushed the vance/captions-skill-updates branch from 0686408 to 2197f18 Compare April 2, 2026 07:23

vanceingalls force-pushed the graphite-base/176 branch from 4e51632 to fc99b40 Compare April 2, 2026 07:23

fix: rename rawWords to words in loadTranscript return

07614c6

vanceingalls force-pushed the graphite-base/176 branch from fc99b40 to 37404f2 Compare April 2, 2026 07:26

vanceingalls changed the base branch from graphite-base/176 to main April 2, 2026 07:26

vanceingalls merged commit d36c178 into main Apr 2, 2026
29 of 40 checks passed

Navigation Menu

Uh oh!

feat(captions): energy-based technique selection and mandatory quality checks - #176

feat(captions): energy-based technique selection and mandatory quality checks#176
vanceingalls merged 55 commits into
mainfrom
vance/captions-skill-updates

vanceingalls commented Apr 1, 2026 •

edited

Loading

Uh oh!

vanceingalls commented Apr 1, 2026 •

edited

Loading

Uh oh!

vanceingalls commented Apr 2, 2026

Uh oh!

vanceingalls commented Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

vanceingalls commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

vanceingalls commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanceingalls commented Apr 2, 2026

Uh oh!

vanceingalls commented Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vanceingalls commented Apr 1, 2026 •

edited

Loading

vanceingalls commented Apr 1, 2026 •

edited

Loading