Skip to content

fix(runtime): tighten audio drift correction to ~80ms for narration#670

Closed
miguel-heygen wants to merge 5 commits intomainfrom
fix/audio-drift-strict-sync
Closed

fix(runtime): tighten audio drift correction to ~80ms for narration#670
miguel-heygen wants to merge 5 commits intomainfrom
fix/audio-drift-strict-sync

Conversation

@miguel-heygen
Copy link
Copy Markdown
Collaborator

Summary

  • Adds a strict sync tier (80ms threshold, consecutive-sample gated) to the runtime media path that catches accumulated drift from pause/play toggling
  • Forces immediate sync on play/pause/seek/rate transitions via a one-shot forceSync flag
  • Guards against false correction during initial buffering by requiring offset stabilization before strict sync fires

Closes #668

What changed

The runtime-owned media sync (syncRuntimeMedia) previously only corrected drift above 0.5s (and only on specific events). Stable drift between 80ms–500ms — common after repeated pause/play — persisted indefinitely. For narration-driven videos this caused audible A/V desync.

The fix adds three mechanisms:

  1. Strict sync — 80ms threshold with 2-consecutive-sample gating (mirrors the parent-proxy path's MIRROR_DRIFT_THRESHOLD_SECONDS strategy)
  2. forceSync on transitions — play/pause/seek/rate changes set mediaForceSyncNextTick which bypasses all drift thresholds on the next tick
  3. Buffering guard — strict sync only fires when offset has stabilized (< 4ms/tick change), preventing false correction during initial buffer catch-up

Test plan

  • All 694 existing tests pass (including the buffering-guard test that expects audio to stay at 0 during initial buffer)
  • New test: forceSync corrects any drift above 20ms immediately
  • Updated test: verifies strict sync fires after 2 consecutive stable over-threshold ticks
  • Pre-commit hooks pass (lint, format, typecheck, commitlint)
  • Manual: load a composition with timed narration, toggle play/pause 40 times, verify narration stays in sync

🤖 Generated with Claude Code

miguel-heygen and others added 5 commits May 7, 2026 16:40
showcase project for the tech article — 17 compositions individually
renderable via the HyperFrames producer.

3 flagship HTML-in-Canvas effects (liquid glass, device mockup, shatter)
plus 14 GLSL shader transitions organized by category.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
four new compositions for the Three.js shaders article:

- vfx-cinematic-camera: dolly zoom, crane shot, orbit, rack focus
  applied to HTML content in a Three.js scene (16s)
- vfx-god-rays: screen-space radial blur god rays through occluding
  pillars with dust particles and camera orbit (8s)
- vfx-particle-field: 8000 GPU particles morphing between text,
  sphere, torus, and noise states with additive blending (10s)
- vfx-ray-march: pure GLSL raymarching through morphing SDF tunnel
  with fractal folds and octahedron pillars (8s)

all render on SwiftShader (no GPU required) at 45-167ms/frame.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
replaces the flat HTML plane with the KhronosGroup DamagedHelmet GLTF
model (CC-BY 4.0, 3.6MB). four camera moves orbit, dolly, crane, and
rack-focus around the PBR helmet with dramatic three-point lighting.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…se cases

Adds a strict sync tier to the runtime-owned media path that catches
accumulated drift from repeated pause/play toggling. Previously, drift
below 0.5s persisted indefinitely (only the 3s catastrophic valve would
correct it). For speech-driven videos where narration, captions, and
visual timing need to stay aligned, this caused audible desync.

The fix adds three mechanisms:

1. **Strict sync (80ms, consecutive-sample gated)**: when the timeline-
   to-media offset has stabilized (not growing from initial buffering)
   and drift exceeds 80ms for 2 consecutive ticks, force a currentTime
   correction. This mirrors the parent-proxy path's 50ms threshold with
   consecutive-sample gating (MIRROR_DRIFT_THRESHOLD_SECONDS).

2. **forceSync on transitions**: play/pause toggles, seek, and playback-
   rate changes set a one-shot flag that forces an immediate sync on the
   next tick, bypassing all thresholds. This prevents drift from
   accumulating across pause/play cycles.

3. **Buffering guard**: strict sync only fires when the offset has
   stabilized (delta < 4ms/tick). During initial buffering, the offset
   grows ~16ms/tick as the timeline advances while media stays at 0 —
   this gradual growth is not corrected, preserving the existing behavior
   of not skipping opening audio.

Closes #668

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@miguel-heygen
Copy link
Copy Markdown
Collaborator Author

Superseded by #671 which includes the strict sync changes plus the full single-clock transport architecture. The strict sync threshold was tightened from 80ms to 40ms in #671, and the single-clock transport eliminates the root cause (two-clock architecture).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runtime-owned audio drifts from GSAP timeline after repeated pause/play

1 participant