feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net

[vanceingalls]

What

Ships drawElement fast capture default-on (no --experimental-fast-capture needed), wrapped in a runtime self-verification safety net that catches the damage classes no compile/init gate can predict and auto-falls-back to the screenshot path. Includes the release tooling (canary suite, corpus routing classifier) and a hardening round from an adversarial 50-agent code review (15 confirmed findings, all fixed in the second commit).

Why

The combined fast-capture stack (#1916–#1928) is ~1.9–2.0× median vs screenshot capture, but drawElement has intermittent, run-dependent damage classes — stale compositor paints, dropped background images, transient blank frames — that static gates cannot see. Validation caught two real ones in the wild that previously shipped silently: comp d95f20b6 intermittently drops its photo background (~16dB, fires on some runs and not others), and 398c0655 renders with a vertical offset. Default-on is only safe if every drawElement render proves itself and falls back when it can't. The failure mode after this PR is a slower render, never wrong pixels.

How

Safety net — three layers:

1. Pre-injection ground truth (engine). At init — after all gates pass, before injectDrawElementCanvas (afterwards Page.captureScreenshot returns the canvas bitmap, not live DOM) — seek K=4 (HF_DE_VERIFY) evenly-spread frames and screenshot each. Seeks go frame 0 first, then ascending (lazy GSAP .from() tweens record start values on first seek; mid-timeline scrubs corrupt them for the whole render — and since DE frames and truth would share the corruption, PSNR would pass on damaged output). Each sample runs the video-injection hook and double-captures (the first screenshot forces the frame that runs rAF callbacks — count-up text counters land one tick after seek() returns). Indices derive from the producer-resolved duration (CaptureOptions.compositionDurationSeconds), not raw __hf.duration, so they always land inside the drained range.
2. Drain-time checks (producer). Every drained frame passes guardFrame: a rolling-median blank guard (byte-identical recapture ⇒ legitimately dark frame, accepted; recapture uses a dedicated verification-grade path with no static-dedup shortcut and no stale-canvas fallback — both can return the wrong frame's pixels at drain time) plus PSNR self-verification against the ground truth (HF_DE_VERIFY_MIN_DB, default 32dB; natural DE-vs-screenshot agreement measures ≥45dB, damage ≤25dB). A breach dumps the frame pair to tmpdir and throws a typed DrawElementVerificationError.
3. Whole-render auto-fallback (orchestrator). One-shot retry with forceScreenshot=true; telemetry flag deSelfVerifyFallback; the recovered first attempt no longer brands the render failed (observability.clearFailure).

Default flip, confined to where the net runs:

• useDrawElement + enableDrawElementWorkerEncode default true; HF_DE_BATCH default 4.
• resolveConfig clamps the default to macOS with a non-software GPU mode ("auto" passes — the stock CLI resolves auto; if it probes to software the SwiftShader init gate still routes away) and requires worker-encode (the only drain with the net).
• The orchestrator disengages default DE for renders taking the disk path (>240s streaming cap, png-sequence) or parallel capture — no drain verifies those — and closes a DE-initialized probe session rather than letting an unverified path reuse it.
• Shader-transition comps prefer page-side compositing over default DE; comps compile-gated out of DE get page-side compositing restored.
• Explicit PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true bypasses every clamp (old attempt-and-gate semantics for debugging).
• Kill switches, all env, no deploy: PRODUCER_EXPERIMENTAL_FAST_CAPTURE=false, HF_DE_WORKER_ENCODE=false, HF_DE_BATCH=0, HF_DE_VERIFY=0.

Tooling: de-canary-suite.sh (7 comps with expected verdicts clean/fallback/any + cross-path PSNR gate), de-gatecheck.sh (init-only corpus routing classifier — detects both engine init-gate and producer compile-gate log shapes), we-render.mjs (the render harness both scripts drive).

Test plan

• Canary suite 7/7 green — clean comps 43–58dB cross-path vs a screenshot render; damage/defect comps correctly diverted to fallback; the two known-intermittent comps covered by any.
• Corpus routing (611-comp sample of 1762): 54% route drawelement, 37.5% gated (blur > mix-blend > backdrop > 3D), 8.3% comp defects.
• Risk-band renders: 12/12 DE-routed comps (incl. a 74-video comp) clean on bare defaults — 48/48 verify samples passed, zero fallbacks.
• Clamp behavior validated end-to-end: PRODUCER_BROWSER_GPU_MODE=auto engages the full stack; forced disk path and HF_DE_WORKER_ENCODE=false disengage default DE with a clear log; forced-failure (HF_DE_VERIFY_MIN_DB=99) trips at the first sample and completes via screenshot; malformed env values fall back to defaults instead of silently disabling verification.
• Engine suite 899 passed (1 failure reproduces identically on pristine main — pre-existing color-grading test). tsc/oxlint/oxfmt clean.
• Adversarial max-effort code review (50 agents): 15 confirmed findings, all fixed in fix(engine,producer): harden the drawElement self-verification net.

🤖 Generated with Claude Code

[vanceingalls]

• feat(engine,producer,cli): drawElement release telemetry on render_complete #2002
• feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net #1998 👈 (View in Graphite)
• perf(engine,producer): batch N drawElement frames per CDP round-trip (HF_DE_BATCH) #1928
• feat(lint,player): fast-capture lint rule + player media sync #1921
• feat(producer): fast-capture render stages + remote bg-image localizer #1920
• feat(engine): frame-capture core — fast-capture routing, worker-encode, dedup extension #1919
• feat(engine): 3D projection + compositor-effect risk gate #1918
• feat(engine): drawElementImage capture service #1917
• feat(engine,cli): drawElement fast-capture config + CLI flag #1916
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[miga-heygen]

PR Review — feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net

TITLE: feat(engine,producer): drawElement fast-capture default-on with runtime self-verification safety net
VERDICT: Approve (comment-only — no blocking issues found)
DIFF_STATS: +725 / -87, 17 files changed

---

SUMMARY

This PR flips drawElement (DE) fast capture from opt-in to default-on, guarded by a three-layer runtime self-verification safety net that guarantees the failure mode is a slower render, never wrong pixels. The safety layers are:

1. Pre-injection ground truth (engine) — K=4 screenshot samples captured before the DE canvas is injected, using a careful seek-order (frame 0 first, then ascending) to avoid corrupting GSAP .from() tween caches.
2. Drain-time guard (producer) — a rolling-median blank guard + PSNR comparison against ground truth at sample indices, with verification-grade recapture that bypasses static-dedup and screenshot-fallback shortcuts.
3. Whole-render auto-fallback (orchestrator) — catches DrawElementVerificationError, re-renders with forceScreenshot=true, clears the observability failure flag.

The default is clamped to macOS + non-software GPU + worker-encode enabled. Explicit env opt-in bypasses all clamps. Kill switches are env-only, no deploy needed. Shader-transition comps are routed back to page-side compositing. Disk-path and parallel-capture renders disengage default DE since they lack the verification drain.

---

PONYTAIL

The architecture here is genuinely well-reasoned. The insight that verification must happen pre-injection (because post-injection screenshots show the canvas bitmap, not the live DOM) is a subtle correctness invariant that's easy to get wrong. The seek-order discipline (frame 0 first to initialize lazy GSAP tweens before sampling) shows deep understanding of the animation runtime. The discriminant-based error detection (deVerificationFailure property instead of instanceof) surviving duplicated module instances across package boundaries is a real-world concern that most PRs ignore. The double-capture trick for rAF-driven counters (first screenshot forces the frame, second captures the result) is a pragmatic workaround for headless Chrome's rAF scheduling. The blank-guard's acceptedSmall optimization avoids re-capturing every frame in a legitimately dark stretch. The explicit rejection of concurrent drain+capture (NOTE comment about 87dB noise floor perturbation) shows the team is measuring empirically and choosing reliability over marginal perf.

---

FINDINGS

(1) Observation — psnrDb spawns ffmpeg per verification sample (non-blocking)
captureStreamingStage.ts — The psnrDb function spawns a child process, writes two files, and parses stderr for each of the K=4 verification frames. At ~50-75ms per call this adds ~200-300ms per render, which is well within the init budget. But if HF_DE_VERIFY is ever raised toward 8, and someone runs a very fast short composition, the overhead could approach a meaningful fraction of total render time. The current default of 4 is fine, and the clamp to 0..8 caps the exposure. Just flagging for awareness — no action needed.

(2) Observation — clearFailure timing and observeRenderStage interaction (verified correct)
renderOrchestrator.ts — The retry flow is:

1. First invokeStreaming() throws through observeRenderStage → stageError sets failedPhase = "capture_streaming".
2. Orchestrator catches, mutates captureForceScreenshot = true, probeSession = null.
3. Second invokeStreaming() succeeds → stageEnd runs but does NOT clear failedPhase.
4. clearFailure("capture_streaming") explicitly clears it.
   This is correct. stageEnd intentionally doesn't clear failures (a later phase could succeed while an earlier one still failed), so the explicit clearFailure is necessary. The closure properly captures captureForceScreenshot and probeSession by reference (let bindings), so the retry reads the updated values. Verified.

(3) Observation — explicitDrawElementOptIn only matches "true" strictly (by design, verified)
config.ts — env("PRODUCER_EXPERIMENTAL_FAST_CAPTURE") === "true" only matches the literal string "true", not "1", "yes", or other truthy representations. This is consistent with the existing env-var parsing convention in resolveConfig (all other boolean envs use the same === "true" pattern). The kill switch === "false" would need the literal "false" to disable via the env var, but the default flip itself means omitting the env var entirely now enables DE (on supported hosts), so the kill switch works by setting it to "false". Setting it to "0" or anything else would leave the default clamping logic in control. This is a bit subtle but consistent.

(4) Low — isNoCachedPaintRecordError changed from export to private (minor scope reduction)
frameCapture.ts — The function was exported before and is now module-private. The PR body doesn't mention this. If any downstream consumer was using it, this would be a breaking change. Since it's not in the engine's index.ts re-exports and is an internal diagnostic, this is fine — just noting the visibility change.

(5) Observation — Canary suite exit behavior in subshell
de-canary-suite.sh — The while read loop runs in a subshell (piped from echo "$CANARIES"). The exit 1 inside the loop exits the subshell, not the script. The || fail=1 after done catches this. However, the fail variable is initialized to 0 before the pipe, and the subshell's exit 1 doesn't set fail=1 — the || fail=1 does. The final check [ "$fail" = 0 ] works correctly because the || fires when the subshell exits non-zero. This is a known bash idiom; it works but is fragile (adding set -e to the script would change the semantics). Since this is a manual test harness, not CI, it's fine.

(6) Observation — Video comp without frame injector skips verification rather than failing
frameCapture.ts — When session.onBeforeCapture is null and the page has <video> elements, verification is skipped entirely ("video comp without frame injector"). The comment explains this avoids false positives (videos screenshot as black without the injector), but it also means a probe session initialized before the render wires up the injector gets no verification at all. The orchestrator's disengage-for-disk-path logic closes the probe session in this case, so it shouldn't reach production — but if a future code path reuses a probe session without wiring the injector, the verification gap would be silent. The log message makes this auditable.

(7) Observation — Binary test fixtures changed
Three binary files changed (css-var-fonts/output/output.mp4, video-hfid-no-id/output/output.mp4, video-hfid-no-id/src/clip.mp4). These are presumably snapshot-pinned test expectations that shifted because DE is now default-on (the render pipeline changed). As long as the test suite passes with these new fixtures, this is expected.

---

Review by Miga

[miguel-heygen]

Current-head review at 8009500002af7b0d0e0bfeae621990785ae8c3d0.

Audited: default-on config flow in packages/engine/src/config.ts, compile-stage routing in packages/producer/src/services/render/stages/compileStage.ts, self-verification threshold parsing in captureStreamingStage.ts, canary scripts, orchestrator fallback, and current check state.

Blocker

• packages/producer/src/services/render/stages/compileStage.ts:210-215 restores page-side compositing any time cfg.useDrawElement is false and HF_PAGE_SIDE_COMPOSITING is not false. That loses the distinction between "resolveConfig disabled page-side compositing because default drawElement was going to run" and "the caller explicitly passed enablePageSideCompositing: false through the programmatic API." In the latter case, a compile-time drawElement gate such as 3D/mix-blend/shader transitions silently flips the caller's explicit override back to true. Please preserve the original explicit config intent, for example by tracking whether this render disabled page-side compositing only because default drawElement was enabled, and only restore in that case.

Important follow-ups before default-on widens

• packages/producer/src/services/render/stages/captureStreamingStage.ts:202-203 only NaN-guards HF_DE_VERIFY_MIN_DB; 0 disables meaningful PSNR fallback and 999 forces screenshot fallback on every verified render. Clamp to a defensible range and warn on out-of-range values.
• packages/producer/de-canary-suite.sh:12 and packages/producer/de-gatecheck.sh:6 use set -u without set -euo pipefail, so pipeline/grep failures can keep the release canaries moving after a broken command. These scripts are part of the safety net for default-on and should fail closed.

Checks: regression shards and Graphite are still pending on this head; no active required failure is visible yet.

Verdict: REQUEST CHANGES
Reasoning: The runtime fallback architecture is sound, but the current compile-stage restore path can override an explicit API config and change capture behavior for direct callers. That is a real contract regression in the default-on PR.

• Magi

[vanceingalls]

All three items addressed in 600612541:

• Blocker (PSC intent): resolveConfig now records pageSideCompositingAutoDisabled only when it turned page-side compositing off itself (because drawElement was on); the compile-stage restore fires only on that flag. An explicit enablePageSideCompositing: false via the programmatic API or HF_PAGE_SIDE_COMPOSITING=false never sets the flag and stays off. Two config tests pin both directions.
• Threshold clamp: HF_DE_VERIFY_MIN_DB clamped to [10, 60] with a warning on out-of-range values (below ~10dB the check passes severe damage; above ~60dB natural encoder deltas force fallback on every render).
• Fail-closed canaries: both scripts run under set -euo pipefail with explicit || true only on expected-nonzero commands, plus a hard FAIL when the PSNR compare produces no value. Full canary suite re-run green (7/7) under the new flags.

[miguel-heygen]

Delta re-review at 9de558449c18c733daaf761ba5c0e894b942d376.

Audited: packages/engine/src/config.ts, packages/engine/src/config.test.ts, packages/producer/src/services/render/stages/compileStage.ts, packages/producer/src/services/render/stages/captureStreamingStage.ts, and the two DE helper scripts.

The prior blocker is fixed. resolveConfig now records pageSideCompositingAutoDisabled only when it itself disabled page-side compositing because drawElement was enabled (config.ts:564-570), and compileStage.ts:210-218 restores page-side compositing only when that flag is present. That preserves explicit programmatic/API intent for enablePageSideCompositing: false. The config tests also pin both the auto-disabled and explicit-opt-out cases.

The prior important notes are also cleaned up: HF_DE_VERIFY_MIN_DB is clamped to [10, 60] with a warning (captureStreamingStage.ts:202-214), and de-canary-suite.sh / de-gatecheck.sh now run under set -euo pipefail with explicit expected-error handling.

CI note: the current active PR check view still has Graphite, preview-regression, and regression shards pending. I am approving the code delta, not declaring the full stack merge-ready before those finish.

Verdict: APPROVE
Reasoning: The config contract regression is closed without clobbering explicit caller intent, and the release-safety hardening notes were addressed as well.

• Magi

[terencecho]

LGTM per my per-PR review + re-review on the Slack thread. Non-blocking nits I raised (where present) are fair as post-merge follow-ups.

— Review by tai (pr-review)

The base branch was changed.