Skip to content

feat(media-use): classify and tag heygen provider telemetry, harden test isolation#2130

Open
miguel-heygen wants to merge 6 commits into
mainfrom
feat/heygen-conversion-telemetry
Open

feat(media-use): classify and tag heygen provider telemetry, harden test isolation#2130
miguel-heygen wants to merge 6 commits into
mainfrom
feat/heygen-conversion-telemetry

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

media-use's heygen-cli integration could tell you a resolve succeeded, but not why it failed along the way, or whether a successful call went through the free (OAuth) or paid (API-key) path. This closes both gaps, plus a real test-isolation bug where local test runs could reach the production telemetry endpoint.

Companion change to heygen-com/heygen-cli#212 (identity unification + auth telemetry) — the two land independently since neither repo's release blocks the other.

What changed

  • Failure classification: heygen-cli errors (not found, not authenticated, outdated, rate limited) now map to a small stable reason instead of a raw, unbounded error string — including a new rate-limit/quota case that wasn't recognized before.
  • Free vs. paid tagging: successful heygen-backed resolves now record whether the call used an OAuth session or an API key, without changing the existing auth-header logic other call sites depend on.
  • Test isolation fix: local test runs previously relied solely on an env-var default to avoid hitting the real telemetry endpoint. That's now backed by a real interception seam (verified with a local test server), so the safety doesn't depend on every future test remembering to set the env var.

Design decisions

  • The new failure reason is a stable reason value, not the display message or raw stderr text — stays a small enum a dashboard can group by, not free text.
  • The free/paid signal is a new property, not encoded into the existing provider string, so it doesn't touch any dashboard tile that already keys off provider name.

Testing

  • Full skills suite: 203 passed, 1 failed (pre-existing, unrelated to this change).
  • oxlint / oxfmt --check: clean across all changed files.

Post-Deploy Monitoring & Validation

  • No runtime/production monitoring needed beyond normal telemetry review — this is additive, best-effort instrumentation with no behavior change to resolve outcomes.
  • Watch for: no increase in resolve failures or latency (telemetry calls are fire-and-forget and never block a resolve).

Adds a stable reason-code classifier (not_found/not_authenticated/
outdated/rate_limited/other) alongside the existing display-message
classifier, and reports it via a new media_use_provider_error event so
quota/rate-limit failures are distinguishable from other provider
errors even when a fallback provider later succeeds.
Adds a sparse auth_method property (oauth/api_key) to media_use_resolve
telemetry for heygen-family providers, derived via a new heygenAuthMethod()
helper alongside the existing heygenAuthHeaders(). Absent for every
non-heygen provider.
…ception seam

resolve.test.mjs already set DO_NOT_TRACK=1 for every spawned resolve.mjs
invocation, so local test runs were not actually leaking events. But that
safety relied entirely on every call site remembering to set the env var,
with no way to prove it. Add a MEDIA_USE_TELEMETRY_HOST override read at the
point telemetry.mjs builds its POST url (falling back to the real endpoint
when unset), and a test that points it at a local HTTP server to assert a
real event is actually intercepted rather than trusting the env-var default
alone.
@miguel-heygen miguel-heygen force-pushed the feat/heygen-conversion-telemetry branch from f5d6abf to e2ed3c6 Compare July 10, 2026 16:55
Warn once on stderr if MEDIA_USE_TELEMETRY_HOST is ever set outside a
test/CI context, since the existing catch{} around the network call
would otherwise swallow a silent redirect with zero signal. Also widen
the rate_limited classifier to catch "too many requests" (the literal
HTTP 429 reason phrase) and "throttled", which were falling through to
"other".
reportHeygenFailure's telemetry call was fire-and-forget with no join
point, and its callers (voice-provider.mjs, heygen-search.mjs) are sync
call sites that can't await it themselves. On the resolve-miss path,
process.exit() could follow shortly after, racing the still-in-flight
request on its own non-keepalive connection and dropping the event.

Track each pending report in a module-level set and flush it before the
miss/success telemetry a process.exit() may follow.

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff end-to-end (all 9 files), traced the telemetry send path + flush ordering, and checked the classifier for exhaustiveness. Code LGTM — no correctness blockers. Telemetry-privacy invariant holds (reason is a fixed enum, never raw stderr; auth_method is oauth/api_key/absent; undefined dropped by JSON.stringify), and the interception-seam test isolation is a real improvement.

✅ Update (re-review 2026-07-10): all prior blockers resolved — merge-ready from my side.

  • CI is green: the Skills: manifest in sync job now passes (manifest regenerated).
  • Email-case identity split fixed: heygenAccountDistinctId now .trim().toLowerCase(), with a matching test. Confirmed the heygen-cli side (identityKey) lowercases symmetrically, so the cross-tool PostHog join actually holds.

Non-blocking nits (optional):

  1. resolve.mjs:420-424...heygenAuthMethodFor(provider) is spread before ...searchResult.metadata?.provenance. Inert today, but a future provider emitting authMethod in provenance would shadow the computed value. Consider spreading it after (the computed value is the intended source of truth).
  2. telemetry.mjsisTestOrCiContext() and optedOut() are near-identical env predicates for different purposes; a cross-referencing comment/shared helper would prevent drift. No bug.
  3. heygen.mjsheygenAuthMethod() and heygenAuthHeaders() independently encode "Authorization ⇒ OAuth"; a shared isOauthCred() would single-source it.
  4. heygen-cli.mjs rate-limit classifier substring matches ("quota"/"throttled") on combined stderr+stdout have an inherent precision ceiling — acceptable since reason is a coarse grouping enum.

Questions (non-blocking):

  1. Cache-hit auth_method replays how the asset was originally fetched, not the current resolve's auth. I think that's correct for a conversion signal — worth a one-line comment since the two paths compute it differently.
  2. reused-explicit (resolve.mjs:1062) preserves the cached authMethod — confirming that's intended, consistent with Q1.

— Somu

… harden test isolation

- skills-manifest.json was stale after the last commit, failing the
  manifest-in-sync CI check.
- heygenAccountDistinctId() now lowercases the email/username it returns,
  so it joins with heygen-cli's own identify call regardless of the
  account's stored casing -- an uppercase-email user would otherwise
  split across two PostHog profiles.
- The telemetry-interception test now sandboxes HEYGEN_CONFIG_DIR
  alongside HOME, so a developer with that var pointing at real
  credentials can't have their real email read into the test's local
  server payload.
- Two clarifying comments: the provenance spread order in resolve.mjs,
  and why a cache-hit's auth_method reports the original fetch's
  credential, not the current resolve's.
@jrusso1020

Copy link
Copy Markdown
Collaborator

E2E verification (complement to Somu/tai's code passes; deferring approval to Terence/Somansh per the cross-tool-telemetry policy).

Ran f0143f26 in a throwaway worktree, exercised the real emit paths, and intercepted the PostHog payloads locally (nothing sent to prod PostHog):

  • Builds + media-use tests greentelemetry.test.mjs 12/12, heygen-cli.test.mjs 20/20, resolve.test.mjs green (incl. the MEDIA_USE_TELEMETRY_HOST interception-seam test); regression "lowercased email regardless of casing" passes.
  • Identity join → one profile (verified): real resolve emitted $identify distinct_id=demo@example.com $anon_distinct_id=shared-anon-e2e-0001 and media_use_resolve distinct_id=shared-anon-e2e-0001; config anonymousId unchanged (read, not regenerated). Same lowercased-email id + same shared ~/.hyperframes/config.json anon as heygen-cli feat(skills): add scene transitions skill with 35-type catalog #212 → single PostHog person; the uppercase-email split does not occur.
  • No gaps: every event fires on every real path; media_use_provider_error awaited before every process.exit. Deliberate non-emit: --help/--stats/--adopt, pre-flight validation errors.
  • PII-clean from captured payloads: props {type,source,provider,local_only,provider_override,surface,$ip:null} — intent text absent, no paths.
  • CI 100% green.

Merge-ready on the e2e dimension.

— Rames Jusso

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants