feat(media-use): classify and tag heygen provider telemetry, harden test isolation#2130
feat(media-use): classify and tag heygen provider telemetry, harden test isolation#2130miguel-heygen wants to merge 6 commits into
Conversation
Adds a stable reason-code classifier (not_found/not_authenticated/ outdated/rate_limited/other) alongside the existing display-message classifier, and reports it via a new media_use_provider_error event so quota/rate-limit failures are distinguishable from other provider errors even when a fallback provider later succeeds.
Adds a sparse auth_method property (oauth/api_key) to media_use_resolve telemetry for heygen-family providers, derived via a new heygenAuthMethod() helper alongside the existing heygenAuthHeaders(). Absent for every non-heygen provider.
…ception seam resolve.test.mjs already set DO_NOT_TRACK=1 for every spawned resolve.mjs invocation, so local test runs were not actually leaking events. But that safety relied entirely on every call site remembering to set the env var, with no way to prove it. Add a MEDIA_USE_TELEMETRY_HOST override read at the point telemetry.mjs builds its POST url (falling back to the real endpoint when unset), and a test that points it at a local HTTP server to assert a real event is actually intercepted rather than trusting the env-var default alone.
f5d6abf to
e2ed3c6
Compare
Warn once on stderr if MEDIA_USE_TELEMETRY_HOST is ever set outside a
test/CI context, since the existing catch{} around the network call
would otherwise swallow a silent redirect with zero signal. Also widen
the rate_limited classifier to catch "too many requests" (the literal
HTTP 429 reason phrase) and "throttled", which were falling through to
"other".
reportHeygenFailure's telemetry call was fire-and-forget with no join point, and its callers (voice-provider.mjs, heygen-search.mjs) are sync call sites that can't await it themselves. On the resolve-miss path, process.exit() could follow shortly after, racing the still-in-flight request on its own non-keepalive connection and dropping the event. Track each pending report in a module-level set and flush it before the miss/success telemetry a process.exit() may follow.
There was a problem hiding this comment.
Reviewed the diff end-to-end (all 9 files), traced the telemetry send path + flush ordering, and checked the classifier for exhaustiveness. Code LGTM — no correctness blockers. Telemetry-privacy invariant holds (reason is a fixed enum, never raw stderr; auth_method is oauth/api_key/absent; undefined dropped by JSON.stringify), and the interception-seam test isolation is a real improvement.
✅ Update (re-review 2026-07-10): all prior blockers resolved — merge-ready from my side.
- CI is green: the
Skills: manifest in syncjob now passes (manifest regenerated).- Email-case identity split fixed:
heygenAccountDistinctIdnow.trim().toLowerCase(), with a matching test. Confirmed the heygen-cli side (identityKey) lowercases symmetrically, so the cross-tool PostHog join actually holds.
Non-blocking nits (optional):
resolve.mjs:420-424—...heygenAuthMethodFor(provider)is spread before...searchResult.metadata?.provenance. Inert today, but a future provider emittingauthMethodin provenance would shadow the computed value. Consider spreading it after (the computed value is the intended source of truth).telemetry.mjs—isTestOrCiContext()andoptedOut()are near-identical env predicates for different purposes; a cross-referencing comment/shared helper would prevent drift. No bug.heygen.mjs—heygenAuthMethod()andheygenAuthHeaders()independently encode "Authorization ⇒ OAuth"; a sharedisOauthCred()would single-source it.heygen-cli.mjsrate-limit classifier substring matches ("quota"/"throttled") on combined stderr+stdout have an inherent precision ceiling — acceptable sincereasonis a coarse grouping enum.
Questions (non-blocking):
- Cache-hit
auth_methodreplays how the asset was originally fetched, not the current resolve's auth. I think that's correct for a conversion signal — worth a one-line comment since the two paths compute it differently. reused-explicit(resolve.mjs:1062) preserves the cachedauthMethod— confirming that's intended, consistent with Q1.
— Somu
… harden test isolation - skills-manifest.json was stale after the last commit, failing the manifest-in-sync CI check. - heygenAccountDistinctId() now lowercases the email/username it returns, so it joins with heygen-cli's own identify call regardless of the account's stored casing -- an uppercase-email user would otherwise split across two PostHog profiles. - The telemetry-interception test now sandboxes HEYGEN_CONFIG_DIR alongside HOME, so a developer with that var pointing at real credentials can't have their real email read into the test's local server payload. - Two clarifying comments: the provenance spread order in resolve.mjs, and why a cache-hit's auth_method reports the original fetch's credential, not the current resolve's.
|
E2E verification (complement to Somu/tai's code passes; deferring approval to Terence/Somansh per the cross-tool-telemetry policy). Ran
Merge-ready on the e2e dimension. — Rames Jusso |
Summary
media-use's heygen-cli integration could tell you a resolve succeeded, but not why it failed along the way, or whether a successful call went through the free (OAuth) or paid (API-key) path. This closes both gaps, plus a real test-isolation bug where local test runs could reach the production telemetry endpoint.
Companion change to heygen-com/heygen-cli#212 (identity unification + auth telemetry) — the two land independently since neither repo's release blocks the other.
What changed
Design decisions
reasonvalue, not the display message or raw stderr text — stays a small enum a dashboard can group by, not free text.Testing
oxlint/oxfmt --check: clean across all changed files.Post-Deploy Monitoring & Validation