Skip to content

feat(auth): identify accounts and add structured login telemetry#212

Open
miguel-heygen wants to merge 6 commits into
mainfrom
feat/conversion-telemetry
Open

feat(auth): identify accounts and add structured login telemetry#212
miguel-heygen wants to merge 6 commits into
mainfrom
feat/conversion-telemetry

Conversation

@miguel-heygen

Copy link
Copy Markdown
Contributor

Summary

heygen-cli's own analytics reported to a separate PostHog project under a locally-minted identity, with no identify call on login — so a successful heygen auth login couldn't be joined to anything the rest of the toolchain already tracks, and login attempts only ever showed up as a generic command-run event with no method or failure reason. This wires heygen-cli into the same cross-tool identity and destination, and makes login itself first-class, queryable telemetry.

Companion change to a matching PR in heygen-com/hyperframes: heygen-com/hyperframes#2130 (media-use side: failure classification + free/paid tagging + test-isolation hardening). The two land independently — neither repo's release blocks the other.

What changed

  • Shared install identity: the analytics client's distinct id now comes from the same cross-tool install identity file other HeyGen CLIs already share, promoting any existing local id on first run so upgrading users don't reset their history.
  • Shared destination + surface tag: events now land in the same ingestion project as the rest of the toolchain, tagged with a surface property so a query can still isolate this CLI's traffic.
  • Account identify on login: a successful login (OAuth or API-key) now links the anonymous install identity to the signed-in account, the same way the rest of the toolchain already does — plus a one-time console notice disclosing this.
  • Structured login telemetry: auth login attempts now emit distinct started/completed/failed events carrying the method (OAuth vs. API-key) and, on failure, a specific reason drawn from the login flow's real failure branches (timeout, cancellation, invalid input, etc.) instead of a generic pass/fail.
  • Reconciliation fix: a handful of rare early-return error paths (malformed credentials, save/clear failures, PKCE/state generation, loopback startup) previously left a started event with no matching completed/failed; every path now reconciles through a safety-net terminal event.

Design decisions

  • Identity adoption is a read-merge-write against the shared identity file, never a blind overwrite — the file is written by more than one tool, so a write here must not drop a field another tool wrote moments earlier.
  • The anon-to-identified merge uses this repo's pinned analytics SDK version's actual merge primitive, verified against that SDK's source rather than assumed from a newer version's API.
  • Every new analytics call preserves the existing fire-and-forget contract (guarded, error-swallowing, non-blocking) — none of this can affect the login flow's own success/failure behavior.

Testing

  • Reviewed end-to-end (correctness, no auth-behavior change, fire-and-forget discipline, identity file handling, SDK usage, test coverage) with no blocking issues found.
  • Go toolchain unavailable in this environment for a live go test run; verified by careful manual reading and cross-checking against the pinned SDK's actual source.

Post-Deploy Monitoring & Validation

  • No additional operational monitoring required beyond normal telemetry review — this is additive, best-effort instrumentation with no change to auth flow behavior, credential storage, or login outcomes.
  • Expected healthy signal: no change in login success/failure rates or latency (every new call is fire-and-forget and swallows its own errors).

…nd structured login events

Switch the analytics client's distinct id to the same cross-tool install
identity already shared by the other HeyGen CLIs (promoting any existing
local id on first run so upgraders keep their identity), and point events
at the same ingestion project so a person's activity is queryable in one
place instead of two. Add a `surface` property to distinguish this CLI's
traffic there, plus IdentifyAccount/AuthLoginStarted/AuthLoginCompleted/
AuthLoginFailed methods so a login attempt becomes structured, queryable
telemetry (method + specific failure reason) instead of a generic event.
Link a successful login (OAuth or API-key) to the account identity via
PostHog identify, disclose telemetry + identity linking on first run, and
emit structured started/completed/failed events per login attempt with
method and failure reason so login funnels are queryable by branch instead
of a generic command-complete event.
…vent

runAPIKeyLogin and runOAuthLogin had several early-return error branches
(malformed credentials file, save/clear failures, PKCE/state generation,
loopback startup, authorize-URL build, response encoding) that returned
without emitting AuthLoginCompleted or AuthLoginFailed, leaving their
AuthLoginStarted event dangling with no outcome.

Rather than instrumenting each scattered return site, wrap both runners
with a deferred catch-all: a per-call `reported` flag is set at each
existing AuthLoginCompleted/AuthLoginFailed call, and a defer fires
AuthLoginFailed(method, "internal_error") only when the function returns
an error and nothing else already reported an outcome. New error paths
added later are covered automatically instead of silently reintroducing
the gap.
@miguel-heygen miguel-heygen force-pushed the feat/conversion-telemetry branch from 1d0a531 to 857dbd3 Compare July 10, 2026 18:06
runOAuthLogin and runAPIKeyLogin marked the login as completed and fired
AuthLoginCompleted before writing the success response via
ctx.formatter.Data. If that write failed (e.g. a broken pipe on stdout),
the command still exited non-zero but telemetry had already recorded a
completed login. Reorder so completion is only reported once the
response write actually succeeds; a write failure now falls through to
the existing deferred reconciliation, which reports AuthLoginFailed
instead.

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff (all 8 files, ~1324 lines), traced the auth + telemetry flow end-to-end, and cross-checked against the hyperframes scheme + the posthog-go SDK. LGTM — no blocking issues. The fire-and-forget contract holds and login behavior isn't altered. I specifically checked the classic login-telemetry leak vectors:

  • No secret/token/PII in event properties — AUTH_LOGIN_* carry only method + a fixed reason enum plus base props (command, cli_version, os, arch, surface, client_origin, $ip: null). No API key, OAuth token, or PKCE verifier.
  • The email/username is sent, but only as the PostHog distinct_id in IdentifyAccount — the intended, disclosed cross-tool account identity that matches hyperframes' email || username. By-design identity, not a leak.

On the Alias{DistinctId: account, Alias: anon} + Identify{DistinctId: account} pair vs hyperframes' single $identify + $anon_distinct_id: verified against posthog-go v1.11.2 (alias.go / identify.go / posthog.go) — Identify has no anon field and Alias serializes to $create_alias, so the two produce an equivalent bidirectional merge. Your reasoning is correct. The only residual difference: heygen-cli also emits a bare Identify (no $set props) alongside the alias — a harmless extra event.

Non-blocking nits:

  1. internal/analytics/analytics.goIdentifyAccount mutates c.identified without synchronization. No real race today (login is single-goroutine), but a one-line comment noting that assumption (like the one on Started()) would harden intent if analytics is ever invoked from the loopback/browser goroutines.
  2. main.go:98-103maybeShowTelemetryNotice writes its own telemetry_notice_shown flag in heygen-cli's config dir rather than the shared telemetryNoticeShown key. Consequence: a user who already saw the notice via media-use sees it again from heygen-cli (and vice versa). Product decision, not a bug — flagging so it's a conscious choice.
  3. auth_login.go device-code path fires AUTH_LOGIN_STARTED + AUTH_LOGIN_FAILED for the unsupported flag on every invocation. Fine — just confirm you want device-code attempts counted in the funnel (probably yes; it measures demand for the feature).

Questions:

  1. Was the Alias+Identify pair validated against the actual PostHog project to confirm the person-merge stitches identically in dashboards? The SDK reasoning checks out statically; a one-time live confirmation would fully close it.
  2. Your PR notes go test wasn't runnable in your env (no Go toolchain) — can we get a CI-green confirmation? Correctness here rests on reading rather than execution, and the test coverage of the new paths is genuinely strong, so a green run is the last mile.

— Somu

…join

An uppercase-cased email or username would otherwise split one HeyGen
account across two PostHog profiles, since hyperframes-oss's side of the
identity join already lowercases.
os.UserHomeDir reads USERPROFILE on Windows, not HOME, so these tests
leaked the real CI runner's home directory on windows-latest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants