Skip to content

fix(cli): track Android service account onboarding actions#2327

Merged
WcaleNieWolny merged 3 commits into
mainfrom
codex/android-sa-onboarding-telemetry
May 22, 2026
Merged

fix(cli): track Android service account onboarding actions#2327
WcaleNieWolny merged 3 commits into
mainfrom
codex/android-sa-onboarding-telemetry

Conversation

@WcaleNieWolny
Copy link
Copy Markdown
Contributor

@WcaleNieWolny WcaleNieWolny commented May 22, 2026

Summary (AI generated)

  • Adds explicit PostHog action events for Android service-account onboarding method selection, validation results, and recovery choices.
  • Persists a serviceAccountForkSeen marker so quitting at the service-account method fork resumes back to the fork instead of defaulting to OAuth.
  • Adds focused Android resume-routing coverage for the new service-account fork marker and a request-shape test for onboarding action telemetry.

Motivation (AI generated)

The imported service-account onboarding path was visible through generic step telemetry, but did not expose the key user choices needed to understand the funnel. It also had an edge-case resume gap: a fresh user who reached the method-selection fork and quit before choosing could be resumed into the legacy OAuth path because no method value had been persisted yet.

Business Impact (AI generated)

  • Improves observability for the Android cloud-build onboarding funnel without collecting service-account file paths or picker/manual source details.
  • Reduces accidental funnel skew by restoring users to the service-account fork when they quit before choosing.
  • Helps the team distinguish validation failures and recovery choices so onboarding friction can be prioritized with better data.

Test Plan (AI generated)

  • bun run test:android-onboarding-progress
  • bun run test:onboarding-telemetry
  • bun run test:android-service-account-validation
  • bun run lint
  • bun run typecheck
  • bun run build

Note: I also tried a direct one-off ESLint run on cli/src/build/onboarding/android/ui/app.tsx; that file reports broad pre-existing TSX style issues under the root config, while the official CLI lint script does not target TSX files.

Summary by CodeRabbit

  • New Features

    • Improved Android onboarding resumption to correctly route between service-account and legacy OAuth flows.
    • Expanded onboarding telemetry to record action events (method selection, validation results, recovery choices) and buffer actions until org resolution.
    • Persisted a marker to ensure correct resume behavior after keystore/service-account steps.
  • Tests

    • Added CLI tests for Android onboarding resume routing and onboarding telemetry.
  • Chores

    • Added a dedicated test script to the test chain for Android onboarding progress.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 22, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 517489c1-d14b-4d26-b0e3-cc1da21b1d62

📥 Commits

Reviewing files that changed from the base of the PR and between c07babc and dc42198.

📒 Files selected for processing (4)
  • cli/src/build/onboarding/android/ui/app.tsx
  • cli/src/build/onboarding/telemetry.ts
  • cli/test/test-android-onboarding-progress.mjs
  • cli/test/test-onboarding-telemetry.mjs
💤 Files with no reviewable changes (1)
  • cli/src/build/onboarding/android/ui/app.tsx

📝 Walkthrough

Walkthrough

This PR adds a serviceAccountForkSeen marker and hasAnyOAuthProgress helper to influence Android onboarding resume routing, implements buffered “Builder Onboarding Action” telemetry with a trackBuilderOnboardingAction API and UI integration (queued emission until org id resolves), persists the fork marker in keystore completion paths, and adds tests plus a package.json test script.

Changes

Android service-account fork resume routing and telemetry

Layer / File(s) Summary
Type contract and resume routing logic
cli/src/build/onboarding/android/types.ts, cli/src/build/onboarding/android/progress.ts
AndroidOnboardingProgress adds optional serviceAccountForkSeen. Adds hasAnyOAuthProgress(progress) and updates getAndroidResumeStep to route to service-account-method-select when the fork marker is present, no method is chosen, and no OAuth progress exists.
Telemetry infrastructure for onboarding actions
cli/src/build/onboarding/telemetry.ts
Introduces BuilderOnboardingAction and TrackBuilderOnboardingActionInput, and trackBuilderOnboardingAction which builds a string-only tags map and sends a Builder Onboarding Action event via sendEvent, swallowing errors.
UI telemetry integration and fork marker persistence
cli/src/build/onboarding/android/ui/app.tsx
Adds pendingActionTelemetryRef queue, drains queued actions when resolvedOrgId becomes available, centralizes emission in trackAction, emits telemetry for method selection, validation results, and recovery choices, and persists serviceAccountForkSeen: true in keystore completion paths; resume decisions refactored to use hasAnyOAuthProgress.
Tests and package script
cli/package.json, cli/test/test-android-onboarding-progress.mjs, cli/test/test-onboarding-telemetry.mjs
Adds test:android-onboarding-progress and includes it in the composite test script. Adds CLI tests covering resume-routing permutations and a telemetry test that verifies /private/events request structure and payload serialization.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

  • Cap-go/capgo#2317: Related Android service-account fork resume-routing changes that this PR builds on.

Suggested labels

codex

Suggested reviewers

  • zinc-builds
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding telemetry tracking for Android service account onboarding actions.
Description check ✅ Passed The description includes Summary, Motivation, Business Impact, and a comprehensive Test Plan with multiple test scripts run and checks passed.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands and usage tips.

@socket-security
Copy link
Copy Markdown

socket-security Bot commented May 22, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedhusky@​9.1.71001006280100
Addedmicromatch@​4.0.810010010083100

View full report

@codspeed-hq
Copy link
Copy Markdown
Contributor

codspeed-hq Bot commented May 22, 2026

Merging this PR will not alter performance

✅ 43 untouched benchmarks
⏩ 2 skipped benchmarks1


Comparing codex/android-sa-onboarding-telemetry (dc42198) with main (6ab0cec)

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@WcaleNieWolny WcaleNieWolny marked this pull request as ready for review May 22, 2026 07:40
@coderabbitai coderabbitai Bot added the codex label May 22, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 39e6b941-f044-4476-b0ae-db37f5dd7298

📥 Commits

Reviewing files that changed from the base of the PR and between 6ab0cec and 72cc087.

📒 Files selected for processing (6)
  • cli/package.json
  • cli/src/build/onboarding/android/progress.ts
  • cli/src/build/onboarding/android/types.ts
  • cli/src/build/onboarding/android/ui/app.tsx
  • cli/src/build/onboarding/telemetry.ts
  • cli/test/test-android-onboarding-progress.mjs

Comment thread cli/test/test-android-onboarding-progress.mjs Outdated
Comment thread cli/src/build/onboarding/android/ui/app.tsx Fixed
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
cli/src/build/onboarding/android/ui/app.tsx (1)

949-950: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don't retrofit the fork marker onto repaired legacy progress files.

These writes are unconditional, so a pre-fork progress file that re-enters the keystore phase for self-healing now gets serviceAccountForkSeen: true on save. On the next resume, getAndroidResumeStep(...) treats that file as a post-fork run and can send the user to service-account-method-select instead of preserving the legacy OAuth default this PR is explicitly trying to keep.

🐛 Proposed fix
           await persist((p) => ({
             ...p,
             keystoreKeyPassword: keyPw,
             _keystoreBase64: base64,
-            serviceAccountForkSeen: true,
+            ...(p.serviceAccountForkSeen || !p.completedSteps.keystoreReady
+              ? { serviceAccountForkSeen: true }
+              : {}),
             completedSteps: { ...p.completedSteps, keystoreReady: ready },
           }))

Apply the same guard in the generated-keystore path and the manual key-password completion path so only fresh post-fork runs gain the marker, while repaired legacy files preserve its absence.

Also applies to: 1005-1006, 1776-1777

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cli/src/build/onboarding/android/ui/app.tsx` around lines 949 - 950, The
writes unconditionally set serviceAccountForkSeen when updating
completedSteps.keystoreReady, which retrofits the fork marker onto repaired
legacy progress files; change the generated-keystore and manual key-password
completion paths so they only set serviceAccountForkSeen = true when the run is
actually post-fork (e.g. check an isPostForkRun flag or the existing
serviceAccountForkSeen value on the loaded progress before mutating), otherwise
leave serviceAccountForkSeen unchanged so getAndroidResumeStep(...) will
continue to treat repaired legacy files as pre-fork; update the code paths that
set completedSteps.keystoreReady to apply this guard.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cli/src/build/onboarding/telemetry.ts`:
- Around line 77-78: The loop currently copies input.tags into tags after
reserved fields are set, allowing callers to overwrite them; change the logic in
the telemetry helper so caller tags are merged first and then the reserved
fields are written (or alternatively skip reserved names inside the loop).
Specifically, update the code that iterates over input.tags (the for (const
[key, value] of Object.entries(input.tags ?? {})) block) to either (a) copy
caller tags first and then assign the reserved keys step, platform, app_id,
action afterwards, or (b) ignore keys named "step", "platform", "app_id", and
"action" during the loop so the helper's reserved values cannot be overwritten.

---

Outside diff comments:
In `@cli/src/build/onboarding/android/ui/app.tsx`:
- Around line 949-950: The writes unconditionally set serviceAccountForkSeen
when updating completedSteps.keystoreReady, which retrofits the fork marker onto
repaired legacy progress files; change the generated-keystore and manual
key-password completion paths so they only set serviceAccountForkSeen = true
when the run is actually post-fork (e.g. check an isPostForkRun flag or the
existing serviceAccountForkSeen value on the loaded progress before mutating),
otherwise leave serviceAccountForkSeen unchanged so getAndroidResumeStep(...)
will continue to treat repaired legacy files as pre-fork; update the code paths
that set completedSteps.keystoreReady to apply this guard.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 26b04ad6-7c1f-4e92-b16c-44a5f2de4096

📥 Commits

Reviewing files that changed from the base of the PR and between 72cc087 and c07babc.

📒 Files selected for processing (7)
  • cli/package.json
  • cli/src/build/onboarding/android/progress.ts
  • cli/src/build/onboarding/android/types.ts
  • cli/src/build/onboarding/android/ui/app.tsx
  • cli/src/build/onboarding/telemetry.ts
  • cli/test/test-android-onboarding-progress.mjs
  • cli/test/test-onboarding-telemetry.mjs

Comment thread cli/src/build/onboarding/telemetry.ts
@WcaleNieWolny WcaleNieWolny merged commit 2d18719 into main May 22, 2026
35 checks passed
@WcaleNieWolny WcaleNieWolny deleted the codex/android-sa-onboarding-telemetry branch May 22, 2026 08:25
@sonarqubecloud
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant