onboarding: gate experiment assignment-context id until would-show

[benibenj]

Problem

Onboarding experiments attach their assignment-context identifier to every telemetry event from application startup — long before the user has seen (or been eligible to see) the onboarding experience. Because scorecards are computed from tagged telemetry, this pollutes the A/B baseline with a window of pre-onboarding activity the experiment could not have influenced, weakening retention/success signals for both arms.

Approach

An onboarding scenario can now declare an experiment with two pre-determined ExP treatment flags:

• behaviorFlag (boolean) — true shows the tour (treatment), false does not (control).
• assignmentContextIdFlag (string) — the identifier the scorecard keys on. It must begin with the reserved onb- prefix.

How it works:

1. Eager gate. A telemetry assignment filter is registered in the service constructor and blocks any onb--prefixed id from the very first event — solving the race where treatment flags resolve after the assignment context is already committed.
2. Active only when fully configured. The experiment is active only when both flags resolve and the id carries the onb- prefix. A non-empty id missing the prefix is rejected (treated as inactive) and reported via onUnexpectedError, so a misconfiguration can never leak an ungated id.
3. Would-show opens the gate. When the user reaches the point where the tour would show (eligibility met, not already shown), the gate opens for that id — in both arms — and the decision is persisted (APPLICATION/MACHINE) so it survives reload/relaunch until the experiment stops. Treatment then shows the tour; control shows nothing and stays eligible.
4. Re-run safety. The discriminator is simply did the user actually see the tour. Treatment users are marked shown (so a later experiment's new id can't re-enroll them); control users remain eligible to take part in a future experiment with a different id.

Telemetry

Adds onboarding.scenarioOutcome (emitted only when a tour is actually shown): scenarioId, outcome, dismissReason (skipButton / escapeKey / targetClick / completed / aborted), lastStepIndex, stepCount, durationMs, experimentActive. The spotlight presentation now threads the dismiss reason and furthest step reached through its run result.

Notes

• The existing newSessionTour was intentionally not placed behind an experiment.
• On-demand runScenario (command/replay) bypasses gating and does not open the telemetry gate.

Tests

onboardingService.test.ts covers: both-flags-required, prefix rejection, treatment-shows-and-opens-gate, control-opens-gate-but-no-show-and-stays-eligible, gate persists across reload, and re-run contamination. Validated with npm run typecheck-client (clean) and the onboarding + spotlight unit tests (passing in Chromium).

[Copilot]

Pull request overview

Updates the onboarding scenario engine to support experiment-driven assignment-context gating, so onb- assignment-context identifiers don’t pollute telemetry until the user reaches the “would-show” moment, and enriches onboarding run results to capture dismiss reasons and progress for telemetry.

Changes:

• Replace the single experiment string with a two-flag experiment configuration (behaviorFlag + assignmentContextIdFlag) and add the reserved onb- prefix contract.
• Add an eager telemetry assignment filter and persisted “opened id” gate, and emit new onboarding.scenarioOutcome telemetry based on richer presentation run results.
• Thread richer run results (dismiss reason, furthest step, step count) through Spotlight presentation/overlay and expand unit tests for gating/persistence behaviors.

Show a summary per file

File	Description
src/vs/workbench/contrib/onboarding/browser/onboardingService.ts	Implements experiment flag resolution, eager telemetry gating/filtering, gate persistence, and scenario outcome telemetry logging.
src/vs/workbench/contrib/onboarding/common/onboardingScenario.ts	Introduces experiment configuration interface, reserved assignment-context prefix, and richer run result/dismiss reason types.
src/vs/workbench/contrib/onboarding/common/onboardingPresentation.ts	Updates the presentation contract to return a structured run result instead of a bare outcome.
src/vs/workbench/contrib/onboarding/browser/spotlight/spotlightPresentation.ts	Produces structured run results (dismiss reason/progress) and handles “advance via target” telemetry details.
src/vs/workbench/contrib/onboarding/browser/spotlight/spotlightOverlay.ts	Emits richer step-end signals (advance source, skip reason) needed for dismiss-reason telemetry.
src/vs/workbench/contrib/onboarding/test/browser/onboardingService.test.ts	Adds tests for two-flag activation, prefix rejection, gate opening/persistence, and re-run contamination behavior.

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 3