Skip to content

feat: add core experimentation service for startup experiments #252566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 27, 2025

Conversation

bhavyaus
Copy link
Collaborator

@bhavyaus bhavyaus commented Jun 26, 2025

Core Experimentation Service for Startup Layout Experiments

This PR introduces a new CoreExperimentationService to manage A/B testing experiments for VS Code startup experiences.

How the Service Works (Pseudocode)

1. On service initialization:
   IF user's first session was more than 1 day ago:
     RETURN (not eligible for startup experiments)
   
   IF experiment already exists in storage:
     RETURN (user already participated)
   
   2. Generate random cohort (0-1)
   
   3. Check if user falls within target percentage:
      - Stable builds: 20% participation  
      - Insider builds: 20% participation
   
   4. IF user is selected:
      - Normalize cohort to experiment range
      - Assign to experiment group based on normalized value:
        * [0-25%): Control group
        * [25-50%): Maximized Chat
        * [50-75%): Split Empty Editor Chat  
        * [75-100%): Split Welcome Chat
      
      - Store experiment data in application storage
      - Set context key for UI consumption
      - Send telemetry event

Service Usage

// Get current user's experiment (returns undefined if not in experiment)
const exp:IExperiment = this.coreExperimentationService.getExperiment();

if (exp?.experimentGroup) {
// Valid values (see StartupExperimentGroup enum): 'control', 'maximizedChat', 'splitEmptyEditorChat', 'splitWelcomeChat'
}

Context Key Usage

The service sets a context key:

// Context key: 'coreExperimentation.startupExpGroup'
// Valid values (see StartupExperimentGroup enum): 'control', 'maximizedChat', 'splitEmptyEditorChat', 'splitWelcomeChat'

Experiment Configuration

  • Stable builds: 20% participation, 4 equal groups (25% each)
  • Insider builds: 20% participation, 4 equal groups (25% each)

@bhavyaus bhavyaus self-assigned this Jun 26, 2025
@bhavyaus bhavyaus force-pushed the dev/bhavyau/core-exp-service branch from d399383 to 296631f Compare June 26, 2025 23:58
@bhavyaus bhavyaus requested a review from bpasero June 27, 2025 05:31
@bhavyaus bhavyaus marked this pull request as ready for review June 27, 2025 05:31
@bhavyaus bhavyaus force-pushed the dev/bhavyau/core-exp-service branch from fa0bdae to f66262e Compare June 27, 2025 05:31
@bhavyaus bhavyaus changed the title feat: add core experimentation service for startup layout experiments feat: add core experimentation service for startup experiments Jun 27, 2025
@vs-code-engineering vs-code-engineering bot added this to the June 2025 milestone Jun 27, 2025
@bpasero bpasero enabled auto-merge (squash) June 27, 2025 11:08
@bpasero
Copy link
Member

bpasero commented Jun 27, 2025

@bhavyaus thanks, lets get this in in its current state. I think its a bit weird that it would not be generic but include a hardcoded set of startup experiments, likely making it harder if we ever want to use this for other experiments. But we can talk about that when we are there.

@bpasero bpasero merged commit a581c38 into main Jun 27, 2025
8 checks passed
@bpasero bpasero deleted the dev/bhavyau/core-exp-service branch June 27, 2025 11:12
lszomoru pushed a commit that referenced this pull request Jun 30, 2025
* feat: add core experimentation service and integrate it into the workbench

* refactor and fix up targetPercentage

* test: add unit tests for CoreExperimentationService functionality

* refactor: update experimentGroup types to use StartupExperimentGroup enum

---------

Co-authored-by: Benjamin Pasero <benjamin.pasero@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants