Investigate using new vscode experiment service

[IanMatthewHuff]

There is a new vscode experiment service that we could consider using ipykernel if we have time. Info pasted from e-mail below.

Hi All,

I hope this will be an appropriate inaugural post to this DL.

tl;dr You can now run controlled experiments for VS Code extensions with a few lines of code.

We just wrapped up work on integrating the Live Share VS Code extension with the ExP platform and I thought I would share because this work should be easily reusable by other teams looking to run controlled experiments (it’s already being reused by IntelliCode!).

Some background on the ExP Platform:
Built and maintained by the Analysis and Experimentation (A&E) team designed to enable products across Microsoft (including Bing, XBox, Office, and Visual Studio) to run and analyze large-scale and trustworthy controlled experiments. If you've run an experiment in Visual Studio, you're already familiar with it. It provides two important capabilities:
· A treatment assignment service (TAS) – a service that will assign users to different experiments and experiment groups – and a web app for defining experiments to be used by the TAS.
· A system for defining and generating scorecards – a set of metrics computed for all groups and segments in an experiment.

Before going into details of the integration and how to leverage it, it’s probably helpful to make things more concrete by showing the e2e of running an experiment:

· Define the experiment in control tower – name the experiment and specify different phases and the populations each phase targets. You can specify traffic filters (e.g. only target users on VS Code 1.41) and what percent of the eligible population you want placed in your experiment. Example experiment.

· Have your code check if the user is part of the experiment (more info on how to get an instance of expService below).

· Generate a scorecard and see if you moved the metrics you wanted to move, while not moving the ones you didn’t.

A few nice things:
· Ensures users are correctly distributed between treatment and control, even with multiple experiments running simultaneously.
· Traffic filters. E.g. we can release an experiment to users on our team, then to MSFT internal users, then to our insiders ring, then to the public. All from control tower.
· Remote control. We can remotely shut-off an experiment with no code changes. This + traffic filters makes this a great setup for feature rollouts.
· Scorecards
-Don't need to manually run Kusto queries. Also ensures that you're looking holistically at the effects of your experiment, instead of only considering a single metric.
-Baked-in statistical analysis.
-Segments – metrics can be sliced to different cohorts of users.

The work of integrating with the ExP Platform consisted of:
Setting up a new ExP tenant for VS Code (so experiments defined for VS Code don't land in other products and vice versa).
Writing a node client library for communicating with the TAS.
Writing a VS Code-specific library that consumes the previously mentioned generic library and provides VS Code implementations of some of the generic interfaces.
Setting up a scorecard.

All of this can be reused by other VS Code extensions with the exception of the scorecard, although I'd be happy to provide some guidance on how to go about setting one up, as would the folks who helped me set one up (thanks @julie Stevenson and @amy Hartwig!).

Appendix
How to consume the TAS node package:
Add a reference to it.
Initialize it. You need to pass in your extension name and version, the population the user is a part of (Team, Internal, Insider, or Public), an implementation of IExperimentationTelemetry (methods for sending a telemetry event and setting a shared property), and the extension context (so we can use the memento storage for your extension to cache experiment info).
Call isFlightEnabledAsync (makes a call to the TAS) or isCachedFlightEnabled (reads cached experiment data, updated in the background every 30 minutes) on the experimentation service using the same flight names you specified when you set up the experiment in control tower.
Based on the result, go down the treatment or control code path.

I’d love to help you leverage this work, so please reach out if you’re interested.

Best,

Dayton

[kimadeline]

Work item for the integration of the ExP platform in the extension: microsoft/vscode-python#10790