config: make all settings experiment-aware by default by sandy081 · Pull Request #300437 · microsoft/vscode

sandy081 · 2026-03-10T12:26:27Z

Summary

Make every configuration setting experiment-aware by default, instead of requiring individual opt-in via the experiment property.

Changes

All settings are now experiment-aware

Removed the experiment property guard in ConfigurationDefaultOverridesContribution — all registered settings are now processed for experiment overrides using the treatment name config.${settingId}.

Simplified `experiment` → `experimentMode`

Renamed the experiment property on IConfigurationPropertySchema to experimentMode for clarity — it now only controls the refresh mode, not whether the setting participates in experiments.
Changed the type from { mode: 'startup' | 'auto'; name?: string } to just 'startup' — the name field was unused and auto is now the default.
Removed the redundant experimentMode: 'auto' declarations across ~40 setting registrations since auto is the default mode.

Extension settings default to startup mode

All extension-contributed configuration properties are unconditionally set to experimentMode: 'startup' since extensions cannot control this property and startup-only is the safe default.

Tests

Added configurationDefaultOverrides.test.ts with tests covering:
- Experiment treatment applied to settings without explicit experimentMode
- Correct experiment name format (config.{settingId})
- No override when treatment value equals the default
- Auto mode (default): re-applies on refetch
- Startup mode: does not re-apply on refetch
- No-op when treatment returns undefined

…emas

…tests for configuration overrides

…figuration

vs-code-engineering · 2026-03-10T12:26:49Z

📬 CODENOTIFY

The following users are being notified based on files changed in this PR:

@bpasero

Matched files:

src/vs/workbench/browser/workbench.contribution.ts

@kycutler

Matched files:

src/vs/workbench/contrib/browserView/electron-browser/browserView.contribution.ts

@jruales

Matched files:

src/vs/workbench/contrib/browserView/electron-browser/browserView.contribution.ts

@jrieken

Matched files:

src/vs/workbench/contrib/inlineChat/common/inlineChat.ts

Copilot

Pull request overview

This PR makes configuration defaults experiment-overridable by default, using a standardized treatment name format (config.${settingId}), and simplifies the schema surface from an opt-in experiment object to an experimentMode refresh control (startup-only vs default auto refresh).

Changes:

Apply experiment overrides to all registered settings by default, with treatments read as config.${settingId}.
Replace experiment schema property with experimentMode?: 'startup' and update built-in settings accordingly.
Force extension-contributed settings to experimentMode: 'startup' and add new tests for default override behavior.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/vs/workbench/services/configuration/test/browser/configurationDefaultOverrides.test.ts	Adds unit tests for experiment-driven default overrides and refresh behavior.
src/vs/workbench/services/configuration/browser/configurationService.ts	Removes per-setting opt-in and applies experiment processing broadly; switches refresh behavior to `experimentMode`.
src/vs/workbench/contrib/welcomeGettingStarted/browser/gettingStarted.contribution.ts	Removes now-redundant `experiment: { mode: 'auto' }` declaration.
src/vs/workbench/contrib/terminalContrib/chatAgentTools/common/terminalChatAgentToolsConfiguration.ts	Removes redundant `experiment: { mode: 'auto' }` declarations from terminal chat agent tools settings.
src/vs/workbench/contrib/terminal/common/terminalConfiguration.ts	Removes redundant `experiment: { mode: 'auto' }` declarations from terminal settings.
src/vs/workbench/contrib/performance/electron-browser/performance.contribution.ts	Migrates a startup-only experiment declaration to `experimentMode: 'startup'`.
src/vs/workbench/contrib/inlineChat/common/inlineChat.ts	Removes redundant auto experiment declarations; keeps startup-only via `experimentMode`.
src/vs/workbench/contrib/extensions/browser/extensions.contribution.ts	Removes redundant `experiment: { mode: 'auto' }` from an extensions setting.
src/vs/workbench/contrib/editTelemetry/browser/editTelemetry.contribution.ts	Removes redundant auto experiment declarations from edit telemetry settings.
src/vs/workbench/contrib/chat/browser/chat.contribution.ts	Removes redundant auto experiment declarations and migrates startup-only to `experimentMode`.
src/vs/workbench/contrib/browserView/electron-browser/browserView.contribution.ts	Migrates startup-only experiment declaration to `experimentMode: 'startup'`.
src/vs/workbench/browser/workbench.contribution.ts	Removes redundant `experiment: { mode: 'auto' }` declarations from workbench settings.
src/vs/workbench/api/common/configurationExtensionPoint.ts	Forces extension-contributed settings into startup-only experiment refresh mode.
src/vs/platform/request/common/request.ts	Removes redundant `experiment: { mode: 'auto' }` declarations from request/proxy settings.
src/vs/platform/configuration/common/configurationRegistry.ts	Replaces `experiment` schema with `experimentMode` and updates validation/messaging around the legacy `onExP` tag.
src/vs/editor/common/config/editorOptions.ts	Removes redundant `experiment: { mode: 'auto' }` declarations from editor option registrations.
src/vs/editor/common/config/editorConfigurationSchema.ts	Removes redundant `experiment: { mode: 'auto' }` declarations from editor configuration schema.

Comments suppressed due to low confidence (3)

src/vs/workbench/services/configuration/test/browser/configurationDefaultOverrides.test.ts:167

setting without experimentMode defaults to auto and re-applies on refetch currently has no assertions, so it can pass even if the refetch path doesn’t actually re-apply anything. Please assert an observable outcome (e.g. check the defaults override value after first apply and after refetch, or verify getTreatment was called again and the effective default override changed from 42 to 99).

	test('setting without experimentMode defaults to auto and re-applies on refetch', async () => {
		configurationRegistry.registerConfiguration({
			id: 'test.experiments.autoDefault',
			properties: {
				'test.experiments.autoDefault.setting': {
					type: 'number',
					default: 0,
				}
			}
		});
		localDisposables.add({ dispose: () => configurationRegistry.deregisterConfigurations([{ id: 'test.experiments.autoDefault', properties: { 'test.experiments.autoDefault.setting': { type: 'number', default: 0 } } }]) });

		assignmentService.setTreatment('config.test.experiments.autoDefault.setting', 42);
		createContribution();

		await Event.toPromise(configurationRegistry.onDidUpdateConfiguration);

		// Now change the treatment and refetch
		assignmentService.setTreatment('config.test.experiments.autoDefault.setting', 99);
		assignmentService.fireRefetch();

		await Event.toPromise(configurationRegistry.onDidUpdateConfiguration);
	});

src/vs/workbench/services/configuration/test/browser/configurationDefaultOverrides.test.ts:200

The refetch test for startup mode also uses a fixed setTimeout delay. Consider asserting refetch behavior via a getTreatment call counter/event on the mock (startup mode should not trigger another read) so the test remains deterministic and faster.

		// Now change the treatment and refetch — startup mode should NOT re-apply
		let overrideRegistered = false;
		const listener = localDisposables.add(configurationRegistry.onDidUpdateConfiguration(() => {
			overrideRegistered = true;
		}));
		assignmentService.setTreatment('config.test.experiments.startupMode.setting', 99);
		assignmentService.fireRefetch();

		// Give the async processing time
		await new Promise(resolve => setTimeout(resolve, 50));

		assert.strictEqual(overrideRegistered, false, 'Startup mode setting should not be refetched');
		listener.dispose();
	});

src/vs/workbench/services/configuration/browser/configurationService.ts:1396

This change adds almost every setting (anything not explicitly experimentMode: 'startup') into autoExperimentalSettings, which means onDidRefetchAssignments will re-run processExperimentalSettings over (nearly) the entire settings set. Even if only a handful of treatments are active, refetch will still do the full scan unless the set is narrowed. Consider tracking only settings that currently have an active experiment override (or deriving the list from the refetched assignment context) so refetch work scales with #active treatments rather than #settings.

			this.processedExperimentalSettings.add(property);
			if (schema.experimentMode !== 'startup') {
				this.autoExperimentalSettings.add(property);
			}

src/vs/workbench/services/configuration/browser/configurationService.ts

src/vs/workbench/services/configuration/test/browser/configurationDefaultOverrides.test.ts

Copilot · 2026-03-10T12:31:09Z

src/vs/workbench/services/configuration/browser/configurationService.ts

 			await this.processExperimentalSettings(Object.keys(this.configurationRegistry.getConfigurationProperties()), false);
 		} finally {
 			// Invalidate defaults cache after extensions have registered
 			// and after the experiments have been resolved to prevent
 			// resetting the overrides too early.
 			await this.extensionService.whenInstalledExtensionsRegistered();
 			this.logService.trace('ConfigurationService#updateDefaults: resetting the defaults');
 			this.configurationService.reloadConfiguration(ConfigurationTarget.DEFAULT);
 		}
 	}

 	private async processExperimentalSettings(properties: Iterable<string>, autoRefetch: boolean): Promise<void> {
 		const overrides: IStringDictionary<unknown> = {};
 		const allProperties = this.configurationRegistry.getConfigurationProperties();
 		const defaultConfigurationsPreventingExperimentOverrides = this.configurationRegistry.getRegisteredDefaultConfigurations().filter(configuration => configuration.preventExperimentOverride);
 		for (const property of properties) {
 			const schema = allProperties[property];
-			if (!schema?.experiment) {
+			if (!schema) {
 				continue;
 			}
 			const defaultValueSource: ConfigurationDefaultSource | undefined = schema.defaultValueSource && !(schema.defaultValueSource instanceof Map) ? schema.defaultValueSource : undefined;
 			if (defaultValueSource && defaultConfigurationsPreventingExperimentOverrides.some(configuration => isConfigurationDefaultSourceEquals(configuration.source, defaultValueSource) && configuration.overrides?.[property] !== undefined)) {
 				continue;
 			}
 			if (!autoRefetch && this.processedExperimentalSettings.has(property)) {
 				continue;
 			}
 			this.processedExperimentalSettings.add(property);
-			if (schema.experiment.mode === 'auto') {
+			if (schema.experimentMode !== 'startup') {
 				this.autoExperimentalSettings.add(property);
 			}
 			try {
-				const value = await this.workbenchAssignmentService.getTreatment(schema.experiment.name ?? `config.${property}`);
+				const value = await this.workbenchAssignmentService.getTreatment(`config.${property}`);
 				if (!isUndefined(value) && !equals(value, schema.default)) {
 					overrides[property] = value;


processExperimentalSettings now iterates over all configuration properties and awaits workbenchAssignmentService.getTreatment for each one. In production getTreatment can block on TAS initialization/network and also logs a telemetry event per call, so this becomes an O(#settings) async loop with potentially thousands of treatment reads and telemetry logs on startup and on each assignment refetch. Consider switching to a batched approach (e.g. parse getCurrentExperiments() / assignment context once into a map of config.* treatments, or add an API to fetch all config.* variables in one shot) and only process keys that actually have a treatment/value present.

This issue also appears on line 1393 of the same file.

- Track requested treatment names in MockAssignmentService - Assert treatment lookup uses config.{settingId} format - Assert override is applied and refetch re-requests treatment for auto mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add whenTreatmentRequested/whenNextTreatmentRequested to mock - Use sentinel auto-mode setting to detect refetch completion - All tests now await specific signals instead of sleeping Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

defaultValueSource is not set when no source is provided to registerDefaultConfigurations. Use getConfigurationDefaultsOverrides() instead to verify overrides were actually registered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This reverts commit e9eb2fd.

…" (#300467) This reverts commit e9eb2fd.

sandy081 added 7 commits March 10, 2026 10:48

fix: improve experiment handling in configuration defaults

807fe07

refactor: simplify experiment property structure in configuration sch…

bc958bb

…emas

Copilot CLI session f7a6aaaa-477b-497b-bfe8-9a59c2c61ed8 changes

71c3555

refactor: remove 'experimentMode: auto' from configuration schemas

fbacbfe

feat: export ConfigurationDefaultOverridesContribution class and add …

86accc4

…tests for configuration overrides

fix: update experimentMode assignment logic in configuration properties

eecaae1

fix: ensure experimentMode is always set to 'startup' in property con…

2d7fc42

…figuration

Copilot AI review requested due to automatic review settings March 10, 2026 12:26

sandy081 self-assigned this Mar 10, 2026

sandy081 enabled auto-merge (squash) March 10, 2026 12:26

Copilot started reviewing on behalf of sandy081 March 10, 2026 12:27 View session

vs-code-engineering bot added this to the 1.112.0 milestone Mar 10, 2026

Copilot AI reviewed Mar 10, 2026

View reviewed changes

sandy081 and others added 4 commits March 10, 2026 13:51

fix: use mock<WorkspaceService>() for type-safe test mock

0e3625a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

alexr00 approved these changes Mar 10, 2026

View reviewed changes

sandy081 merged commit e9eb2fd into main Mar 10, 2026
19 checks passed

sandy081 deleted the sandy081/copilot/zoophagous-mackerel branch March 10, 2026 15:01

sandy081 added a commit that referenced this pull request Mar 10, 2026

Revert "config: make all settings experiment-aware by default (#300437)"

281f173

This reverts commit e9eb2fd.

sandy081 added a commit that referenced this pull request Mar 10, 2026

Revert "config: make all settings experiment-aware by default (#300437)…

ce4a1cf

…" (#300467) This reverts commit e9eb2fd.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config: make all settings experiment-aware by default#300437

config: make all settings experiment-aware by default#300437
sandy081 merged 11 commits intomainfrom
sandy081/copilot/zoophagous-mackerel

sandy081 commented Mar 10, 2026

Uh oh!

vs-code-engineering bot commented Mar 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sandy081 commented Mar 10, 2026

Summary

Changes

All settings are now experiment-aware

Simplified experiment → experimentMode

Extension settings default to startup mode

Tests

Uh oh!

vs-code-engineering bot commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📬 CODENOTIFY

@bpasero

@kycutler

@jruales

@jrieken

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Simplified `experiment` → `experimentMode`

vs-code-engineering bot commented Mar 10, 2026 •

edited

Loading