[Feature] VS Code as a Playwright target

[mrmckeb]

I was recently tasked with writing tests for a VS Code extension. The provided test helper @vscode/test-electron is capable, but doesn't support ESM and needs TypeScript test files to be built before running tests, and it wasn't designed to support things like image regression testing.

I then remembered that Playwright could test Electron apps. I realised that today this is targeted at Electron app development, but it made me wonder if VS Code could be a target for Playwright?

This would help countless extension authors to validate their extensions (including Playwright). Looking at some popular examples, they all seem to be mocking VS Code, and are not performing automation testing.

• https://github.com/microsoft/playwright-vscode
• https://github.com/jest-community/vscode-jest
• https://github.com/vitest-dev/vscode

Some other popular extensions have only unit testing in place.

[mxschmitt]

Do you want to access WebViews from your VSCode extension or how do you want to leverage Playwright in that testing scenario.

[mrmckeb]

The goal is to test the extension itself, so Playwright would launch VS Code with params - as it does with browsers/Electron - and then it would run through tests just as I would with a browser. For example, if I click button X, the extension outputs an error.

Sorry I wasn't clearer in the initial description.

[ryankeysight]

I have a similar interest in testing vscode extensions with playwright, with the goal of running the same tests against both vscode on a desktop and vscode in a browser (such as code-server).

I have proven it is possible to launch vscode from playwright by using the electron executable in vscode. For example with typescript on macOS:
await electron.launch({ executablePath:'/Applications/Visual Studio Code.app/Contents/MacOS/Electron' })

The problem is I can only run this when my instance of vscode is not open. Otherwise the second instance opens and closes immediately which fails the test.

[jsphstls]

I was able to run in the Extension Development Host by using @vscode/test-electron.

Here's a simplified example:

import path from 'node:path'
import { downloadAndUnzipVSCode } from '@vscode/test-electron'
import { _electron as electron, test } from '@playwright/test'
import type { ElectronApplication } from '@playwright/test'

let electronApp: ElectronApplication

const rootPath = path.resolve(__dirname, 'path/to/your/extension-files')

const args = [
  '--disable-gpu-sandbox', // https://github.com/microsoft/vscode-test/issues/221
  '--disable-updates', // https://github.com/microsoft/vscode-test/issues/120
  '--disable-workspace-trust',
  '--extensionDevelopmentPath=' + rootPath,
  '--new-window', // Opens a new session of VS Code instead of restoring the previous session (default).
  '--no-sandbox', // https://github.com/microsoft/vscode/issues/84238
  '--profile-temp', // "debug in a clean environment"
  '--skip-release-notes',
  '--skip-welcome'
]

test.beforeEach(async () => {
  electronApp = await electron.launch({
    executablePath: await downloadAndUnzipVSCode(),
    args
  })
})

test('launches vscode', async () => {
  const page = await electronApp.firstWindow()

  await page.getByRole('button', { name: 'Some button' }).click()
})

test.afterEach(async () => {
  await electronApp.close()
})

[roco1234]

@jsphstls Using the example above, is there any reason for the test to fail on firstWindow()?

Error: electronApplication.firstWindow: Target page, context or browser has been closed

[jsphstls]

@roco1234 It's been a while since I have worked on that project but IIRC from my earlier attempts either the application failed to launch or Playwright failed to hook into the electron app. Using executablePath: await downloadAndUnzipVSCode() from the example above should prevent this.

The vscode repo uses Playwright to test vscode so you may also find insight there.

[mxschmitt]

Note: We are also using Playwright for testing our Playwright for VSCode extension: microsoft/playwright-vscode#353

[roco1234]

@mxschmitt I cloned the playwright-vscode repo and the issue persists on:
vscode: 1.85.1
mac: sonoma 14.2.1

[roco1234]

Update: I can run the Playwright tests on VSCode Insiders. See the vscode testing limitation: https://code.visualstudio.com/api/working-with-extensions/testing-extension#using-insiders-version-for-extension-development

[roco1234]

@mxschmitt is there a way to get coverage metrics on these tests? The vscode-js-debug extension has a coverage addon but can't see anything for the playwright extension