Skip to content

ci(e2e-autotest): add Linux and macOS to the runner matrix#1621

Merged
wenytang-ms merged 2 commits into
microsoft:mainfrom
wenytang-ms:ci/e2e-multiplatform-matrix
May 20, 2026
Merged

ci(e2e-autotest): add Linux and macOS to the runner matrix#1621
wenytang-ms merged 2 commits into
microsoft:mainfrom
wenytang-ms:ci/e2e-multiplatform-matrix

Conversation

@wenytang-ms
Copy link
Copy Markdown
Contributor

@wenytang-ms wenytang-ms commented May 20, 2026

Add Linux and macOS to the e2e-autotest matrix so every PR / schedule run exercises every plan on Windows, Linux, and macOS. The autotest framework was already cross-platform; the workflow just hadn't wired the matrix axis through.

Why

The job is currently pinned to runs-on: windows-latest, even though:

  • the autotest CLI (@vscjava/vscode-autotest) supports Windows / Linux / macOS,
  • this workflow already maps runner.os to win32 / linux / darwin for VSIX resolution (see the $platformMap block).

Coverage on Linux and macOS catches a class of Java-extension regressions (file casing, classpath separator, terminal shell, JDK detection on non-Windows paths, AppKit / GTK keyboard focus differences) that Windows-only CI cannot see.

Changes

Area Before After
runs-on hard-coded windows-latest ${{ matrix.os }}
matrix plan only os: [windows-latest, ubuntu-latest, macos-latest] × plan
JDK 25 step Windows New-Item Junction only cross-platform pwsh: junction on Windows, ln -sfn to /opt/jdk-25 on Linux / macOS, plus an in-place rewrite of the Windows-style java.jdt.ls.java.home in the java25 plan yaml
Linux display n/a new Setup virtual display (Linux) step (xvfb + DISPLAY=:99), gated on runner.os == 'Linux'; macOS has a real display so no step is needed
Artifact name results-<plan> results-<plan>-<os> so per-OS results of the same plan don't collide
analyze aggregator flattened to plan name preserves the <plan>-<os> suffix so cross-platform results stay distinct

Validation plan

  1. Opening this PR triggers the windows-latest, ubuntu-latest, and macos-latest matrix legs for the 17 plans.
  2. If a per-OS leg surfaces a plan-level issue (e.g. a hard-coded Windows path inside a plan yaml), it's fixed in the plan in a follow-up — the framework-level cross-platform support is already in place.
  3. After the first green run, the existing daily schedule (cron: '0 5 * * 1-5') starts producing cross-platform coverage automatically — no further workflow change needed.

Notes

  • Run count grows from 17 to 51 per trigger (17 plans × 3 OSes). Wall-clock impact is minimal because the matrix is fully parallel; matrix concurrency on the repo's runner pool is the only limit. macOS runners are billed at a 10× multiplier, but the existing trigger gating (PR + weekday schedule, not push) keeps the total within budget.
  • --no-sandbox is already passed to the VS Code launch args by the autotest framework, which is required for Electron on GitHub-hosted Linux runners.
  • actions/setup-java@v4 already handles JDK install on all three OSes; only the JDK 25 path-fixup step needed platform branching.
  • PowerShell Core is preinstalled on macos-latest and ubuntu-latest, so the shell: pwsh steps run identically across the matrix.

The e2e-test job was hard-coded to `runs-on: windows-latest` even though
the autotest framework and the platformMap in this workflow were already
written to be OS-aware. This change wires the matrix axis through so we
exercise every plan on both Windows and Linux on every PR / schedule run.

Changes:
* matrix.os = [windows-latest, ubuntu-latest]; runs-on uses it
* JDK 25 step is now cross-platform pwsh (junction on Windows,
  symlink to /opt/jdk-25 on Linux, plus an in-place rewrite of the
  Windows-style java.jdt.ls.java.home in the java25 plan)
* New `Setup virtual display (Linux)` step installs xvfb and
  exports DISPLAY=:99 before the autotest CLI launches VS Code
  (the autotest framework does not spawn Xvfb itself)
* Artifact name suffix: results-<plan>-<os> so Windows and Linux
  results don't collide
* analyze.organize-results: preserve the <plan>-<os> suffix so the
  aggregate report keeps Windows and Linux runs distinct

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds `macos-latest` to the matrix.os list. The cross-platform pwsh
branch added in the previous commit already covers macOS:

* `\True` is false on macOS so the symlink path runs unchanged
* macOS supports `sudo ln -sfn` to /opt/jdk-25 (added `mkdir -p /opt`
  so the path exists before the link is created)
* macOS GitHub-hosted runners have a real display, so no xvfb step is
  needed (the xvfb step stays gated on runner.os == 'Linux')
* PowerShell Core is preinstalled on macos-latest runners
* @vscode/test-electron downloads the darwin VSIX automatically

Run count grows from 17 plans × 2 OSes to 17 × 3.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@wenytang-ms wenytang-ms changed the title ci(e2e-autotest): add Linux to the runner matrix ci(e2e-autotest): add Linux and macOS to the runner matrix May 20, 2026
@wenytang-ms wenytang-ms merged commit c429a05 into microsoft:main May 20, 2026
97 of 104 checks passed
@wenytang-ms wenytang-ms deleted the ci/e2e-multiplatform-matrix branch May 20, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants