Skip to content

Development and Validation

Jump to bottom
issuelens[bot] edited this page Oct 12, 2026 · 3 revisions

Development and Validation

Java Pack: Java 27 UI validation

These are source-backed test assertions and CI configuration in microsoft/vscode-java-pack, not independently observed test passes, released Java support, or a language-server implementation change. Verified at full merged/source-tip commit eb203464c082d7aa98d15202236a8bdeb91b3b56, PR #1767. The existing Test Plan remains the manual feature checklist; Java Pack owns this automated-validation entry point.

Coverage and runtime boundaries

Five AutoTest plans share test-fixtures/java27. Both Maven and Gradle build files target release 27 with preview enabled; each plan selects its intended importer. Plans register the project runtime as JavaSE-27 without setting java.jdt.ls.java.home: the project toolchain is distinct from the language-server runtime. The fixture also includes JDT preview preferences, separate from Gradle compiler arguments. [1][2]

Plan in test-plans/ Configured checks
java-maven-java27.yaml Maven import, generated JDT source level 27, zero diagnostics, clean compilation and execution.
java-gradle-java27.yaml Gradle 9.8.1 import/toolchain, source level 27, zero diagnostics, clean compilation and execution; Maven importer disabled.
java27-api.yaml LazyConstant and Set.ofLazy fixture; completion for Set.ofL includes ofLazy; compile/run assertions.
java27-preview.yaml Maven preview enabled → disabled → enabled; diagnostics and compiler failure while disabled, then recovery.
java27-primitive-patterns.yaml switch, instanceof, and nested record patterns; intentional unresolved type, deletion/save back to zero errors, compile/run assertions.

Coverage sources: the five named plans [1] and fixture/build files [2].

verify.mjs runs from the VS Code terminal, requires JAVA27_HOME with Java specification version 27, and sets the child build/runtime environment. It checks actual clean-build exit status, class-file major version 71 (also preview minor 65535 for non-project scenarios), and exact application output before emitting a success marker absent from the sent command. Preview-disabled verification requires both compilation failure and the expected primitive-pattern preview-disabled diagnostic, not an arbitrary failure. [3]

The primitive-patterns plan alone loads the test-only java27-autotest-support extension. Its java27.autotest.insertUnresolvedTypeAfterClose command rejects dirty or mismatched text, waits for actual document close, performs a redhat.java project-settings round trip checking source level 27, writes one unresolved-type insertion, then reopens a new document lifecycle and checks text, another round trip, and the specific fresh diagnostic. It rejects duplicate-field errors and incremental insertion events; the Java 27 AutoTest channel reports REOPEN_INSERTION_CONFIRMED or REOPEN_FAILED. This covers full-file reloading, not continuous editor edits or a production JDT fix. The fixture guide explains the former disk-write/Revert race; external JDT processing guarantees are not established here. [1][4][6]

CI toolchain and reproduction

The E2E workflow runs on Windows, Linux, and macOS. For Java 27 plans it provisions JDK 27, sets JAVA27_HOME, and adjusts plan paths per platform. Gradle preparation verifies 9.8.1 and exports the resolved absolute JAVA27_CI_GRADLE. With GITHUB_ACTIONS=true or CI=true, the verifier requires that absolute executable for the primary Gradle build; it neither falls back to PATH nor retries a failed CI build with another Gradle. Local execution retains PATH-based Gradle; a diagnostic control build, when configured, cannot replace the original failure or emit its success marker. Gradle diagnostics use --stacktrace --info. [3][5]

For reproduction, the fixture guide describes Maven/JDK 27/Gradle 9.8.1 prerequisites, JAVA27_HOME, matching the plan's project-runtime path, and a platform-specific Java extension VSIX. CI manual inputs test_plan, pre_release, and vsix_urls allow a selected-plan/exact-release comparison; retain the failed run's extension release when isolating a toolchain problem. No particular external extension release's support or CI success is inferred from these instructions. [5][6]

E2E artifacts and failure diagnosis

Discovery resolves @vscjava/vscode-autotest@latest once, then matrix and analysis jobs install that exact resolved release; there is no permanent 0.7.34 pin. Discovery uses nonrecursive test-plans/*.yaml and excludes java-fresh-import, so test-plans/config/artifacts.yaml is configuration, not an executable plan. [5]

Each case writes test-results/<plan>/ with file logging and case analysis. Pre-run logs and redirected CLI output are staged separately in RUNNER_TEMP/autotest-<plan> because startup clears the output directory; JAVA27_DIAGNOSTICS_DIR points to its terminal subdirectory. The run uses --logs, case-local --log-output, --artifacts-config, and --analysis-mode case. A separate unconditional autotest collect step supplements artifacts, followed by unconditional upload under results-<plan>-<os> with 30-day retention. The original test exit code is preserved, and collection has its own status. Aggregate organization skips artifact roots lacking results.json with a warning: inspect the original uploaded artifact for setup failures. [5]

The aggregate analyze job's Analyze results step sets AUTOTEST_AGGREGATE_ANALYSIS_MAX_TOKENS="16000" for autotest analyze test-results --output test-results --analysis-mode case --report-only. This override is step-local, not applied to the matrix's per-case run step; PR #1764 changes no per-case analysis configuration. Verified in microsoft/vscode-java-pack, .github/workflows/e2e-autotest.yml, jobs.analyze, at full merged/source-tip commit 2f31798e4350d228aac40d2a237d98163df212b7, PR #1764; source identity microsoft/vscode-java-pack#1764:2f31798e4350d228aac40d2a237d98163df212b7. This records the workflow's configured aggregate budget, not verified CLI internals or proof that response truncation is eliminated.

The consumer artifact configuration declares these optional sources and destinations; the fixture guide documents the root-relative archive layout and artifacts/manifest.json. Missing optional sources are not evidence of a clean run. Consult collection status as well as test results. [6][7]

Diagnostic question Configured artifact destination
Which toolchain/command ran, and how did compilation or execution fail? diagnostics/ci/ (version probes, toolchain JSON, and terminal/ logs/exit metadata); diagnostics/workspace/.autotest/.
Did the IDE, extension host, or JDT LS fail? logs/ide/logs/ and logs/jdtls/User/workspaceStorage/, sourced from persistent .vscode-test/user-data.
What did the CLI or display process report? logs/console/console.log; Linux logs/display/xvfb.log.
Was there a native macOS failure? logs/crashes/, restricted by configuration to matching Node/Electron/Code/Java reports modified since run start.

Workspace, IDE, JDT LS, and CI diagnostic sources request evidence: tail; full diagnostic archives and bounded analysis evidence serve different purposes. Console, display, and native reports use the collect phase. Product-specific paths live in this repository's configuration; collector internals and package-release guarantees are outside this source verification. The webview-migration plan also enables local logging at ../test-results/java-webview-migration/logs. [6][7][8]

Sources for Java 27 validation

References [1]–[8] are in microsoft/vscode-java-pack at full source commit eb203464c082d7aa98d15202236a8bdeb91b3b56, PR #1767; source identity microsoft/vscode-java-pack#1767:eb203464c082d7aa98d15202236a8bdeb91b3b56. The complete 21-file PR inventory was inspected, PR identities rechecked after paging, and final file identities/content verified at this merged revision.

Java Pack: runtime unit checks and Gradle Java 21/25 validation

Verified at merged/source-tip commit a1fd732e13a2cf9b73c9762d3de3cb445acea545, PR #1768. These are configured checks, not independently observed passes or release claims. For the pack-owned runtime behavior under test, see Java Runtime Requirements.

Node test entry point

npm test runs node --test. Ordinary CI uses Node 22 on Ubuntu and runs npm ci, npm test, then npm run build. tests/java-runtime.test.js uses mocked VS Code/upstream APIs and rendered components to check tooling thresholds, no implicit activation or tooling discovery, explicit JDK validation, retained project inventory on errors, FAQ guidance, and startup scheduling. tests/gradle-toolchain.test.js exercises the verifier with mocked filesystem/subprocess behavior across Windows, Linux, and macOS, including paths with spaces, conflicting PATH tools, bindings, version/output/bytecode failures, diagnostics, and plan sequencing. These unit checks do not replace execution inside VS Code. [9]

Gradle import and build contract

The two plans use the self-contained test-fixtures/gradle-import fixture. Maven and wrapper import are disabled; BSP and generated project metadata are enabled. Explicit compatible selections do not test automatic tooling fallback or wrapper precedence. [10][11]

Plan Explicit project/importer toolchain Language-server setting
java-gradle.yaml JDK 21, Gradle 8.5, javaRelease=21 No explicit tooling-home override; separate from the Gradle/project JDK.
java-gradle-java25.yaml JDK 25, Gradle 9.8.1, javaRelease=25 Explicit java.jdt.ls.java.home for JDK 25.

Both assert Gradle build/run task entries, exact GRADLE_MODEL:<gradle>:<release>:<actual-java> output without Error getting build, BSP nature, classpath and source level. They insert and reload an incomplete dependency-backed StringUtils.cap probe, require capitalize completion, then delete and save the probe before terminal build/run and zero-error checks. A final log check requires the task model to remain healthy. [10]

build.gradle accepts only release 21 or 25 and rejects mismatched Gradle/JVM pairs during configuration. The verifier runs clean, classes, and run, checks actual Java/Gradle versions, exact application output, and non-preview class-file major versions 65/69 with minor 0 before emitting GRADLE_JAVA21_BUILD_PASSED or GRADLE_JAVA25_BUILD_PASSED. Locally, run node verify.mjs 21 8.5 or node verify.mjs 25 9.8.1 from the fixture with matching JAVA21_HOME/JAVA25_HOME and Gradle on PATH; the verifier sets child JAVA_HOME and PATH to the requested JDK. [11]

Shared CI toolchain and diagnostics

The E2E workflow provisions Gradle 8.5 for the Java 21 plan and 9.8.1 for Java 25/27, exporting resolved GRADLE_CI_GRADLE and GRADLE_CI_JAVA_HOME. For Java 21/25 it exports ORG_GRADLE_PROJECT_javaRelease before IDE startup so separate task-model requests share the importer's release property. Java 27 retains the version-verified JAVA27_CI_GRADLE contract documented above. [12]

For the Java 21/25 verifier, specifically GITHUB_ACTIONS=true requires absolute Gradle, staging and project-JDK paths plus a matching shared release property; it does not silently substitute PATH Gradle. Toolchain/version/build diagnostics go under AUTOTEST_STAGING_DIR/terminal, or the fixture's local .autotest directory when no staging directory is supplied. The workflow retains the existing staged artifact collection and upload path; inspect toolchain identity as well as IDE/task-model logs when diagnosing mismatches. [11][12]

Sources for runtime and Java 21/25 validation

References [9]–[12] are in microsoft/vscode-java-pack at full source commit a1fd732e13a2cf9b73c9762d3de3cb445acea545, PR #1768; source identity microsoft/vscode-java-pack#1768:a1fd732e13a2cf9b73c9762d3de3cb445acea545. The complete 30-file PR inventory was inspected, PR base/head/merge identities rechecked after paging, and final source content or matching full-file blob identities verified at this revision.

Clone this wiki locally