feat(e2e): run the Kubernetes e2e suite on cargo-nextest with machine- and human-readable reports

[jgarciao]

User Story

As an OpenShell contributor, I want the Kubernetes e2e suite to produce concise, consistent results with a machine-readable report and protection against hung tests, so that I can triage failures quickly — whether running locally or reviewing CI output — feed results into tooling/dashboards, and not have a single stuck test block an entire cluster-backed run.

Problem Statement

On main, all Rust e2e suites — Kubernetes, Docker, Podman, and VM — run via plain cargo test. The workspace unit tests already run on cargo-nextest in CI (.github/workflows/branch-checks.yml, --profile ci), but no e2e suite uses it. All of them share the same limitations:

• no machine-readable (JUnit/xUnit) report,
• no concise fail-focused output (they rely on -- --nocapture, interleaving passing and failing output), and
• no per-test timeout, so a hung e2e test runs until an external CI/job timeout kills the whole job.

This is an inconsistency between how e2e tests and the rest of the test base are executed and reported.

Scope: This request targets only the Kubernetes e2e suite (e2e:kubernetes and its variants). The same approach could later be applied to Docker, Podman, and VM suites, but those are out of scope here. Kubernetes is the natural first candidate because it is the longest, most resource-intensive, and hardest to debug remotely.

Impact / Why This Matters

Without this feature, contributors must:

• Scroll raw cargo test output to find which e2e test failed and why — slow and error-prone for a suite of 85+ tests running against a live cluster (~97s wall time).
• Hand-roll any structured output, because cargo test cannot emit JUnit/xUnit. There is no artifact for dashboards, test-reporting UIs, or historical trend tracking to ingest.
• Wait out hung tests. A single stalled e2e test (e.g. a sandbox that never becomes ready) holds a live cluster and the CI job until an unrelated outer timeout fires — often 30–60 min on hosted runners — wasting cluster time and lengthening feedback loops.

This matters because e2e runs are the slowest, most resource-intensive part of the test base and the hardest to debug remotely; they are exactly where concise reporting, ingestible artifacts, and hung-test protection pay off most. The rest of the project already gets these from nextest, so the e2e suite is the outlier.

Proposed Design

From the contributor's perspective, mise run e2e:kubernetes (and every variant) behaves the same to invoke, but:

1. Runs on cargo-nextest, giving per-test process isolation and a concise summary that surfaces slow and failing tests instead of a wall of output.
2. Writes a JUnit/xUnit report to a predictable, git-ignored location (results/e2e-kubernetes.xml) that CI systems and dashboards can ingest.
3. Writes a standalone HTML report next to it (results/e2e-kubernetes.html) — a single self-contained page with a pass/fail/skip summary and per-suite tables that inline failure messages — for fast local triage. Report generation is best-effort and never changes the pass/fail result of the run.
4. Terminates hung tests. A test is flagged slow after a threshold and hard-terminated after a longer bound tuned for live-cluster durations, so one stuck test no longer blocks the whole run.
5. Requires no extra developer setup — the test runner is provisioned automatically via mise (the project's standard task runner), so mise run e2e:kubernetes works without the Nix devShell or any manual install.

Runner and configuration choices (which test runner, exact report format, timeout values) are implementation details for the people building it; the observable outcomes below define success.

Acceptance Criteria

• mise run e2e:kubernetes and all e2e:kubernetes:* variants run the suite and report results with concise, fail-focused output.
• A JUnit/xUnit XML report is written to a predictable, git-ignored path after the run.
• A human-readable HTML report is written to the same location with the run's summary and per-test outcomes, including failure detail.
• Report generation is best-effort: if the report cannot be produced, the run still reports the correct pass/fail exit status.
• A hung/stalled test is automatically terminated rather than blocking the entire run.
• The task runs without requiring the Nix devShell (test runner available via standard project tooling).
• Generated report artifacts are git-ignored.
• Existing e2e behavior (feature selection, single-test scoping, credential-driver variants, cluster setup/teardown) is preserved.
• Reports (XML and HTML) are produced on both passing and failing runs.
• The workspace ci nextest profile is unaffected (no change to its timeout or report configuration).

Alternatives Considered

• Keep cargo test and bolt on a JUnit converter. cargo test's libtest output is not designed for structured export; parsing it is brittle, and it still provides no per-test timeout or isolation. Rejected — it would reimplement what the project's chosen runner already provides for unit tests.
• Reuse the shared ci nextest profile for e2e. The report path is resolved relative to the runner's per-profile store directory, whose depth differs between the standalone e2e crate and the workspace; a shared path plus e2e-appropriate (longer) timeouts would leak into or misdirect the workspace run. Rejected in favor of a dedicated e2e profile.
• HTML via a third-party package (Python junit2html, npm xunit-viewer). Both add a package/runtime dependency and (for the ephemeral-install path) a network fetch, and produce much larger output. Rejected in favor of an approach with no committed dependency and no network requirement, so it also works in restricted build environments; the human-readable report is a local convenience and stays out of the required CI path. The XSLT approach assumes xsltproc (libxslt) is available; it ships by default on macOS and most Linux distributions but may be absent in minimal container images. Since the HTML render is best-effort and never affects the exit code, this is acceptable.
• Do nothing. Leaves the e2e suite inconsistent with the rest of the project and without ingestible artifacts or hung-test protection. Rejected.

Agent Investigation

A reference implementation exists on the cargo-nextest-e2e-kubernetes branch and was validated end to end on a live OpenShift cluster:

• Converted the suite to cargo-nextest with a dedicated e2e-kubernetes profile; a full run reported 85 tests, 85 passed, 1 skipped (~97s wall time) with concise output.
• Confirmed the JUnit report is written to the git-ignored results/ directory and that the HTML render (via a committed XSLT stylesheet) produces a ~24 KB self-contained page with the summary and any failing test's message inlined.
• Confirmed report generation is best-effort (missing renderer only warns; exit code preserved) and that a per-test timeout is scoped to the e2e profile without affecting the workspace ci profile.

[jgarciao]

I have a branch implementing these changes here. If the feature is accepted I can send a PR. Thanks!

[krishicks]

I think since nextest is being adopted we should continue to expand its use.