Skip to content

E2E Test Reference

github-actions[bot] edited this page Aug 9, 2026 · 3 revisions

E2E Test Reference

Living reference documenting every automated Playwright end-to-end test in e2e/: what it does, what it actually asserts (real backend effect vs. UI state), its preconditions, and which VMs it's meant to run against. Update this file in the same PR that adds, changes, or removes a spec.

For the backlog of tests still to be written, see E2E Test Inventory. For how to run the suite and manage VMs, see VM Testing and docs/testing.md.

Shared infrastructure (e2e/helpers/)

File Purpose
base.ts baseData(page) — logs in, dismisses the podman-only startup prompt if present, imports/scans the pre-staged compose directory. Retries the whole open→fill→submit sequence up to 5 times because rootless Podman's local component state gets reset for a few seconds after a runtime switch (see "Known flaky behavior" below). dismissStartupPodmanPrompt(page) — standalone, for specs that don't need a scan. FIXTURE_STACKS — names of every pre-staged fixture stack (kept in sync with scripts/test-vm.config.sh).
stacks.ts Composable stack actions: downedCard, stackRow, upStack, downStack, openYamlEditor (uses force: true on its click — see "Known flaky behavior"), yamlEditorContent, ensureDown (force a stack down first, self-healing against a previous run's leaked state), withRunningStack (runs a callback against a temporarily-up stack, always brings it back down in a finally, even on failure; calls ensureDown first).
runtime.ts `switchRuntime(page, 'docker'
admin.ts loginWithAdminAccess(page, pluginName?) — logs in and switches Cockpit to real Administrative access before navigating to the plugin. Use instead of the shared pluginPage fixture whenever a spec needs genuine superuser escalation (e.g. rootful Podman with no rootless socket available) — pluginPage navigates straight to the plugin's own iframe URL and never sees the outer Cockpit shell, so it can never reach the "Limited access" control at all. In these test VMs (passwordless sudo) clicking it grants admin access instantly, no password prompt.
vm.ts sshExec(projectName, command) — runs a command on the test VM over SSH, out-of-band from the app under test (e.g. restarting a container from "another terminal" to produce real events the UI can't trigger itself, like events.spec.ts). Uses execFile with an argv array, not an interpolated shell string — $(...) command substitution inside a double-quoted shell string still runs on the local host, not the remote VM, which silently ran every restart against the host's own (usually absent) podman/docker for an entire debugging session before this was caught. Maps project name → SSH port via the same ALL_VMS/SSH_BASE scheme as scripts/test-vm.config.sh.

Design principle: every test here asserts a real effect (file content, container/volume state, a status attribute) rather than only "is this element visible" — see the discussion in issue #227.

VM scope legend

  • Canonical — runs on arch-podman, arch-docker, arch-both (one VM per engine-mode); most feature tests only need this.
  • Full matrix — meant to also run across debian-*/fedora-* because it's specifically about rootless/rootful or engine differences.
  • Podman-only — self-skips via test.skip() unless the Playwright project name includes podman.

Tests

e2e/login.spec.ts

Test Asserts VM scope
reaches the plugin page URL matches the plugin route and #root renders. Canonical
shows the Compose Stacks heading Main heading renders after dismissing the podman-only startup prompt if present. Canonical

e2e/stacks.spec.ts

Test Asserts VM scope
Compose Stacks heading is visible Heading renders (exact: true — an empty-state heading "No compose stacks are running" would otherwise substring-match "Compose Stacks"). Canonical
scan finds pre-staged stacks Scanning /home/test/testcompose surfaces the gotify fixture in the downed-stacks list. Canonical
each found stack has an Edit compose file button Each downed stack card exposes an "Edit compose file" action. Canonical

e2e/yaml-editor.spec.ts

Test Asserts VM scope
YAML editor modal opens Dialog renders for gotify. Canonical
editor contains YAML content CodeMirror content actually contains gotify (real file content, not a placeholder). Canonical
Save button is present Save control renders once Edit is clicked. Canonical
Cancel exits edit mode without closing the modal Cancel reverts to read-only (Save disappears) without closing the dialog. Canonical
closing the modal returns to the main view Close hides the dialog and returns to the main heading. Canonical

e2e/stack-lifecycle.spec.ts

Test Asserts VM scope
Up starts a downed stack, Down removes it again Clicking Up → confirming the recreate-warning dialog → closing the progress modal results in a real `data-status="running" "partial"row (not just a closed dialog); Down then removes that row entirely.afterEachforce-downsgotify` even if the test throws, so a failure can't leak a running container into the next test/run.

e2e/rootless-compose-root.spec.ts

Test Asserts VM scope
Create Stack suggests the home directory when rootless and no stacks exist yet The Create dialog's directory field defaults to /home/test/compose under rootless Podman. Self-skips on Docker VMs (they run rootful here). Podman-only

e2e/runtime-rootless.spec.ts

Regression coverage for issue #242 — a rootful Compose stack (sudo podman compose up, no rootless socket present at all) was invisible in the dashboard because discovery ran unprivileged with no way to reach the root-owned socket, and even once that was fixed, a second bug (src/api/containers.ts had no escalation at all) meant the Stack Info modal's container list stayed empty even when the stack card correctly showed "running". Uses loginWithAdminAccess (not pluginPage) since this needs real Cockpit Administrative access, not just Podman's own passwordless sudo.

Test Asserts VM scope
discovers a rootfully-started stack, shows it running, and loads real container info The gotify stack (started outside the plugin via sudo podman compose up -d during VM provisioning) appears with data-status of running/partial, and opening its Stack Info modal shows real container data (.sim-no-containers absent) with a visible "running" state — not just a correct stack-level badge with nothing underneath. fedora-podman-rootful only (self-skips elsewhere — every other VM has a rootless socket that would mask this bug)

e2e/logs.spec.ts

Test Asserts VM scope
Logs modal streams, filters by service, searches, pauses, clears, and refreshes Real log lines from the worker service (multi fixture, emits worker-tick every 3s) stream in; filtering to one service still shows its lines; searching for a non-matching string hides all lines; Pause/Resume toggles the button label; Clear empties the viewer and new lines repopulate it; Refresh doesn't error. Brings multi up via withRunningStack and back down afterward. Canonical

e2e/exec.spec.ts

Test Asserts VM scope
Shell opens a real terminal in the container and shows real command output Typing echo e2e-exec-marker-12345 into the xterm session and pressing Enter produces that exact marker in the terminal output — proves a command actually executed inside the worker container, not just that a terminal widget rendered. Requires an explicit terminal.click() before typing: xterm.js only routes keyboard input to itself once it has DOM focus, and clicking "Open shell" leaves focus on that button, not the newly mounted terminal — without the click, keystrokes go nowhere and the assertion fails looking like the terminal itself vanished. Canonical

e2e/scale.spec.ts

Test Asserts VM scope
Scaling a service up actually creates the extra replicas Scaling multi's worker service from 1→3 via the Scale dialog results in a real "×3" replica badge in the expanded container list — not just that the dialog closed. worker has no host port bindings, so this avoids the port-conflict path. Canonical

e2e/prune.spec.ts

Test Asserts VM scope
Prune removes real stopped containers, not just closes the dialog Stopping volumes-test and pruning with the Containers checkbox actually removes the named, stopped volumes-test-db-1 container — verified by name in the preview, then confirmed the stack drops out of the running list entirely (zero containers left). Canonical

Volume pruning specifically (testing guide §6.16.7, "dangling named volume") looks unreachable through the current UI — see the file's own header comment for the full writeup. Docker only considers a volume "dangling" once zero containers (running or stopped) reference it (docker volume ls --filter dangling=true, src/api/stacks.ts listDanglingVolumes()), but the per-stack Prune action only exists on a row in the running stacks list — and that row disappears (moves to the downed section, which has no Prune action at all) the instant the stack's container count hits zero. Verified live: pruning volumes-test's stopped containers via this same Prune modal made the row vanish immediately, before a second Prune pass targeting the now-truly-dangling pgdata volume could ever be reached. By the time a volume qualifies as prunable, the UI no longer offers a way to prune it for that stack. This is flagged as a product question worth raising, not something fixed in the test.

e2e/create-stack.spec.ts

Test Asserts VM scope
Create Stack (manual method) actually creates a compose file on disk Creating a stack via the Manual method (pre-filled stub YAML, no template/git dependency) results in a directory a fresh rescan can actually find on disk. Deletes the throwaway stack afterward so repeat runs stay clean. Canonical

e2e/events.spec.ts

Test Asserts VM scope
Events modal streams real container lifecycle events, not just a static table EventsModal auto-starts streaming on mount (useEffect(() => start(), [])) — there is no manual "Stream events" click needed in the normal open flow, only visible again after Stop. Restarting gotify out-of-band over SSH (helpers/vm.ts sshExec, since the row underneath the modal's backdrop can't be clicked) produces a real "start" action row in the table, not a fabricated one. Stop/Clear empties it back out. Canonical

e2e/run-command.spec.ts

Test Asserts VM scope
Run command executes a real one-off command and streams its actual output Running echo <marker> against multi's worker service in the default "args" mode produces the real marker in the output and a "Command complete" status. Canonical
Run command with --entrypoint override replaces the entrypoint instead of appending arguments With "Override entrypoint" checked, /bin/echo <marker> runs as the container's entrypoint (not appended as args to the default one) and the marker appears in the output. Canonical

e2e/pull-images.spec.ts

Test Asserts VM scope
Pull images warns about the unpinned tag, then actually re-pulls the image gotify's implicit :latest tag (image: gotify/server, no explicit tag) triggers the unpinned-image warning in the confirm dialog; confirming actually re-pulls (real progress output, "Pull complete" status), not just closes the dialog. Canonical
Run in Background sends the pull to the background task queue instead of blocking the modal Clicking "Run in Background" closes the modal immediately and the task genuinely appears (then completes) in the Background Tasks panel. Canonical

e2e/stack-info.spec.ts

Test Asserts VM scope
Stack Info shows real services, images, volumes, and networks — not empty placeholders Real service names/status, real image repos, the real named volume (volumes-test_pgdata), and the real default network (volumes-test_default) all appear — none of the "No X found" empty states. Found and fixed a real bug: listNetworkConnectedProjects broke the whole Networks section on Podman 6.0.1 (see src/api/stacks/query.ts and the Notes section of E2E Test Inventory). Closes the modal at the end — otherwise withRunningStack's teardown Down click lands on a row still covered by the modal's backdrop and hangs. Canonical

e2e/env-editor.spec.ts

Test Asserts VM scope
Env file editor reads real values, edits and saves them, and the change persists to disk Real .env content on open (not a blank form); editing DEBUG and saving actually persists to disk, confirmed by reopening. Row lookups use each row's live .inputValue() (rowByKey helper) rather than a CSS input[value="..."] attribute selector — React sets a controlled input's value via the DOM property, not the HTML attribute, so attribute selectors only ever match a row's very first render. Canonical
Env file editor warns on a duplicate key added via Table mode instead of silently saving it Adding a second APP_ENV row via Table mode's "Add variable" triggers the "Save with issues?" duplicate-key warning; canceling leaves the file untouched. Found a real doc/behavior gap: docs/testing.md §6.15 describes this warning firing for a duplicate added "in raw mode" too, but EnvModal.tsx only computes hasDuplicates via EnvTable's callback, which never runs while the Raw editor is mounted — a raw-mode-only duplicate silently saves with no warning. Canonical

e2e/downed-stacks-bulk.spec.ts

Test Asserts VM scope
Bulk Up on selected downed stacks actually starts every selected stack, not just the first Selecting gotify + env-test and confirming a bulk Up (which runs as background tasks per Bulk-Actions.md) results in both stacks actually running, not just the confirm dialog closing. The first row's checkbox needs { force: true } — per Bulk-Actions.md it's CSS-hidden until hover/focus or at least one stack is already selected. Canonical
Select all toggles every visible downed stack and shows an indeterminate state for a partial selection A partial selection leaves "Select all" unchecked (not indeterminate-rendered-as-checked); toggling it selects/deselects every downed stack's real checkbox state. Canonical

e2e/ports.spec.ts

Test Asserts VM scope
An external-bound port badge is clickable and calls window.open with the real host:port URL gotify's 8080:80 (bound to 0.0.0.0, classified "external") is clickable and calls window.open() with a URL containing the real host port. Captures the window.open call directly (monkey-patched before the click) rather than letting a real popup navigate — the container port isn't necessarily reachable from inside the guest VM's own browser, and a failed navigation lands on a chrome-error:// page before Playwright can observe the originally-requested URL. Found a real doc/behavior mismatch: docs/wiki/Stacks-Dashboard.md describes port clicks going through an external-link confirmation modal first, but the real code (StatsCell.tsx/PrettyCard.tsx/UnixRow.tsx) calls window.open() directly — that modal only exists for ContainerTable.tsx's per-service changelog link. Canonical

e2e/adversarial.spec.ts

Deliberately-hostile input — the point is "does the app fail safely," not "does it render."

Test Asserts VM scope
Saving malformed YAML warns instead of silently corrupting the compose file Replacing gotify's compose file with invalid YAML and clicking Save surfaces a "Save with issues?" confirm dialog (per YamlModal.tsx handleSave()) rather than saving silently. Canceling out of that dialog leaves the on-disk file untouched — reopening the editor shows the original content, not the malformed text. Canonical
Scanning a directory that does not exist fails gracefully instead of crashing Scanning a nonexistent path produces a visible "Nothing found"/"Scan failed" state and the rest of the UI (main heading) stays usable — no hang, no dead page. Canonical
Scan-depth stepper clamps at its documented bounds (1-5) Clicking the decrease button 8 times in a row still leaves the depth at 1 (disabled, doesn't go to 0 or negative); clicking increase 10 times in a row leaves it at 5 (disabled, doesn't run away past its documented max). Canonical

Verification status

Everything above was written by reading the source (src/components/, src/i18n/locales/en.json, src/api/) for exact selectors/behavior, then actually run against live QEMU VMs repeatedly — not just typechecked. Every failure hit along the way was individually root-caused (see git history / PR discussion for the blow-by-blow); the fixes are already folded into the specs and helpers described above.

  • Confirmed passing, repeatedly, on arch-docker: login, stacks, yaml-editor, stack-lifecycle, create-stack, logs (needs VM_MEM=2048+, see below), exec, scale, prune, and two of adversarial's three tests (nonexistent-directory, scan-depth bounds).
  • Confirmed passing on arch-podman: rootless-compose-root (podman-only, self-skips elsewhere), plus the general suite after fixing the podman-startup-prompt and scan-panel-reset issues described below.
  • One test with unresolved intermittent flakiness: adversarial.spec.ts's "Saving malformed YAML..." case. Its underlying behavior has been directly verified correct multiple times over via screenshot (the "Save with issues?" dialog shows the right content; canceling genuinely leaves the file untouched) — the remaining failures are a TimeoutError on a single click (openYamlEditor's "Edit compose file" button) that happens under sustained host load. In the session this was diagnosed in, the host (not just the test VM) was down to ~1.4GB free RAM with swap active and a load average of ~4, concurrently running the user's own interactive desktop (VSCode, browser tabs) alongside the QEMU/Chromium/ffmpeg test workload — force: true on the click made no difference, which rules out a CSS-transition/stability-check issue and points at genuine host-level contention instead. Re-run this specific test on a quiet host before trusting its result; if it still fails there, treat it as a real regression.

Known flaky behavior (not test bugs)

  • VM memory pressure on Postgres/exec-heavy specs: volumes-test (used by prune.spec.ts) runs Postgres, and exec.spec.ts drives a real PTY/websocket session — both meaningfully heavier than the rest of the suite. At the VM_MEM=1024 default, free -h inside the guest showed it actively swapping (~950 MB total RAM available to the VM), producing exactly the symptoms of a hung app (dialogs timing out, a session silently failing to open). Not parallel-worker contention — identical failures reproduced with --workers=1, fully serial. Fix: VM_MEM=2048+ for VMs running these specs (logs.spec.ts went from consistently failing to consistently passing on the same VM after only a memory bump, no code change). See VM Testing.
  • Host-level contention beyond the VM: distinct from the above — if the host machine itself is under load (swapping, high load average, especially from concurrent interactive use), individual Playwright actions can time out waiting for elements that are demonstrably present and correct (confirmed via screenshot). force: true does not help here since the bottleneck isn't an actionability/stability check, it's the browser/renderer process not getting scheduled promptly. There is no good in-test fix for this beyond generous timeouts; if you hit it, check free -h and uptime on the host before assuming a code regression.
  • Rootless Podman panel resets: for a few seconds after switching to Podman (or on podman-only VMs where the app auto-switches on load), something keeps resetting DownedStacksSection's local state — an opened import panel, typed directory, or even an in-flight scan can get wiped mid-flow. baseData() retries the entire open→fill→submit sequence (not just individual clicks) to absorb this. If you see TimeoutError waiting for .dss-stack-name specifically on a podman VM, this is the likely cause — check the screenshot before assuming a real regression.
  • A previous run's leaked state cascades into the next: if a hard test-timeout kills the page before afterEach/finally cleanup runs, a stack (gotify, multi, volumes-test) can be left running, which then makes the next run's upStack() fail immediately (it expects to find the stack in the downed list) — a single real failure can look like several in a row. ensureDown() (in helpers/stacks.ts, called by withRunningStack and at the start of stack-lifecycle/prune) self-heals this by forcing the stack down before assuming its starting state.
  • Element detached / retry #1 passes: occasional Playwright-level auto-retry on clicks (element was detached from the DOM, retrying) under heavy VM load. The suite's retries: 1 config absorbs this; if a test fails on both the initial run and the retry, treat it as real.

Adding a new test

  1. Add the spec under e2e/, reusing e2e/helpers/ where possible — don't re-implement login/scan/stack-action flows inline.
  2. Prefer exact: true on getByRole name matches for any short/common word ("Up", "Import", "Create", "Compose Stacks") — this codebase has repeatedly hit substring collisions (e.g. "Up" matching "Backup", "Create" matching "Create stack"). Assume collision until proven otherwise.
  3. If the test mutates state (creates/starts/deletes a stack), guarantee cleanup with try/finally or a test.afterEach, not just a final step — a thrown assertion mid-test must not leak state into the next run.
  4. Add an entry to this file and, if it fills in a 🚧 row, flip it to ✅ in E2E Test Inventory.

Clone this wiki locally