Repository navigation
E2E Test Reference
Living reference documenting every automated Playwright end-to-end test in e2e/: what it does, what it actually asserts (real backend effect vs. UI state), its preconditions, and which VMs it's meant to run against. Update this file in the same PR that adds, changes, or removes a spec.
For the backlog of tests still to be written, see E2E Test Inventory. For how to run the suite and manage VMs, see VM Testing and docs/testing.md.
| File | Purpose |
|---|---|
base.ts |
baseData(page) — logs in, dismisses the podman-only startup prompt if present, imports/scans the pre-staged compose directory. Retries the whole open→fill→submit sequence up to 5 times because rootless Podman's local component state gets reset for a few seconds after a runtime switch (see "Known flaky behavior" below). dismissStartupPodmanPrompt(page) — standalone, for specs that don't need a scan. FIXTURE_STACKS — names of every pre-staged fixture stack (kept in sync with scripts/test-vm.config.sh). |
stacks.ts |
Composable stack actions: downedCard, stackRow, upStack, downStack, openYamlEditor (uses force: true on its click — see "Known flaky behavior"), yamlEditorContent, ensureDown (force a stack down first, self-healing against a previous run's leaked state), withRunningStack (runs a callback against a temporarily-up stack, always brings it back down in a finally, even on failure; calls ensureDown first). |
runtime.ts |
`switchRuntime(page, 'docker' |
admin.ts |
loginWithAdminAccess(page, pluginName?) — logs in and switches Cockpit to real Administrative access before navigating to the plugin. Use instead of the shared pluginPage fixture whenever a spec needs genuine superuser escalation (e.g. rootful Podman with no rootless socket available) — pluginPage navigates straight to the plugin's own iframe URL and never sees the outer Cockpit shell, so it can never reach the "Limited access" control at all. In these test VMs (passwordless sudo) clicking it grants admin access instantly, no password prompt. |
vm.ts |
sshExec(projectName, command) — runs a command on the test VM over SSH, out-of-band from the app under test (e.g. restarting a container from "another terminal" to produce real events the UI can't trigger itself, like events.spec.ts). Uses execFile with an argv array, not an interpolated shell string — $(...) command substitution inside a double-quoted shell string still runs on the local host, not the remote VM, which silently ran every restart against the host's own (usually absent) podman/docker for an entire debugging session before this was caught. Maps project name → SSH port via the same ALL_VMS/SSH_BASE scheme as scripts/test-vm.config.sh. |
Design principle: every test here asserts a real effect (file content, container/volume state, a status attribute) rather than only "is this element visible" — see the discussion in issue #227.
-
Canonical — runs on
arch-podman,arch-docker,arch-both(one VM per engine-mode); most feature tests only need this. -
Full matrix — meant to also run across
debian-*/fedora-*because it's specifically about rootless/rootful or engine differences. -
Podman-only — self-skips via
test.skip()unless the Playwright project name includespodman.
| Test | Asserts | VM scope |
|---|---|---|
reaches the plugin page |
URL matches the plugin route and #root renders. |
Canonical |
shows the Compose Stacks heading |
Main heading renders after dismissing the podman-only startup prompt if present. | Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Compose Stacks heading is visible |
Heading renders (exact: true — an empty-state heading "No compose stacks are running" would otherwise substring-match "Compose Stacks"). |
Canonical |
scan finds pre-staged stacks |
Scanning /home/test/testcompose surfaces the gotify fixture in the downed-stacks list. |
Canonical |
each found stack has an Edit compose file button |
Each downed stack card exposes an "Edit compose file" action. | Canonical |
| Test | Asserts | VM scope |
|---|---|---|
YAML editor modal opens |
Dialog renders for gotify. |
Canonical |
editor contains YAML content |
CodeMirror content actually contains gotify (real file content, not a placeholder). |
Canonical |
Save button is present |
Save control renders once Edit is clicked. | Canonical |
Cancel exits edit mode without closing the modal |
Cancel reverts to read-only (Save disappears) without closing the dialog. | Canonical |
closing the modal returns to the main view |
Close hides the dialog and returns to the main heading. | Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Up starts a downed stack, Down removes it again |
Clicking Up → confirming the recreate-warning dialog → closing the progress modal results in a real `data-status="running" | "partial"row (not just a closed dialog); Down then removes that row entirely.afterEachforce-downsgotify` even if the test throws, so a failure can't leak a running container into the next test/run. |
| Test | Asserts | VM scope |
|---|---|---|
Create Stack suggests the home directory when rootless and no stacks exist yet |
The Create dialog's directory field defaults to /home/test/compose under rootless Podman. Self-skips on Docker VMs (they run rootful here). |
Podman-only |
Regression coverage for issue #242 — a rootful
Compose stack (sudo podman compose up, no rootless socket present at all) was invisible in the
dashboard because discovery ran unprivileged with no way to reach the root-owned socket, and even
once that was fixed, a second bug (src/api/containers.ts had no escalation at all) meant the
Stack Info modal's container list stayed empty even when the stack card correctly showed "running".
Uses loginWithAdminAccess (not pluginPage) since this needs real Cockpit Administrative access,
not just Podman's own passwordless sudo.
| Test | Asserts | VM scope |
|---|---|---|
discovers a rootfully-started stack, shows it running, and loads real container info |
The gotify stack (started outside the plugin via sudo podman compose up -d during VM provisioning) appears with data-status of running/partial, and opening its Stack Info modal shows real container data (.sim-no-containers absent) with a visible "running" state — not just a correct stack-level badge with nothing underneath. |
fedora-podman-rootful only (self-skips elsewhere — every other VM has a rootless socket that would mask this bug) |
| Test | Asserts | VM scope |
|---|---|---|
Logs modal streams, filters by service, searches, pauses, clears, and refreshes |
Real log lines from the worker service (multi fixture, emits worker-tick every 3s) stream in; filtering to one service still shows its lines; searching for a non-matching string hides all lines; Pause/Resume toggles the button label; Clear empties the viewer and new lines repopulate it; Refresh doesn't error. Brings multi up via withRunningStack and back down afterward. |
Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Shell opens a real terminal in the container and shows real command output |
Typing echo e2e-exec-marker-12345 into the xterm session and pressing Enter produces that exact marker in the terminal output — proves a command actually executed inside the worker container, not just that a terminal widget rendered. Requires an explicit terminal.click() before typing: xterm.js only routes keyboard input to itself once it has DOM focus, and clicking "Open shell" leaves focus on that button, not the newly mounted terminal — without the click, keystrokes go nowhere and the assertion fails looking like the terminal itself vanished. |
Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Scaling a service up actually creates the extra replicas |
Scaling multi's worker service from 1→3 via the Scale dialog results in a real "×3" replica badge in the expanded container list — not just that the dialog closed. worker has no host port bindings, so this avoids the port-conflict path. |
Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Prune removes real stopped containers, not just closes the dialog |
Stopping volumes-test and pruning with the Containers checkbox actually removes the named, stopped volumes-test-db-1 container — verified by name in the preview, then confirmed the stack drops out of the running list entirely (zero containers left). |
Canonical |
Volume pruning specifically (testing guide §6.16.7, "dangling named volume") looks unreachable through the current UI — see the file's own header comment for the full writeup. Docker only considers a volume "dangling" once zero containers (running or stopped) reference it (docker volume ls --filter dangling=true, src/api/stacks.ts listDanglingVolumes()), but the per-stack Prune action only exists on a row in the running stacks list — and that row disappears (moves to the downed section, which has no Prune action at all) the instant the stack's container count hits zero. Verified live: pruning volumes-test's stopped containers via this same Prune modal made the row vanish immediately, before a second Prune pass targeting the now-truly-dangling pgdata volume could ever be reached. By the time a volume qualifies as prunable, the UI no longer offers a way to prune it for that stack. This is flagged as a product question worth raising, not something fixed in the test.
| Test | Asserts | VM scope |
|---|---|---|
Create Stack (manual method) actually creates a compose file on disk |
Creating a stack via the Manual method (pre-filled stub YAML, no template/git dependency) results in a directory a fresh rescan can actually find on disk. Deletes the throwaway stack afterward so repeat runs stay clean. | Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Events modal streams real container lifecycle events, not just a static table |
EventsModal auto-starts streaming on mount (useEffect(() => start(), [])) — there is no manual "Stream events" click needed in the normal open flow, only visible again after Stop. Restarting gotify out-of-band over SSH (helpers/vm.ts sshExec, since the row underneath the modal's backdrop can't be clicked) produces a real "start" action row in the table, not a fabricated one. Stop/Clear empties it back out. |
Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Run command executes a real one-off command and streams its actual output |
Running echo <marker> against multi's worker service in the default "args" mode produces the real marker in the output and a "Command complete" status. |
Canonical |
Run command with --entrypoint override replaces the entrypoint instead of appending arguments |
With "Override entrypoint" checked, /bin/echo <marker> runs as the container's entrypoint (not appended as args to the default one) and the marker appears in the output. |
Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Pull images warns about the unpinned tag, then actually re-pulls the image |
gotify's implicit :latest tag (image: gotify/server, no explicit tag) triggers the unpinned-image warning in the confirm dialog; confirming actually re-pulls (real progress output, "Pull complete" status), not just closes the dialog. |
Canonical |
Run in Background sends the pull to the background task queue instead of blocking the modal |
Clicking "Run in Background" closes the modal immediately and the task genuinely appears (then completes) in the Background Tasks panel. | Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Stack Info shows real services, images, volumes, and networks — not empty placeholders |
Real service names/status, real image repos, the real named volume (volumes-test_pgdata), and the real default network (volumes-test_default) all appear — none of the "No X found" empty states. Found and fixed a real bug: listNetworkConnectedProjects broke the whole Networks section on Podman 6.0.1 (see src/api/stacks/query.ts and the Notes section of E2E Test Inventory). Closes the modal at the end — otherwise withRunningStack's teardown Down click lands on a row still covered by the modal's backdrop and hangs. |
Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Env file editor reads real values, edits and saves them, and the change persists to disk |
Real .env content on open (not a blank form); editing DEBUG and saving actually persists to disk, confirmed by reopening. Row lookups use each row's live .inputValue() (rowByKey helper) rather than a CSS input[value="..."] attribute selector — React sets a controlled input's value via the DOM property, not the HTML attribute, so attribute selectors only ever match a row's very first render. |
Canonical |
Env file editor warns on a duplicate key added via Table mode instead of silently saving it |
Adding a second APP_ENV row via Table mode's "Add variable" triggers the "Save with issues?" duplicate-key warning; canceling leaves the file untouched. Found a real doc/behavior gap: docs/testing.md §6.15 describes this warning firing for a duplicate added "in raw mode" too, but EnvModal.tsx only computes hasDuplicates via EnvTable's callback, which never runs while the Raw editor is mounted — a raw-mode-only duplicate silently saves with no warning. |
Canonical |
| Test | Asserts | VM scope |
|---|---|---|
Bulk Up on selected downed stacks actually starts every selected stack, not just the first |
Selecting gotify + env-test and confirming a bulk Up (which runs as background tasks per Bulk-Actions.md) results in both stacks actually running, not just the confirm dialog closing. The first row's checkbox needs { force: true } — per Bulk-Actions.md it's CSS-hidden until hover/focus or at least one stack is already selected. |
Canonical |
Select all toggles every visible downed stack and shows an indeterminate state for a partial selection |
A partial selection leaves "Select all" unchecked (not indeterminate-rendered-as-checked); toggling it selects/deselects every downed stack's real checkbox state. | Canonical |
| Test | Asserts | VM scope |
|---|---|---|
An external-bound port badge is clickable and calls window.open with the real host:port URL |
gotify's 8080:80 (bound to 0.0.0.0, classified "external") is clickable and calls window.open() with a URL containing the real host port. Captures the window.open call directly (monkey-patched before the click) rather than letting a real popup navigate — the container port isn't necessarily reachable from inside the guest VM's own browser, and a failed navigation lands on a chrome-error:// page before Playwright can observe the originally-requested URL. Found a real doc/behavior mismatch: docs/wiki/Stacks-Dashboard.md describes port clicks going through an external-link confirmation modal first, but the real code (StatsCell.tsx/PrettyCard.tsx/UnixRow.tsx) calls window.open() directly — that modal only exists for ContainerTable.tsx's per-service changelog link. |
Canonical |
Deliberately-hostile input — the point is "does the app fail safely," not "does it render."
| Test | Asserts | VM scope |
|---|---|---|
Saving malformed YAML warns instead of silently corrupting the compose file |
Replacing gotify's compose file with invalid YAML and clicking Save surfaces a "Save with issues?" confirm dialog (per YamlModal.tsx handleSave()) rather than saving silently. Canceling out of that dialog leaves the on-disk file untouched — reopening the editor shows the original content, not the malformed text. |
Canonical |
Scanning a directory that does not exist fails gracefully instead of crashing |
Scanning a nonexistent path produces a visible "Nothing found"/"Scan failed" state and the rest of the UI (main heading) stays usable — no hang, no dead page. | Canonical |
Scan-depth stepper clamps at its documented bounds (1-5) |
Clicking the decrease button 8 times in a row still leaves the depth at 1 (disabled, doesn't go to 0 or negative); clicking increase 10 times in a row leaves it at 5 (disabled, doesn't run away past its documented max). |
Canonical |
Everything above was written by reading the source (src/components/, src/i18n/locales/en.json, src/api/) for exact selectors/behavior, then actually run against live QEMU VMs repeatedly — not just typechecked. Every failure hit along the way was individually root-caused (see git history / PR discussion for the blow-by-blow); the fixes are already folded into the specs and helpers described above.
-
Confirmed passing, repeatedly, on
arch-docker:login,stacks,yaml-editor,stack-lifecycle,create-stack,logs(needsVM_MEM=2048+, see below),exec,scale,prune, and two ofadversarial's three tests (nonexistent-directory, scan-depth bounds). -
Confirmed passing on
arch-podman:rootless-compose-root(podman-only, self-skips elsewhere), plus the general suite after fixing the podman-startup-prompt and scan-panel-reset issues described below. -
One test with unresolved intermittent flakiness:
adversarial.spec.ts's "Saving malformed YAML..." case. Its underlying behavior has been directly verified correct multiple times over via screenshot (the "Save with issues?" dialog shows the right content; canceling genuinely leaves the file untouched) — the remaining failures are aTimeoutErroron a single click (openYamlEditor's "Edit compose file" button) that happens under sustained host load. In the session this was diagnosed in, the host (not just the test VM) was down to ~1.4GB free RAM with swap active and a load average of ~4, concurrently running the user's own interactive desktop (VSCode, browser tabs) alongside the QEMU/Chromium/ffmpeg test workload —force: trueon the click made no difference, which rules out a CSS-transition/stability-check issue and points at genuine host-level contention instead. Re-run this specific test on a quiet host before trusting its result; if it still fails there, treat it as a real regression.
-
VM memory pressure on Postgres/exec-heavy specs:
volumes-test(used byprune.spec.ts) runs Postgres, andexec.spec.tsdrives a real PTY/websocket session — both meaningfully heavier than the rest of the suite. At theVM_MEM=1024default,free -hinside the guest showed it actively swapping (~950 MB total RAM available to the VM), producing exactly the symptoms of a hung app (dialogs timing out, a session silently failing to open). Not parallel-worker contention — identical failures reproduced with--workers=1, fully serial. Fix:VM_MEM=2048+ for VMs running these specs (logs.spec.tswent from consistently failing to consistently passing on the same VM after only a memory bump, no code change). See VM Testing. -
Host-level contention beyond the VM: distinct from the above — if the host machine itself is under load (swapping, high load average, especially from concurrent interactive use), individual Playwright actions can time out waiting for elements that are demonstrably present and correct (confirmed via screenshot).
force: truedoes not help here since the bottleneck isn't an actionability/stability check, it's the browser/renderer process not getting scheduled promptly. There is no good in-test fix for this beyond generous timeouts; if you hit it, checkfree -handuptimeon the host before assuming a code regression. -
Rootless Podman panel resets: for a few seconds after switching to Podman (or on podman-only VMs where the app auto-switches on load), something keeps resetting
DownedStacksSection's local state — an opened import panel, typed directory, or even an in-flight scan can get wiped mid-flow.baseData()retries the entire open→fill→submit sequence (not just individual clicks) to absorb this. If you seeTimeoutErrorwaiting for.dss-stack-namespecifically on a podman VM, this is the likely cause — check the screenshot before assuming a real regression. -
A previous run's leaked state cascades into the next: if a hard test-timeout kills the page before
afterEach/finallycleanup runs, a stack (gotify,multi,volumes-test) can be left running, which then makes the next run'supStack()fail immediately (it expects to find the stack in the downed list) — a single real failure can look like several in a row.ensureDown()(inhelpers/stacks.ts, called bywithRunningStackand at the start ofstack-lifecycle/prune) self-heals this by forcing the stack down before assuming its starting state. -
Element detached / retry #1 passes: occasional Playwright-level auto-retry on clicks (
element was detached from the DOM, retrying) under heavy VM load. The suite'sretries: 1config absorbs this; if a test fails on both the initial run and the retry, treat it as real.
- Add the spec under
e2e/, reusinge2e/helpers/where possible — don't re-implement login/scan/stack-action flows inline. - Prefer
exact: trueongetByRolename matches for any short/common word ("Up", "Import", "Create", "Compose Stacks") — this codebase has repeatedly hit substring collisions (e.g. "Up" matching "Backup", "Create" matching "Create stack"). Assume collision until proven otherwise. - If the test mutates state (creates/starts/deletes a stack), guarantee cleanup with
try/finallyor atest.afterEach, not just a final step — a thrown assertion mid-test must not leak state into the next run. - Add an entry to this file and, if it fills in a 🚧 row, flip it to ✅ in E2E Test Inventory.
Cockpit Compose
Using the UI
- Stacks Dashboard
- Managing Stacks
- Bulk Actions
- Background Tasks
- Viewing Logs
- Editing Configuration
- Stack Info
- Pulling Images
- Events
- Process Viewer
- Shell Access
- Running Commands
- Prune Resources
- Scaling Services
Stack Management
Runtime Support
Help
Development