Repository navigation
E2E Test Inventory
Tracking doc for issue #227.
Maps every manual feature scenario in the Testing Guide §6-7 to a
planned (or existing) automated spec in e2e/, grouped into implementation waves.
Status legend: ✅ done, confirmed passing on live VMs (repeatedly) ·
Every new test should assert a real effect, not just DOM visibility — e.g. after
an edit, re-read the compose file content; after Up/Down, confirm the stack's actual
status rather than only a badge class. See e2e/helpers/ for the shared helpers that
make this practical (baseData, stacks.ts, runtime.ts).
| Testing guide § | Scenario | Spec (planned) | Status |
|---|---|---|---|
| 6.1 | Stack listing & status badges (running/stopped/partial/paused) | e2e/stacks.spec.ts |
✅ (status transitions covered) |
| 6.2 | Import / discover downed stacks, depth setting, runtime-switch clears path |
e2e/stacks.spec.ts, e2e/runtime-toggle.spec.ts
|
✅ runtime-switch clears path covered, and depth setting's real effect on what's found (not just its clamping bounds, already covered via adversarial.spec.ts) |
| 6.3 | Start (Up) — simple | e2e/stack-lifecycle.spec.ts |
✅ |
| 6.4 | Start (Up) — with profiles | e2e/stack-lifecycle.spec.ts |
✅ |
| 6.5 | Stop / Restart | e2e/stack-lifecycle.spec.ts |
✅ |
| 6.6 | Down (remove containers) | e2e/stack-lifecycle.spec.ts |
✅ |
| 6.7 | Kill | e2e/stack-lifecycle.spec.ts |
✅ |
| 6.8 | Logs (filter, search, pause/continue, clear, refresh) | e2e/logs.spec.ts |
✅ |
| 6.9 | Events (stream, table rows, stop, clear) | e2e/events.spec.ts |
✅ (streaming start/stop + real restart events land in the table; note: EventsModal auto-starts on mount, no manual "Stream events" click needed) |
| 6.10 | Exec / shell into a running container | e2e/exec.spec.ts |
✅ |
| 6.11 | Run command (one-off, --rm) | e2e/run-command.spec.ts |
✅ (args mode + --entrypoint override mode) |
| 6.12 | Pull images (progress, unpinned warning, cancel) | e2e/pull-images.spec.ts |
✅ (unpinned warning + real re-pull + Run in Background) |
| 6.13 | Stack info (services/images/volumes/networks tabs, shared networks) | e2e/stack-info.spec.ts |
✅ (found & fixed a real bug — see Notes; shared-networks case covered by connecting a real container to another project's network via SSH) |
| 6.14 | Edit YAML (multi-file tabs, add/delete file, import, snapshot history/diff/restore) | e2e/yaml-editor.spec.ts |
✅ (add/delete file found & documented a real accessibility bug — see Notes and issue #277) |
| 6.15 | Edit env file (table/raw modes, duplicate-key warning, add file) | e2e/env-editor.spec.ts |
✅ (found a real doc/behavior mismatch — see Notes) |
| 6.16.1 | Prune — basic flow (containers; volumes appears unreachable via UI, see reference doc) | e2e/prune.spec.ts |
✅ image, shared-image and container prune tests all pass, Podman included since #274 was fixed (#308) — see Notes |
| 6.17 | Scale services (increase replicas, port-conflict warning) | e2e/scale.spec.ts |
✅ |
| 6.19 | Create stack (manual/template/git-URL methods, validation bypass) | e2e/create-stack.spec.ts |
✅ (all three methods + validation bypass covered; git-URL clones a real public repo — see Notes for the fixture choice and required VM package, and issue #277 for a related intermittent flake) |
| downed-stacks bulk actions (select-all + bulk Up, commit 85fe38d) | Bulk select/Up on downed stacks table | e2e/downed-stacks-bulk.spec.ts |
✅ |
| 6.22 | Ports — clickable link, localhost-only vs all-interfaces tooltip | e2e/ports.spec.ts |
✅ (external, localhost, and specific-bind cases all covered; found a doc/behavior mismatch — see Notes; the localhost/specific-bind fixture also surfaced and fixed real arch-podman VM infra bugs — see Notes) |
| adversarial | Malformed YAML, nonexistent scan directory, scan-depth bounds | e2e/adversarial.spec.ts |
✅ all 3 fixed and stable (3/3 × 2 full-file runs) — see Notes, these were real test bugs, not flakes |
Run these against the full matrix (batched per §"Managing resource usage" in VM Testing) since behavior depends on engine + rootless/rootful + distro.
| Testing guide § | Scenario | Spec (planned) | Status |
|---|---|---|---|
| 6.20, 6.21 | Footer + runtime toggle (Docker↔Podman switch, rootless badge, socket path, "not installed" revert) | e2e/runtime-toggle.spec.ts |
✅ (arch-both; "not installed" case breaks the real podman binary via SSH, always restored) |
| 7.1 | Docker rootless mode |
e2e/runtime-rootless.spec.ts (fedora-full — the one VM with a genuine per-user Docker socket) |
✅ Rootless badge shown, full Up/Down lifecycle works under rootless Docker |
| 7.2 | Podman via podman compose external provider |
(every spec that runs against arch-podman) |
✅ — on arch-podman, podman compose's "external compose provider" genuinely is /usr/bin/podman-compose (verified via SSH: podman compose version prints the "Executing external compose provider" banner naming that exact binary); every Wave 1/3/4 spec already run against this VM exercises this path, no dedicated spec needed |
| 7.3 | Podman via podman-compose (Python, no docker-compose) |
(every spec that runs against arch-podman) |
✅ — same finding as 7.2: arch-podman has no docker-compose at all and its podman-compose is confirmed genuine Python (/usr/bin/podman-compose is #!/usr/bin/python, reports podman-compose version 1.6.0), so 7.2 and 7.3 are literally the same code path on this VM, both already covered by the existing suite |
| 7.4 | Podman root (system socket), no rootless socket at all — issue #242 regression: discovery + Stack Info container list under real Cockpit Administrative access, and that the two agree on state at the same moment after a real transition (Troubleshooting.md's status-agreement regression) |
e2e/runtime-rootless.spec.ts (fedora-podman-rootful only) |
✅ |
| 7.5 | Neither runtime present — error state | e2e/runtime-unavailable.spec.ts |
✅ (arch-docker; hides both docker and the standalone docker-compose legacy binary via SSH — hiding docker alone isn't enough, detectComposeCommand() falls back to it) |
| Rootless/Rootful socket-mode toggle (added alongside #242's fix) — switching modes when both sockets are detected, footer badge, stale-list refresh on switch | e2e/socket-mode-toggle.spec.ts |
✅ (fedora-full only, via loginWithAdminAccess — Rootful stays disabled without real admin access) |
|
| superuser prompt | Compose file owned by another uid (composeFileSuperuser(), src/api/cockpit.ts:185-215) |
e2e/superuser-prompt.spec.ts |
✅ (fedora-full only — the one VM with a genuine rootless Docker socket; see spec header comment for why a stricter-permissions fixture was tried and reverted) |
Recheck button vs. a socket that exists but doesn't respond (checkSocketHealth(), docs/wiki/Troubleshooting.md) |
none | 🚧 not attempted — the only way to produce this state is mutating a shared VM's live systemd --user socket/service, which isn't reproducible and would need a full VM reset afterward; needs a dedicated VM image with a genuinely broken/stub socket (e.g. a Unix socket bound via socat/nc that accepts connections but never replies) baked into cloud-init instead of a live hack. Deliberately deferred rather than attempted this pass — checkSocketHealth()'s probe command (podman ps / docker version) has no visible client-side timeout in src/api/cockpit.ts, so a genuinely hung socket could make the health check (and this test) hang indefinitely; that needs verifying against a real hung socket in isolation before wiring up the full VM/spec, which is a larger, more novel piece of work than anything else in this pass. |
|
| distro quirks | Debian cgroupfs pause/unpause workaround, Fedora SELinux label=false, pasta/nftables networking |
covered implicitly by running Wave-1 lifecycle specs against the full matrix rather than a dedicated spec | ✅ partial — e2e/stacks.spec.ts + stack-lifecycle.spec.ts run clean on debian-podman (4/4 + 9/9), debian-docker (13/13), and debian-both (14/14, both fresh-start and rebuilt); fedora-podman 12/13 (1 known host-load flake, passed alone); fedora-docker 4/13 — the other 9 failures are not distro-quirk test gaps, they're issue #272 (a real app-level race, also reproduces on arch-docker, not Fedora-specific) / 🚧 fedora-both — every automated Up attempt stalls silently (confirm dialog and progress-modal clicks all report success, but zero dockerd activity ever happens and the row never leaves the undiscovered "Down" list), reproduces consistently across multiple clean VM rebuilds and both parallel and serial runs, but a manual Up through a real browser session on the same VM works fine — looks like an automation/timing-specific issue rather than a real app bug, not yet root-caused; see Notes |
| Testing guide § | Scenario | Spec (planned) | Status |
|---|---|---|---|
| 5.3 / 6.4 | Multi-service stack with profiles | covered by 6.4 above | ✅ |
| 5.4 / 6.14 | Multi-file compose (extra compose file) | covered by 6.14 above | ✅ |
| 6.16.2-6.16.6 | Prune edge cases (shared image across stacks still in use, exited one-shot container removal by name) |
e2e/prune.spec.ts (additional cases) |
✅ |
| 6.16.7 | Dangling named volume | e2e/prune.spec.ts |
❓ the earlier "unreachable via the current UI" finding turned out to rest on a false premise specific to Podman — see issue #274 and e2e/prune.spec.ts's header comment; revisit once #274 is fixed before deciding whether this is still a product question |
| 6.18 | Backup & restore (archive create, restore into new dir, restored stack starts) | e2e/backup-restore.spec.ts |
✅ (found & documented a UX asymmetry vs BackupModal — see Notes) |
| 6.14 (malformed) | Save rejects invalid YAML with error details |
e2e/yaml-editor.spec.ts (additional case) |
✅ (see Notes for a known flake under cumulative session load) |
| 6.23 | Clickable external links show a warning modal before navigating |
e2e/stacks.spec.ts (additional case) |
✅ — not in StackInfoModal (its port badges call window.open() directly, no confirmation, see #260/ports.spec.ts); the real warning-modal flow is ContainerTable's per-service image link inside an expanded row, covered here instead |
| healthcheck / restart-policy / named-network / crash-loop / long-logs fixture stacks (pre-staged, §5) | Status badge & info correctness for each fixture |
e2e/fixture-stacks.spec.ts (parametrized cases) |
✅ |
Discovered while cross-checking docs/wiki/*.md against this table for a coverage
push — these are real, documented features with zero automated coverage that were
never even tracked here (the table above only maps docs/testing.md §6-7's manual
scenarios).
| Feature | Doc | Spec (planned) | Status |
|---|---|---|---|
| Background Tasks (queue, states, Stop/Remove, real container-state completion, runtime-switch cancellation, log view) | Background-Tasks.md | e2e/background-tasks.spec.ts |
✅ (Pending→Running→Complete against real container state, Stop terminates the underlying process, Remove actually drops the task, runtime-switch cancellation race verified on arch-both, log-view-through-panel found a real z-index bug — see issue #283 and Notes) |
| Bulk Actions on running stacks (per-layout selection, select-all indeterminate, per-action confirm, runtime-switch clears selection) | Bulk-Actions.md | e2e/bulk-actions.spec.ts |
✅ (Power User checkbox, Minimal card-click, Pretty card-click, Unix bracket-toggle, runtime-switch-clears-selection all covered) |
Process Viewer / Top (docker compose top equivalent) |
Process-Viewer.md | e2e/process-viewer.spec.ts |
✅ |
| Keyboard shortcuts (U/D/L/E/I) | Stacks-Dashboard.md | fold into e2e/stacks.spec.ts
|
✅ |
| Layout selector (4 layouts) | Stacks-Dashboard.md | fold into e2e/stacks.spec.ts
|
✅ |
| Status filter chips + auto-refresh degrade/Retry | Stacks-Dashboard.md | fold into e2e/stacks.spec.ts
|
✅ (auto-refresh degrade/Retry verified against a real broken podman binary on the VM, not a mock — see Notes) |
Found via a final systematic audit (cross-referencing every src/components/*.tsx
against every e2e/*.spec.ts, and every documented sub-flow in docs/wiki/*.md
against actual test assertions, not just "a spec file with this feature's name
exists").
All ten are written as specs and pass on real VMs. Full-suite runs on 2026-10-03,
one VM at a time with VM_MEM=2048 and --workers=1:
-
fedora-full: 88 passed, 0 failed, 0 retries, 9 skipped (VM-specific skips plus thetest.fixmewaiting on #319) -
arch-both: 83 passed, 0 failed, 13 skipped; the one flaky test (scale port conflict, closing the dialog before Apply had settled) was fixed and then passed 3/3 with retries off
The first real runs needed a fix-up pass, and found four app bugs, all fixed: the backup
archive written under a different name than previewed, Restore ignoring its overwrite
confirmation (#320), the Scale confirm heading showing a raw {{name}} placeholder, and
Run in Background reporting Failed for a stack that was running (#319, fixed in #333
together with #272). The #319 spec is active again, joined by a regression test for #272
(closing the Up dialog with ✕ mid-run backgrounds the Up instead of discarding it). Both
passed 3/3 with retries off on fedora-full. No spec in the suite is test.fixme anymore.
Each asserts a real backend effect (engine state or files over SSH), in keeping with
the rest of the suite. SSH-side engine checks go through the new
engineCli(projectName) helper in e2e/helpers/vm.ts, which picks the CLI the app
itself ends up using on that VM (podman on podman-only VMs, sudo podman on
fedora-podman-rootful, docker elsewhere).
| Feature | Doc | Spec | Status |
|---|---|---|---|
Global image prune (GlobalPruneModal — the host-wide "Prune images" button) |
Prune-Resources.md |
e2e/prune.spec.ts — 2 tests: removes gotify/server once gotify is fully down (verified via images over SSH); the acknowledgement checkbox gates the Prune button |
✅ passing on fedora-full + arch-both
|
| Pause / Unpause | Managing-Stacks.md |
e2e/stack-lifecycle.spec.ts — every multi container reports paused, the menu flips to Unpause, then all report running
|
✅ passing on fedora-full + arch-both
|
| Scale's port-conflict warning | Scaling-Services.md |
e2e/scale.spec.ts — scale-test's web (8085:80): inline warning, confirm-step alert + icon naming only web, Apply not blocked, never >1 running web, port-less worker still scales to 2 |
✅ passing on fedora-full + arch-both — see correction 1
|
Command history (HistoryDatalist) |
Running-Commands.md, Shell-Access.md |
e2e/exec.spec.ts + e2e/run-command.spec.ts — history cleared, command used, modal reopened, <datalist> <option> present and wired via the input's list attribute |
✅ passing on fedora-full + arch-both
|
| YAML editor "Import" | Editing-Configuration.md |
e2e/yaml-editor.spec.ts — file staged over SSH, imported as a second tab, and an edit saved in that tab lands at the same path on disk |
✅ passing on fedora-full + arch-both
|
| Backup archive deletion | Backup-and-Restore.md |
e2e/backup-restore.spec.ts — 2 tests: both confirmations then archive gone from disk; cancelling confirmation 2 keeps it |
✅ passing on fedora-full + arch-both
|
| Restore's "Target already exists" gate | Backup-and-Restore.md |
e2e/backup-restore.spec.ts — Restore disabled until #rm-target-confirm is ticked, then the colliding docker-compose.yml is really replaced |
✅ passing on fedora-full + arch-both
|
| Restore's "Name conflict" | Backup-and-Restore.md |
e2e/backup-restore.spec.ts — with gotify existing, restoring its backup warns and pre-fills gotify-restored, which lands beside the original |
✅ passing on fedora-full + arch-both — see correction 2
|
| "Find best match" | Importing-Stacks.md |
e2e/find-best-match.spec.ts (new) — 3 tests: resolves the real compose root and a scan of it finds stacks; disabled with nothing running; same inference from Create Stack |
✅ passing on fedora-full + arch-both
|
| Selective recreation after edit + re-Up | Editing-Configuration.md |
e2e/yaml-editor.spec.ts — only cache gets a new container ID after its definition changes; web and worker keep theirs; new container carries the change |
✅ passing on fedora-full + arch-both — see correction 3
|
Run Command's --rm unchecked |
Running-Commands.md |
e2e/run-command.spec.ts — exactly one new exited container for multi afterward (its worker never exits, so only the one-off can be) |
✅ passing on fedora-full + arch-both
|
These were checked against the code rather than taken from the planned shapes, as this document asked:
-
Scale port conflict does not "still scale". The plan expected Compose to bind
only the first replica's port and carry on. The app's own copy
(
scale_modal.port_conflict_inline) says scaling past one replica will fail. The spec asserts what must hold either way — never two runningwebcontainers — and that Apply warns rather than blocks. -
"Name conflict" is not about colliding archive names. The plan guessed two
backups with the same auto-filled name. In
RestoreModal.tsxit means the archive's stack name matches an existing stack (case-insensitive), and the app suggests<name>-restored. - Selective recreation changes an env var, not the image tag. A tag bump would make the test depend on a registry pull. Adding an environment variable is an equally real change to the service definition, recreated the same way, with no network involved. The file is edited over SSH (CodeMirror auto-indent makes exact YAML insertion brittle); the Up that acts on it goes through the UI.
Both bugs that earlier specs had to work around are fixed, and the workarounds were replaced by real interactions, which now act as the regression tests:
| Bug | Issue | Fix | Regression test |
|---|---|---|---|
| Nested modals lose ARIA accessibility after typing | #277 | PatternFly 6.6.2 (#312) — its Modal hid every body child except its own backdrop on each render, so the outer modal hid the inner one |
getByRole restored in yaml-editor, env-editor; plus src/lib/nestedModalA11y.test.tsx against plain PatternFly (fails on 6.6.1) |
| Background Tasks drawer intercepts the log modal's Close | #283 | Drawer moved below the modal backdrop's z-index tier (#309) | Real .click() restored in background-tasks.spec.ts
|
-
The 5 "flaky" specs, actually debugged with a trace/screenshot pass — of
the 5 specs previously tagged as an unresolved host-contention flake class,
4 turned out to be real, deterministic bugs (in the tests, and in one case
the app), not timing flakes at all:
-
e2e/adversarial.spec.ts's malformed-YAML test usedpage.getByRole('dialog', { name: 'Confirm save' })— the modal's real accessible name is itsModalHeadertitle ("Save with issues?"), which takes ARIA precedence over thearia-labelprop. Fixed to filter by visible content instead. A second, separate bug in the same test: after canceling the confirm dialog, it calledopenYamlEditor()again to verify the on-disk content — but Cancel only exits edit mode, it doesn't close the modal, so the second open raced the first modal's own backdrop. Fixed to assert directly on the still-open modal instead of closing and reopening. -
e2e/adversarial.spec.ts's other two tests (nonexistent scan dir, scan-depth bounds) never calleddismissStartupPodmanPrompt()— on a podman-only VM the auto-suggest-Podman modal blocks the Import button they immediately try to click. Fixed by adding the missing call. -
e2e/backup-restore.spec.ts: a prior run's restored stack directory (e2e-restored-gotify) was never fully deleted —downStack()only runscompose down, and the app's own "Delete compose file" action only removes the.yml(not gotify's bind-mounteddata/subdirectory), so the directory survived either cleanup path and permanently disabledRestoreModal's Restore button on the next run (targetExistsstays true, requiring an overwrite-confirm checkbox neither run handled). Fixed afterEach/setup torm -rfthe directory via SSH — the only way to guarantee it's actually gone. Also now cleans up accumulated.bak.tar.gzarchives, which had grown to 8+ from repeated runs. -
e2e/prune.spec.ts'svolumes-testcase surfaced a real app bug: see issue #274 —podman container pruneis a silent no-op on containers that belong to a pod, which every podman-compose stack's containers are. The test's old assertion (row disappears from the running list) was based on a false assumption; the row actually stays "stopped" forever since the containers never get removed. This is what the §6.16.7 dangling-volume "unreachable" note above was built on — also revisit once #274 is fixed. Both affected tests now assert the correct behavior (real container removal, verified via Stack Info). They were markedtest.fixmeon Podman until #274 was fixed (#308), and now run there unchanged — they never needed a rewrite. The sibling image-prune tests in the same file are unaffected (different underlying command, no pod involvement). - The only genuine remaining flakiness:
e2e/backup-restore.spec.ts's final "Up" step occasionally needs more than 20s when run right after several other heavy tests in the same batch (passes reliably alone) — ordinary host-load timing, not a logic bug.
-
-
Real bugs found and fixed while writing Wave 1 specs this pass (proof this
approach works — writing the test against actual app behavior, not assumptions,
surfaces genuine issues):
-
listNetworkConnectedProjects(src/api/stacks/query.ts) used apodman ps --format {{index .Labels "..."}}Go template that errors on Podman 6.0.1 ("cannot index slice/array with type string"), breaking the entire Stack Info Networks section with a generic "Could not load networks" alert. Fixed to go through--format json+ JS-side parsing instead, matching every other podman fallback in that file. Covered by bothe2e/stack-info.spec.tsand a new unit test insrc/api/stacks/query.test.ts. -
docs/wiki/Stacks-Dashboard.mdanddocs/testing.md§6.22 describe port-badge clicks as going through an "external-link confirmation" — the real code (StatsCell.tsx/PrettyCard.tsx/UnixRow.tsx) callswindow.open()directly with no modal in between.e2e/ports.spec.tsasserts the real behavior; the docs need a fix or a product decision on which is intended. -
docs/testing.md§6.15 describes a duplicate-key warning triggering from edits made "in raw mode" —EnvModal.tsxonly computeshasDuplicatesviaEnvTable'sonDuplicatesChange, which never fires while the Raw (CodeMirror) editor is mounted, so a raw-mode-only duplicate silently saves with no warning.e2e/env-editor.spec.tsexercises the real (Table-mode) path where the check genuinely runs and documents the Raw-mode gap. -
e2e/yaml-editor.spec.ts's add/delete-file case (previously cut for "unreliable" Create-file clicks) turned out to have a real, reproducible cause: typing into the nested "Add compose file" modal's Filename input gets both that modal's and the outerYamlModal's backdrop stuck ataria-hidden="true"— a PatternFly focus-trap bug in modal-in-modal nesting. Functionally harmless to mouse/keyboard users (a plain click still creates the file for real) but breaks accessible-name resolution for everything inside both modals from that point on — a genuine a11y regression for screen reader users. Filed as issue #277. The spec worked around it with a CSS-based click until PatternFly 6.6.2 fixed it (#312); it now usesgetByRoleagain and still verifies the real effect (file genuinely created/removed on disk via SSH, not just tab count). - Create Stack's git-URL method needed a real
gitbinary on the VM — none of the scenarios installed one (fetchComposeFromGitshells out togit clone). Addedgitunconditionally toextra_packages()inscripts/test-vm.config.shfor every VM. The test clonescoollabsio/coolify(a widely-used, actively-maintained public repo with a realdocker-compose.ymlat its default branch's root) rather than a fixture hosted in this project, to keep it pointed at a genuine external git host. The same intermittent #277-class flake (stuckaria-hiddenafter filling a modal's TextInput) was also observed on this file's pre-existing manual-method and validation tests (not introduced by this change) — noted inline in the spec. - Ports' localhost/specific-bind cases (a new
port-bindsfixture:127.0.0.1:8100->80and127.0.0.2:8101->80, both loopback so no real non-loopback network config is needed) hit a cascade of real, pre-existingarch-podmanVM infra bugs on a fresh rebuild, unrelated to the app: (1) the harness'spackage_upgrade: falseplus Arch being rolling-release meant a freshly-installedpodmancould end up linked against a newerlibsubidABI than the base image'sshadowpackage already had on disk — fixed with apacman -Syu --noconfirmruncmd step for Arch; (2) rootless Podman's overlay storage doesn't work directly over the base image's btrfs root fs — fixed by addingfuse-overlayfsto every podman-scenario VM's packages; (3) upgrading the kernel via (1) without rebooting left netavark trying to create a bridge against stale, mismatched kernel modules ("Netlink error: Operation not supported") — fixed with arebootas the last Arch runcmd step, after every other setup command, so the new kernel is actually active before any test ever starts a container. All three fixes are inscripts/test-vm.config.sh.
-
-
arch-bothfindings (Wave 2, first pass):- This VM's Docker only exposes a Rootful socket — no rootless Docker
at all.
bulk-actions.spec.ts's four pre-existing Docker-default tests (Bulk Down,Minimal layout,Unix layout,Select all) need real Cockpit Administrative access to bring a stack up under a rootful socket, which the sharedpluginPagefixture doesn't provide (onlyloginWithAdminAccess, used byruntime-rootless.spec.ts's rootful spec, does). Scoped away fromarch-bothwithtest.skipin atest.describeblock rather than silently left failing — a real gap, not fixed in this pass, worth a dedicated admin-access variant. -
gotify's./databind mount on this shared VM had been left root-owned by an earlier Docker run (Docker's daemon runs as root) — rootless Podman then can't write its sqlite db ("attempt to write a readonly database"), so the container exits immediately with the app correctly reporting it as stopped. Not an app bug — fixed the ownership on the VM directly (sudo chown -R test:test) and switched the Podman-side runtime-switch tests toenv-test(no bind mount) to avoid depending on that VM-specific fixup persisting. - Confirmed while chasing the above:
podman composeonarch-bothdelegates to/usr/lib/docker/cli-plugins/docker-compose(Docker's own compose-v2 plugin, reporting itself as "Podman Compose: 5.3.1") rather than thepodman-compose(Python) providerarch-podmanuses — a legitimate difference in what "the podman external provider" means per-distro/install, not a misconfiguration. - Attempted a dedicated admin-access variant of the four skipped tests
(
loginWithAdminAccess+ Bulk Down under Rootful Docker): the underlying escalation capability is already proven working elsewhere (e2e/superuser-prompt.spec.tsonfedora-full), but onarch-bothspecifically the very first "Up" click afterloginWithAdminAccessonly ever registered as a hover (the row's tooltip appeared, no confirm dialog) — combined with a "Cockpit admin mode mismatch" banner (DownedStacksSection.tsx's intentional warning for scanning inside the admin user's own home directory, not a bug) muddying the picture. Left unfixed rather than shipping a test that doesn't reliably pass — the gap documented above stands; a real follow-up should probably scan a directory outside the admin user's home to sidestep the mismatch banner entirely before re-attempting.
- This VM's Docker only exposes a Rootful socket — no rootless Docker
at all.
-
Issue #272 (found running the distro-quirks matrix on
fedora-docker): clicking Close on the Up progress modal the instant it becomes clickable can silently discard a still-finishingdocker compose up— the modal shows the full success log through "Started" but the container never actually exists (docker ps -aempty, zerodocker eventsactivity). Rigorously reproduced with matched-timing before/after comparisons (3 runs each) onfedora-dockerandarch-docker; does not reproduce ondebian-dockerorarch-podmandespitearch-docker/debian-dockerhaving identical Docker and Compose versions — ruled out cgroup driver and version as the differentiator. Root cause not confirmed (best working theory: acockpit.spawnchannel-close timing race, not something fixable from this repo alone) — filed with the full repro rather than guessing further. This is whyfedora-dockeronly passed 4/13 in the distro matrix above: the other 9 failures are all this same bug hittingupStack()'s Close click, not per-test gaps. -
Second pass on Wave 3/4 remainder findings:
-
e2e/yaml-editor.spec.ts's malformed-YAML test joins the same host-load flake class noted above: passes reliably in isolation but can time out waiting for the "Save with issues?" confirm dialog when run after many prior edits to the same fixture (gotify) in a long-lived VM session — every failure screenshot shows the correct final state already rendered, consistent with CodeMirror's linter work piling up under cumulative session load rather than a real app bug. A settle wait after typing and generous timeouts reduce but don't eliminate it. - §6.23's external-link warning modal doesn't live where the inventory
originally assumed (StackInfoModal) — that modal's own port badges are
the same direct-
window.open()behavior #260 already covers. The realExternalLinkModal-backed warning flow isContainerTable's per-service image name link, rendered inside an expanded stack row in the Power User/Pretty/Unix layouts — moved toe2e/stacks.spec.tsaccordingly. - The auto-refresh degrade/Retry test breaks the VM's real
podmanbinary (sudo mvit aside via SSH, always restored in afinally) rather than mocking a network error — the app's poll failure, the "Failed to load stacks" alert, and the Retry recovery are all exercised against a genuinely broken runtime. - Both the Background Tasks and Bulk Actions runtime-switch-race scenarios
needed a VM with both Docker and Podman installed to exercise a real
runtime switch (
arch-podmanonly has Podman) — completed againstarch-bothas part of Wave 2 instead, see thearch-bothfindings above.
-
-
fedora-bothUp-flow stall, unresolved (this pass): every automated Up attempt againstgotify(viaupStack()or an inlined equivalent) freezes right after the "Up" button click — the confirm dialog and progress modal's "Close" button clicks all report success with no thrown errors, but a frame-by-frame video capture shows literally zero visible UI change for the test's entire duration, andjournalctl -u dockeron the VM shows zero dockerd activity during the attempt. Ruled out: stale VM disk state (freshvm rebuildreproduces it identically), host resource contention (stopping 4 other running VMs first didn't help), worker parallelism races (--workers=1reproduces it too, in fact slightly worse), rootful-Docker permission gaps (confirmeddocker compose up/downboth work fine unprivileged over plain SSH), and leftover container/network name conflicts (confirmed cleandocker ps -abefore each attempt). A manual Up through a real (non-Playwright) browser session against the same VM instance worked correctly.debian-both— identical scenario config, same test files — passes 14/14 cleanly on both a fresh start and a full rebuild. Left unresolved as a real, reproducible-but-unexplained gap for a future session with interactive browser debugging tools (breakpoints/devtools), rather than continuing to guess blindly. -
Wave 3/4 additional findings (this pass):
-
RestoreModal's target directory field is the PARENT directory (the app appends the stack name itself), same convention as Create Stack — the first version ofbackup-restore.spec.tsassumed otherwise and produced a nested directory; fixed in the test, not a code bug. -
RestoreModalauto-closes viaonRestoredon success with no dedicated success screen, unlikeBackupModalwhich shows one — a real UX asymmetry between the two modals, documented ine2e/backup-restore.spec.tsbut not changed (no product decision requested for this pass). - Background Tasks: clicking a finished task to view its captured log
(
BackgroundTaskLogModal) — the earlier "fragile" flakiness was two real, separate causes, both root-caused and fixed in a later pass: (1) a leftover task from an earlier failed debugging run made the task-row locator match a stale row instead of the fresh one under test, looking like a real race but wasn't — fixed by clearing leftovergotifytasks at the start of the test; (2) a genuine app bug — the still-open drawer's own header physically overlaps and intercepts pointer events meant for the log modal's footer "Close" button (both use the same z-index token), so a real mouse click there silently does nothing for a real user too. Filed as issue #283; the spec worked around it with a programmatic click until the fix (#309), and now uses a real.click().
-
- The "check config didn't break" principle from issue #227 has no literal
Caddyfile-equivalent in this app (no reverse-proxy feature exists). It's applied
here as: after YAML/env edits, assert file content really changed via a re-scan or
cockpit.file()read; after Up/Down/Scale, assert real container/service state (via Stack Info or a second scan) rather than only a UI class/badge. - Wave 1 specs should be written first and always run on the canonical 3-VM set in CI-equivalent local runs; Wave 2 specs are the ones that justify the full matrix.
- This table should be kept up to date as specs are written — flip 🚧 to ✅ in the same PR that adds the spec.
Cockpit Compose
Using the UI
- Stacks Dashboard
- Managing Stacks
- Bulk Actions
- Background Tasks
- Viewing Logs
- Editing Configuration
- Stack Info
- Pulling Images
- Events
- Process Viewer
- Shell Access
- Running Commands
- Prune Resources
- Scaling Services
Stack Management
Runtime Support
Help
Development