clean machine has been red on main since it was added, and both its MISSes are real

[phmatray]

Problem

The clean machine workflow (the install check) has failed on every run of main since it was
merged in #4 — 6a931b1, 1510f6d, 4f10635, 8fdab67, b6ee2ad. A check that is always red is not
a gate: PR #8 landed with it failing and nobody could tell from the check whether that PR caused it.

Its two MISSes are not CI artifacts. Both are real, and one of them is a user-visible defect on a
genuinely fresh install:

MISS failed user units: macarchy-auto-appearance.service macarchy-bar-contrast.service macos-dynamic-wallpaper.service
MISS dynamic wallpaper config: macos-dynamic-wallpaper: wallpaper not found:
     /home/runner/.config/omarchy/backgrounds/apple-glass/26-tahoe-beach-day.jpg

The wallpaper MISS is a genuine bug the laptop cannot see. macos-dynamic-wallpaper seeds its
default config from examples/dynamic-wallpaper.json, which names the tahoe-beach set
(26-tahoe-beach-dawn/day/dusk/night.jpg). apple-glass/backgrounds/ ships three files:

1-sequoia-dusk.jpg   2-graphite.jpg   3-aurora.jpg

Nothing in REPOS installs a 26-tahoe-* file. It is green on the maintainer's laptop only because
those files already sit in ~/.config/omarchy/backgrounds/apple-glass/ from local history — exactly
the class of bug the comment at the top of clean-machine.yml says the workflow exists to catch.

The failed-units MISS cannot currently be diagnosed at all. tests/clean-machine.sh never dumps
journalctl --user -u <unit> for a unit it reports as failed, so CI states the symptom and withholds
the cause. Whether those three units legitimately cannot run without a compositor, or are genuinely
broken on a clean machine, is not answerable from the run log.

Expected

clean machine is green on main, and goes red only when a change actually breaks a clean install —
so it can gate again.

Area

CI / doctor.sh / the component install chain.

Related: #4 (introduced the workflow), #8 (evidence gathered in its comment)

🧠 Brainstorm

Problem / context

clean-machine.yml was added in #4 with an explicit rationale: "Nothing ever ran install.sh anywhere
but the one laptop, and that is exactly how two 'nobody wired the startup' bugs survived." It worked —
it found a third such bug immediately. But the run was never brought back to green, so the signal was
lost the moment it was produced: five consecutive red runs later, the check carries no information,
and the one PR open against the repo had to be merged past it by hand.

The two MISSes are different in kind and must not be treated as one:

• The wallpaper one is a product defect. A user who installs the suite today on a clean machine
  gets a dynamic-wallpaper config pointing at four files nobody ships. It is not about CI at all; CI
  is merely the only place it is visible.
• The failed-units one is unknown. It may be a product defect too, or it may be three units that
  legitimately cannot start without a compositor. The run log does not say, and guessing which is
  precisely the failure mode ci: prove install.sh and doctor.sh on a clean machine #4 was written to end.

Approaches

A. Gate the failing checks off on CI (MACARCHY_NO_HARDWARE-style skip) and move on.
Green immediately, one line per check. But it would suppress the wallpaper MISS, which is a real bug
on real machines — it would turn the one workflow that found a genuine defect into one that is
configured not to. Rejected.

B. Fix the defects, leave the workflow alone.
Honest, but blocked out of the gate: the failed-units MISS cannot be fixed before it can be read, and
nothing in the run log says why those units failed.

C. Diagnose first, then fix, then let the check gate again. (recommended)
Make CI say why a unit failed (Task 1), which costs a handful of lines and permanently improves
every future red run. Then fix the wallpaper defect at its source (Task 2), and dispose of the three
units on the evidence Task 1 produces (Task 3) — either a real fix, or a narrow documented skip for
what genuinely needs a compositor, decided from a journal rather than from a guess.

Recommendation

C. The order is the point: Task 1 is what turns Task 3 from speculation into a decision, and it is
the cheapest of the three. Approach A is explicitly rejected — a skip is only acceptable for something
proven to need hardware CI does not have, which is what Task 1 establishes and Task 3 applies.

📋 Spec

Goal

clean machine green on main and meaningful as a gate, with both MISSes resolved on their merits —
no MISS silenced that a real machine would also report.

Scope

• tests/clean-machine.sh — dump the journal of every failed user unit before doctor.sh decides.
• The seeded dynamic-wallpaper default — make a clean install name wallpapers that are actually installed.
• The three failing units — fix, or narrowly skip with a stated reason, per Task 1's evidence.

Non-goals

• Any change to doctor.sh's verdicts on a real laptop. The doctor is reporting the truth; this issue
  is about making the truth actionable and then making it green.
• Restructuring the component install chain.

Design

flowchart TD
    A[install.sh twice] --> B[dump journal of each failed user unit]
    B --> C[doctor.sh decides]
    C -->|wallpaper MISS| D[seeded default names a shipped set]
    C -->|units MISS| E{journal says why}
    E -->|needs a compositor| F[narrow skip, reason stated]
    E -->|genuinely broken| G[fix the unit]

Loading

The wallpaper fix has two viable shapes, and Task 2 picks one on evidence rather than in advance:
either apple-glass ships the 26-tahoe-beach-* set it is already assumed to own, or
the seeded default names a set apple-glass does ship (1-sequoia-dusk and friends). The second is
smaller and stays inside this repo's dependency direction; the first is right if those wallpapers were
always meant to be part of the theme. Whichever lands, the check that proves it is the same.

Edge cases

• A unit that failed and was then restarted must not be reported — systemctl --user --failed already
  reflects only the current state.
• The journal dump must not fail the step when a unit has no journal (|| true), and must not run at
  all when nothing failed.
• macarchy-failed@<unit>.service instances fold onto their target in doctor.sh:63; the dump must
  use the folded name, not the instance.

Assumptions

• The three units fail for want of a compositor / Omarchy install rather than a packaging defect —
  unverified, and Task 1 exists precisely to settle it. Task 3 is written to accept either answer.
• 26-tahoe-beach-*.jpg on the maintainer's laptop came from local history, not from any repo in
  REPOS — established by find over apple-glass returning nothing.

Acceptance criteria

1. On a clean runner, a failed user unit's journal appears in the run log, under the unit's name.
2. clean machine is green on main — doctor.sh reports 0 missing.
3. A fresh install produces a dynamic-wallpaper config whose every referenced file exists on disk;
   macos-dynamic-wallpaper status exits 0 with no wallpaper not found.
4. Any check skipped on CI names the reason in its skip line, and still reports MISS on a real
   laptop where the condition does not hold.
5. Re-running tests/clean-machine.sh twice in a row is still idempotent — the second run adds no new MISS.

Testing decisions

Seams under test: tests/clean-machine.sh's stdout and exit code (the whole workflow step is one
script, already the CI seam), and doctor.sh's MISS/ok lines — the same text-level seam
tests/test_doctor_notify.sh and tests/test_doctor_touchbar_modules.sh already assert through.
Prior art: tests/test_doctor_touchbar_modules.sh — temp HOME, PATH stubs, a stubbed
journalctl replaying a fixture.
A good test here: stub systemctl --user --failed to report one unit, assert the journal for that
unit reaches stdout and that a run with nothing failed emits no dump.

Out of scope

• Making clean machine a required check — that is a branch-protection decision for the owner once it
  is reliably green.
• The 26-tahoe-* wallpapers' provenance or licensing.

🛠️ Implementation plan

For agentic workers: execute this plan task-by-task with implement-issue (subagent-per-task for broad plans, inline for small ones). Steps use checkbox (- [ ]) syntax for tracking.

Goal: clean machine green on main and gating again, with both MISSes resolved on their merits.
Architecture: plain bash — install.sh, doctor.sh, tests/*.sh; the suites are hermetic (temp HOME + PATH stubs) and CI runs tests/test_*.sh plus bash -n.
Tech stack: bash 5, gh, GitHub Actions on ubuntu-24.04-arm.

Seams under test: tests/clean-machine.sh's stdout and exit code, and doctor.sh's MISS/ok lines — the text-level seam the existing doctor suites already assert through.

Global constraints

• Hermetic suites only: temp HOME, PATH stubs, no compositor, no Apple hardware. CI runs tests/test_*.sh by glob — a new suite must match it.
• bash -n must pass on install.sh, boot.sh, doctor.sh and tests/*.sh.
• Never silence a check that a real laptop would legitimately report. A skip states its reason and is gated on a condition that is false on the laptop.
• Conventional Commit PR title decides the release bump; commits below use fix.
• Commit trailer: Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>.

Task 1: CI says WHY a user unit failed

Files: modify tests/clean-machine.sh; test tests/test_clean_machine_journal.sh (new).

Interfaces: after the install.sh runs and before doctor.sh, for each unit in systemctl --user --failed --no-legend --plain, print a labelled journalctl --user -u <unit> --no-pager -n 40 block. Folds macarchy-failed@<u>.service onto <u> the same way doctor.sh:63 does.

• Step 1: Write the failing test in tests/test_clean_machine_journal.sh (seam: the script's stdout) — stub systemctl to report one failed unit and journalctl to emit a marker line; assert the marker and the unit name both reach stdout.
• Step 2: Add the second case to the same suite — systemctl --user --failed reports nothing, assert no dump block is emitted.
• Step 3: Run bash tests/test_clean_machine_journal.sh → FAIL (no dump exists yet).
• Step 4: Implement the dump block in tests/clean-machine.sh, || true on the journal call so a unit with no journal cannot fail the step.
• Step 5: Re-run the suite → PASS, and bash -n clean on every tests/*.sh.
• Step 6: Commit: fix(ci): dump the journal of every failed user unit on a clean machine.
• Step 7: Push, let clean machine run, and record the three units' journals in a comment on this issue — Task 3 is written against that output.

Task 2: a fresh install names wallpapers it actually installs

Files: modify whichever of macos-dynamic-wallpaper's seeded default or apple-glass/backgrounds/ the evidence selects (cross-repo — see note); modify tests/clean-machine.sh only if a new assertion is needed.

Interfaces: after a clean install.sh, macos-dynamic-wallpaper status exits 0 and every path in the seeded config exists.

• Step 1: Establish which repo owns the 26-tahoe-beach-* set — git log in apple-glass for a removal, and the provenance of the copies in ~/.config/omarchy/backgrounds/apple-glass/. Record the finding in a comment on this issue.
• Step 2: Decide on that evidence: apple-glass ships the set, or the seeded default names a set apple-glass ships. State the choice and why in the same comment.
• Step 3: Write the failing assertion — extend tests/clean-machine.sh so a wallpaper not found in doctor.sh's output fails the step with that line quoted (seam: the script's exit code + stdout).
• Step 4: Run tests/clean-machine.sh locally against a temp HOME → FAIL on the missing wallpaper.
• Step 5: Apply the chosen fix in the owning repo. If it lands outside macarchy-install, open the PR there and link it here — do not vendor another repo's assets into this one.
• Step 6: Re-run → PASS.
• Step 7: Commit: fix(install): seed a dynamic-wallpaper default whose files are installed.

Task 3: dispose of the three failing units on Task 1's evidence

Files: modify doctor.sh and/or the failing units' installers, per the journals; test tests/test_doctor_notify.sh (extend).

Interfaces: doctor.sh reports 0 missing on a clean runner without hiding anything a laptop would report.

• Step 1: Read the journals Task 1 published on this issue and classify each of the three units: needs a compositor / Omarchy, or genuinely broken on a clean machine.
• Step 2: For every unit in the first class — write the failing test first: assert doctor.sh prints skip with a stated reason when the gating condition is set, and MISS when it is not (seam: doctor.sh's output lines).
• Step 3: Run → FAIL.
• Step 4: Implement, following MACARCHY_NO_HARDWARE's existing shape — a named condition CI sets and the laptop never does.
• Step 5: For every unit in the second class, fix the unit itself and note the defect in a comment; do not skip it.
• Step 6: Re-run the full suite and bash -n → PASS.
• Step 7: Commit: fix(doctor): resolve the user units that fail on a clean machine.
• Step 8: Confirm clean machine is green on the PR, then say in the PR body that the check is ready to be made required.

[phmatray]

Task 1 evidence — the three journals, from the first run that could print them

Run 33925266524, on fix/9-…. Trimmed to the line that decides each:

=== journal: macarchy-auto-appearance.service ===
macarchy-auto-appearance: macarchy-sun could not compute today's sun; theme left alone
macarchy-auto-appearance.service: Main process exited, code=exited, status=1/FAILURE

=== journal: macarchy-bar-contrast.service ===
macarchy-bar-contrast: grim missing
macarchy-bar-contrast.service: Main process exited, code=exited, status=1/FAILURE

=== journal: macos-dynamic-wallpaper.service ===
macos-dynamic-wallpaper: wallpaper not found: /home/runner/.config/omarchy/backgrounds/apple-glass/26-tahoe-beach-day.jpg
macos-dynamic-wallpaper.service: Main process exited, code=exited, status=1/FAILURE

Classification for Task 3 — and it is not what the plan assumed

The plan's Task 3 offered two buckets: needs a compositor → narrow skip, or genuinely broken → fix. The journals say all three are the second bucket, and two of them share one defect:

auto-appearance and bar-contrast both decide correctly, then exit 1 anyway. Each detects that its precondition is absent (macarchy-sun has no sun to compute; grim is not installed), each says so, each declines to act — "theme left alone" is a deliberate no-op, not a failure — and each then returns 1. A oneshot unit that exits non-zero for a decision it made on purpose stays in systemctl --user --failed for ever and fires its OnFailure= notifier. This is a real bug on a real laptop too: boot before grim is installed, or with no coordinates set, and the same two units go permanently failed.

So there is nothing here to gate off on CI. The fix is that a deliberate no-op exits 0.

macos-dynamic-wallpaper is Task 2's bug, seen from the other side — the same missing wallpaper, reported by the unit instead of by the doctor. Task 2 closes it; no separate work in Task 3.

Two things the dump surfaced that were not in the plan

• Failed to enqueue OnFailure= job, ignoring: Unit macarchy-failed@macarchy-auto-appearance.service.service not found. — the failure notifier's template unit is not installed for these units on a clean machine. The doctor's failure notifier check passes because it only looks for macarchy-failed@.service in ~/.config/systemd/user; it never checks that a unit naming it in OnFailure= can actually reach it. Out of scope here — worth its own issue.
• The two units live in macarchy-core, not this repo, so Task 3 lands there and is linked back, exactly as Task 2's Step 5 already anticipates for the wallpaper.

[phmatray]

Tasks 2 and 3 — and where the plan was wrong

clean machine is green: 37 ok, 0 missing, 2 skipped, printed twice (both install.sh passes, so idempotence holds). The two skips are the pre-existing legitimate ones — no macsmc battery, no Hyprland session. No new skip was added; nothing was silenced.

Task 2 turned out to have two causes, not one

The plan assumed the seeded default named the wrong wallpapers. It did — but fixing that alone left CI red on 3-aurora.jpg, a file apple-glass does ship.

The second cause: nothing ever copies a theme's backgrounds/ into ~/.config/omarchy/backgrounds/<theme>/, which is where macos-dynamic-wallpaper looks. Only omarchy theme set populates it, so a machine without Omarchy — and CI, which never has one — never gets them. This is true on the maintainer's laptop too: a dry-run rsync there adds 1-sequoia-dusk.jpg, 2-graphite.jpg and 3-aurora.jpg and touches none of the 23 files already present. The glass default would have failed there as well.

Then a third, smaller one: the copy was first placed with the theme sync, which runs after the step that starts the unit — and a oneshot that fails once stays failed. It now runs before.

Landed as:

• fix: seed a default whose wallpapers are actually installed macos-dynamic-wallpaper#3 — the default is the glass set, built from what apple-glass ships, ordered by measured luminance.
• fix(install): put the theme's wallpapers where the tool looks for them + fix(install): copy the wallpapers in before the unit that reads them (this PR). --ignore-existing, never --delete: that directory is also where a user keeps their own wallpapers.

Task 3's classification held, and the fix was not a skip

macarchy/macarchy-core#12. Both auto-appearance and bar-contrast decided correctly, said so, and then returned 1. auto-appearance already used exit 0 for its sibling decision ("user chose something else; leave it alone") — the sun case now matches its own convention. Genuine failures still red: exec omarchy theme set propagates, and bad sample exits 1 is untouched.

Three tests asserted the old exit code and were changed with their reasoning. The invariant they defend — "it must not GUESS a theme" — is the does not switch assertion beside each, and that one was not touched.

Two plan steps satisfied by existing machinery rather than new code

Task 2's steps 3 and 4 asked for a new assertion in tests/clean-machine.sh that quotes a wallpaper not found. Not written — doctor.sh's own dynamic wallpaper config check already fails the run and quotes that exact line, and Task 1's journal dump now quotes it a second time from the unit's side. The red→green transition is in the CI transcript. A third assertion for the same fact would have been redundant, so the boxes are ticked for the outcome, not for a duplicate.