0.1.0: backup-staged bare-OS reinstall of homeserver + VPS, then converge install scripts into one configurable installer

[Xore]

Purpose

Single tracking issue for a staged secret/golden-image backup, then a full bare-metal-up reinstall of the homeserver and VPS via scripts/install-homeserver.sh / scripts/install-vps.sh, using the current machine (SSH alias backup) as the intermediate backup host, fixing both install scripts live against real hardware, and converging them toward one shared, configurable install script. This is a 0.1.0-adjacent gate — related to but distinct from #1524 (which covers the dashboard bug-hunt + VPS-deploy-pipeline + golden-image/detonation checklist) and #787 (the previous full clean-reinstall, already closed). Cross-reference both rather than duplicating their scope.

Why now, specifically: secrets have been lost across multiple past reinstalls. A live audit for this issue (2026-08-18, see "Live audit findings" below) found a concrete, previously-unknown root cause: the nightly backup timer on the homeserver has been silently failing for 24h+. This issue exists to fix that class of problem structurally, not just patch the one instance.

Roles for this pass

Host	SSH alias	Role in this plan
This machine	backup	Intermediate backup host — stages secrets + golden images before either target is wiped
Homeserver	homeserver	Reinstall target 1
VPS	vps	Reinstall target 2

(Real addresses/ports intentionally omitted from this public issue — see the operator-only backup documentation staged on the USB drive for connection details.)

Live audit findings (2026-08-18)

Read-only audits were run against both live hosts to ground this plan in reality rather than assumption. Key findings:

Homeserver

• Root cause candidate for lost secrets: backup-honeypot.timer fires nightly, but backup-honeypot.service has failed since at least 2026-08-16 — analysis/backup-honeypot.sh lost its executable bit on the live host (mode 644, owned github-deploy-runner), so every run since has been Permission denied (exit 203/EXEC). /opt/backups/honeypot is stale by at least a day, likely much longer. This is very plausibly the actual mechanism behind the repeated secret loss, not just an unrelated bug.
• /opt/stacks/apiary (the old monorepo checkout) is stale/legacy, not a sign of a broken deploy pipeline. It sits at cfa6fbb (171 commits behind main) with a large uncommitted diff. Initial read of this looked alarming, but per clarification: since feat(deploy): migrate home APIARY stacks to Arcane directory-aware Git sync #1502, the 32 honeypot-* stacks are each independently materialized by Arcane's own directory-aware Git sync straight from arcane/manifests/home-production.json into /var/dockge/stacks/<syncName>/ — confirmed by reading docs/ARCANE-GIT-SYNC.md and the manifest (38 entries, each its own gitRepo/branch/dockerComposePath). /opt/stacks/apiary is not the live source of truth for those 32 stacks and this drift is expected, not a symptom. It's still the entry point for the 6 self-contained stacks (auth-events-worker, llm-worker, ml-worker, analysis/ghidra, sandbox/ghosts, pihole) and for the install script itself — worth confirming during the reinstall smoke test that those 6 are being read from a checkout on the correct ref, but this is not evidence of an out-of-band deploy path.
• One already-live local hotfix to analysis/backup-honeypot.sh (added retention_days + an ES-snapshot tar-exclude) predates the repo's current arcane/home/... path layout post-feat(deploy): migrate home APIARY stacks to Arcane directory-aware Git sync #1502. Not being reconciled as part of this issue — the ES-snapshot step it touches is out of scope entirely now that the reinstall intentionally starts with a fresh Elasticsearch store (see Non-goals).
• Golden images + ISOs + VM overlays total ≈106G (golden-images/: win11-analysis 24G, win11-cape 18G, win11-ghosts 24G; isos/: 20G; vms/ thin overlays: ~20G).
• Ollama models: Docker volume ghidra_ollama_models, 17.6G, cleanly isolated to the ghidra compose project — safe to exclude from backup (re-pulled on reinstall), confirmed nothing else co-located.
• Disk: OS 233G/138G avail, /mnt-2 440G/333G avail, /mnt-1 1.8T/1.7T avail (near-empty), /var (Docker/Arcane/sandbox) 1.8T/961G avail.
• Secrets enumeration needs root: this session's xore account can't read most stack .env files, /etc/wireguard/, or Arcane's token store — the actual backup run needs sudo (or run as root directly).

VPS

• Secrets: /root/vps/.env, /root/vps/traefik/certs/{origin.pem,origin-key.pem} (live Cloudflare origin cert), /etc/wireguard/wg0.conf (tunnel confirmed healthy — real handshake, 315G/25G transferred all-time), /root/.ssh/authorized_keys.
• Two separate checkouts that don't map onto install-vps.sh's single-REPO_DIR model: /root/apiary-repo (full monorepo git checkout, clean, HEAD 6ebf11f) vs. /root/vps (the actual deployed stack dir — plain rsync target, not a git repo at all). The install script's mental model (one REPO_DIR, deploy in place) doesn't match how the VPS is actually kept in sync today (CI rsyncs a subset out of the checkout). Reconcile during the live run.
• No backup/sync off the VPS exists at all today — /root/vps-backups/ only holds pre-deploy tarballs written by CI as a same-host safety net, nothing copied off-box.
• Preflight: Release 0.1.0 #1524's "VPS deploy pipeline broken (SSH key untrusted)" item may already be resolved — auth.log shows the unlabeled deploy key successfully authenticating as root as recently as 2026-08-15, and honeypot-dashboard-ghidra-report's Traefik router (the confirmed-missing casualty cited in Preflight: Release 0.1.0 #1524) is present in the live dynamic.yml. Re-verify before treating that item as still blocking; don't re-do the fix if it's already shipped.
• SSH admin port (2222) matches the conf example exactly, no drift.

Repo sweep (stale scripts/docs)

• Two concrete, low-risk fixes found (not deferred — fold into this issue's cleanup pass):
  • docs/OPERATIONS.md:31,139 — still says "Dockge's .env" as the live management tool; should name Arcane.
  • docker-compose.yml:1 — header comment says "runs under Dockge", contradicted by its own body 5 lines later (kept only as a deploy.yml sanity-check marker post-split).
• Concrete instance of the "unify the two install scripts" goal: scripts/install-vps.sh:364 hardcodes REPO_DIR="/root/apiary-repo" with no config variable at all, while scripts/install-homeserver.sh makes REPO_DIR fully configurable via install-homeserver.conf.example. The two scripts don't even agree today on whether this path is configurable.
• docs/ROADMAP.md's staleness (flagged by the earlier Full-installation smoke test + single-entrypoint setup script (home + VPS) #518 research doc) is already fixed — no action needed there.
• An orphaned-script pass (every .sh under repo root/scripts//analysis//sandbox/ cross-referenced against compose files, workflows, and docs) was not completed — the sweep timed out. This needs a real pass as part of this issue, see checklist below.

This backup host

• 124G free on /. No existing APIARY-related backup staging directory. /var/backups is just stock apt/dpkg housekeeping, unrelated. 124G is tight against the ~106G of golden images alone once secrets and any staging overhead are added — plan for streaming the transfer rather than double-buffering, or attach/borrow more headroom before starting (see checklist).

Live-verified corrections to the above (2026-08-18, second pass)

A few items above were checked directly against the live hosts rather than taken from docs/scripts, and either confirmed or corrected:

• Docker default-address-pools fix is already automated — scripts/install-homeserver.sh's step_docker_daemon_config (L1736) writes exactly the live /etc/docker/daemon.json (172.16.0.0/12//24), confirmed byte-identical. docs/STACK-REBUILD.md's address-pool-exhaustion pitfall (hit once historically) is already closed by the install script itself — no action needed here, just don't re-introduce it while touching that step.
• Traefik dynamic.yml currently matches between repo and live VPS — 39 routers, identical key set, confirmed by direct diff. There's no live drift to reconcile right now. The structural risk (an unconditional overwrite with no live-vs-repo diff check — see "Known failure modes" below) is still real for the future, so the pre-wipe-diff checklist item stays, just not because of a currently-known gap.
• GPU passthrough for the Windows sandbox is a bigger, concrete gap than the initial pass suggested — three separate, independently-verified problems, not one:
  1. sandbox/windows/packer/win11-kvm.xml's own comment (around the <hostdev> block) claims the passthrough card is "a real AMD Radeon PRO WX 2100" bound via /etc/modprobe.d/vfio-gpu-passthrough.conf "already bound at boot." Neither is true on the live homeserver right now: that file does not exist (No such file or directory), and the card actually occupying that exact PCI address (17:00.0/17:00.1) is an NVIDIA Quadro P2200 (lspci confirms), not an AMD card — matching Preflight: Release 0.1.0 #1524's own description, not the XML comment. The XML comment is stale and describes hardware that isn't there. Fix it as part of the stale-docs cleanup (Phase 6).
  2. IOMMU itself does appear active (87 groups under /sys/kernel/iommu_groups) despite no explicit intel_iommu=on/amd_iommu=on in /etc/default/grub — likely default-enabled by this kernel/BIOS combination. Not a gap, just undocumented.
  3. scripts/install-homeserver.sh has no step anywhere that defines or starts a VM from win11-cape.qcow2. virsh list --all on the live box shows only win11-ghosts and win11-sandbox defined — no win11-cape domain exists, and grepping the install script for cape only turns up a comment about the golden image file itself, never a run_step that instantiates it. This directly blocks Preflight: Release 0.1.0 #1524's "all three golden images boot cleanly" acceptance item — it can't pass today because the automated install path never creates that VM in the first place, independent of whether GPU passthrough itself works. This issue should land the missing win11-cape VM-creation step (mirroring win11-sandbox's existing thin-clone pattern) and the missing vfio-gpu-passthrough.conf provisioning, not just document the gap.

Known failure modes & things to pay attention to

Both install scripts carry dense incident commentary from real past runs (#518, #787, #1524 and others). This is a condensed, file:line-traceable punch list of what to actually watch for on this run — read before starting, don't rediscover these live a second time.

Things a live run could silently destroy if not careful

• install-vps.sh's step_render_traefik_dynamic (~L446-464) unconditionally overwrites /root/vps/traefik/dynamic.yml with a domain-substituted copy of the repo's own committed file (deliberately cat, not mv, so Traefik's bind mount keeps tracking the same inode). Anything live on the VPS that isn't also committed to vps/traefik/dynamic.yml in the repo gets silently deleted — this is exactly what VPS deploy: Traefik dynamic.yml domain-substitution step left the live config with placeholder hostnames #1225 hit once already. Currently repo and live match (verified 2026-08-18, see above), but diff live vs. repo immediately before the VPS is wiped, and commit anything live-only first.
• install-vps.sh's firewall-base step does ufw --force reset; ufw default deny incoming; ufw --force enable before firewall-honeypot-ports. It opens SSH_ADMIN_PORT (2222) as part of the same step, so it's safe if the operator is already connected via 2222 (or console/OOB) when it runs — but running it over a port-22 session, or before confirming 2222 is reachable, can lock the operator out mid-step on a remote box with no console access. docs/CGNAT-DEPLOYMENT.md's manual bootstrap order flags the identical hazard independently. Confirm admin access is via 2222 (or console) before this step runs.
• step_restore_certs's failure path is non-fatal by design — if cert restore from the backup host fails or is partial, compose-up still proceeds and Traefik comes up without valid TLS, and container-health checks won't catch it (Traefik reports "Up" either way). Add an explicit post-run TLS check (e.g. curl -v against a live hostname) rather than trusting verify-containers.
• umask 077 set anywhere in these scripts leaks shell-wide unless scoped to a subshell — this exact bug (0.1.0 release gate: full clean-reinstall + end-to-end smoke test on homeserver + VPS #787) silently downgraded every file the subsequent git clone wrote from 0644/0755 to 0600/0700, breaking non-root container reads of bind-mounted files. Watch for this pattern in any new code added while fixing the scripts live.

Two-pass / manual-intervention checkpoints (plan for these explicitly, don't get surprised mid-run)

• ARCANE_API_TOKEN (install-homeserver.sh): irreducibly two-pass. First run lands Arcane + Keycloak; a human logs into Arcane's UI once, mints an API key by hand (Settings → API Keys), fills it into the conf, re-runs for step_arcane_import_stacks to complete.
• WireGuard keypairs (both scripts): leave blank to bootstrap fresh on a genuinely first run; pin to real values from the resulting wg0.conf on any re-run to avoid unnecessary peer churn. Order: run install-vps.sh first (writes a placeholder peer entry), then install-homeserver.sh's step_wireguard_sync_vps_peer fills in home's real key second.
• NVIDIA driver install requires a reboot before the kernel module loads — step_gpu_verify_or_note_reboot detects this and tells the operator to reboot + re-run with --force-rerun-from gpu-verify rather than hard-failing. Know the correct step-id in advance so the re-run doesn't accidentally redo (or skip) more than intended.
• GHOSTS client enrollment is deliberately deferred (--skip-enroll-test) — the real confirmation (build the Windows client, run it once, poll the API) is a manual post-restore step, not part of the unattended flow.

Step-ordering dependencies called out in the scripts themselves

• wireguard-config → wireguard-sync-vps → wireguard-verify must complete before anything reaching the VPS-mounted sshfs paths later.
• start-elasticsearch → start-init → start-remaining (Arkime/ES setup jobs need a healthy Elasticsearch first, or they hang/loop — see docs/STACK-REBUILD.md).
• ghidra-provision → ghidra-start → ollama-model-pull → ghidra-worker-install → ml-worker-start — skipping the worker-install step leaves the containers up but triage completely non-functional (Ghidra automated triage worker (ghidra-worker.py) never deployed by install-homeserver.sh #636: zero completed analyses, easy to miss since nothing errors).
• render-traefik-dynamic runs after restore-certs but before compose-up — see the non-fatal cert-failure note above.

Known live pitfalls (from docs/STACK-REBUILD.md and docs/CGNAT-DEPLOYMENT.md, worth re-checking for on this run)

• snare needs literal chown root:root on its directory (not the 65534:65534 UID every other Tanner-group sensor uses) or it crash-loops — check_privileges() runs before privilege drop while withholding DAC_OVERRIDE.
• A fresh Elasticsearch data wipe makes Arkime cycle through a "fresh install... must do init" restart loop until honeypot-init's setup jobs actually run — expected, not a bug, but easy to mistake for one mid-smoke-test.
• Elasticsearch has no host-published port — curl localhost:9200 never works from either host directly; go through a throwaway container on the honeynet Docker network.
• NIC hardware GRO on the VPS (rx-gro-hw on virtio-net) makes Suricata log real traffic as "truncated packet" (SID 2200003/2200122) unless disable-nic-hw-gro.sh --apply has run (Suricata: 'truncated packet' alerts flooding logs due to virtio-net rx-gro-hw #342) — not a functional break, but will actively mislead an "is IDS working" smoke-test check if skipped.
• check-firewall-portbridge-sync.sh must be re-run after touching either honeypot-firewall.sh or portbridge's RULES env var — the two are required to stay hand-in-hand (Reconcile VPS firewall policy for FTP, SSH, Telnet, and SMTP honeypot ports #152).
• Docker ps status-text checks have a blind spot for crash-looping containers ("Up" is what shows between crashes, "Restarting" only during the brief transition) — this exact gap let Suricata crash-loop 428 times over ~6 hours with zero IDS coverage before being caught (0.1.0 release gate: full clean-reinstall + end-to-end smoke test on homeserver + VPS #787). Use docker inspect --format '{{.RestartCount}}' in verification steps, not a status-text grep.

GPU passthrough — see the corrected findings above

Treat this as a real gap to close in this issue (missing vfio-gpu-passthrough.conf provisioning + missing win11-cape VM-creation step), not just a note for later.

Phase 0 — Quick fix, before anything else

• Restore the executable bit on the homeserver's live analysis/backup-honeypot.sh — done and verified live (2026-08-18): the config/secrets archive (stack-config-state.tar.gz) now succeeds on a real manually-triggered run, confirmed by a fresh file in /opt/backups/honeypot/. The nightly timer will succeed on the config/secrets side going forward.
• Reconcile the local hotfix diff (retention_days + geoip exclude) into a proper PR — not needed. Fixing the permission bug exposed a separate, pre-existing bug in the same script's ES-snapshot step (curl: 400), but that step is now explicitly out of scope: the reinstall intentionally starts with a fresh Elasticsearch store, no alerts/payloads/historical data carried over — see Non-goals. Nothing here needs reconciling for this issue's purposes.
• Add a CI or deploy-time guard so a shell script silently losing +x on deploy can't recur unnoticed (e.g. .github/workflows/deploy.yml chmod +x on known executable scripts post-rsync, or a quick step that fails the deploy if a run_step/timer-invoked script isn't executable).

Phase 1 — Design and execute the backup (before either host is wiped)

• Enumerate every real secret location on both hosts as ground truth (not from docs) — a root/sudo pass against the homeserver's 38 Arcane-materialized stack directories (/var/dockge/stacks/<syncName>/.env per arcane/manifests/home-production.json, plus the 6 self-contained stacks) and the VPS's /root/vps/.env + certs + /etc/wireguard/.
• Write (or extend analysis/backup-honeypot.sh) a script that pushes that secret set to this backup host, matching the directory convention scripts/install-homeserver.conf.example's BACKUP_HOST_PATH already expects on restore (<path>/home-env/<stack>/.env, <path>/vps/.env + certs) — today only the restore side of that convention exists; there is no corresponding export script.
• Write a golden-image backup script (none exists today — BACKUP_HOST_SANDBOX_PATH is restore-only, same gap): rsync golden-images/, isos/, vms/, and the libvirt XML defs to <path>/home-sandbox on this host, with checksums (sha256sum, matching the .sha256/.sha256.verified convention already used for golden images) verified after transfer.
• Explicitly exclude ghidra_ollama_models (17.6G) and any other LLM model storage — confirmed safe, re-pulled on reinstall, not backed up.
• Verify backup-host disk headroom before starting the golden-image transfer (124G free vs. ~106G of images — cutting it close; free space, mount additional storage, or stream via rsync directly to a mount with more room rather than staging a second local copy).
• Verify the backup: checksum everything, and do a scoped dry-run restore of at least one stack's .env and one golden image's checksum before trusting the set as complete.
• Confirm nothing depends on /opt/stacks/apiary's own uncommitted local drift being preserved — if any of that 128-file diff is real unshipped work (not just stale checkout noise), pull it out and land it as a proper PR before the homeserver is wiped, since a bare-OS reinstall discards it.

Phase 2 — Homeserver bare-OS reinstall

• Operator-driven (outside this session's scope): reinstall Ubuntu Server on the homeserver per docs/HOMESERVER-DISK-LAYOUT.md's autoinstall config — storage partitioning stays manual by design, reproduce the existing 4-disk layout (nvme0n1 OS, sdb→/var, sdc→/mnt-1, sda→/mnt-2).
• Confirm the freshly-installed base box matches the documented disk layout before proceeding.

Phase 3 — Move the backup onto the freshly-installed homeserver

• Transfer the staged backup from this machine to the fresh homeserver.
• Point install-homeserver.conf's BACKUP_HOST/BACKUP_HOST_PATH (and the VPS conf's equivalents) at wherever it landed so the scripts' existing restore steps can consume it directly.

Phase 4 — Live reinstall smoke test (grant SSH access, then run both scripts)

• Run scripts/install-vps.sh first per its own header's documented bootstrap order (fresh WireGuard keypair generated there feeds into the homeserver conf), then scripts/install-homeserver.sh — confirm this ordering still makes sense given Phase 2/3 puts the homeserver's OS reinstall first; reconcile the sequencing explicitly before starting rather than assuming.
• Fix bugs in the scripts live, as they're hit — every fix lands as its own commit, not batched into one after the fact.
• Re-verify the Preflight: Release 0.1.0 #1524 VPS-deploy-pipeline item against fresh state rather than assuming it's still broken (see live audit finding above — evidence suggests it may already be fixed).
• Keep a running findings log for this pass the way docs/research/518-smoke-test-research.md did for the last one, so the next person doesn't have to re-derive live state from scratch again.

Phase 5 — Converge toward one configurable install script

• Extract the duplicated framework (with_retry, run_step, logging, marker-based resumability — install-vps.sh's own header already says this is "identical... deliberately not reimplemented differently") into a shared, sourced file instead of two copies.
• Make every path configurable in both scripts consistently — fix install-vps.sh's hardcoded REPO_DIR="/root/apiary-repo" (line 364) to match install-homeserver.sh's configurable pattern, and vice versa for anything homeserver hardcodes that VPS doesn't.
• Land a single entry point with a --profile home|vps (or equivalent) that defaults every path to this project's real homeserver/VPS setup, while remaining overridable — not a hypothetical third environment, just making the existing two profiles' defaults explicit and consistent instead of one script having a default and the other not.
• Update both .conf.example files to match whatever the merged variable set ends up being.

Phase 6 — Stale scripts/docs cleanup (folded in, not deferred)

• Fix docs/OPERATIONS.md:31,139 — "Dockge's .env" → name Arcane.
• Fix docker-compose.yml:1 header comment to stop claiming it "runs under Dockge".
• Fix sandbox/windows/packer/win11-kvm.xml's <hostdev> comment — describes "a real AMD Radeon PRO WX 2100," but the live card at that PCI address is an NVIDIA Quadro P2200 (matches Preflight: Release 0.1.0 #1524's own description, not this file's). Update the comment to match reality.
• Run a real orphaned-script pass: every .sh under the repo root, scripts/, analysis/, sandbox/ cross-referenced against compose files, .github/workflows/*.yml, and docs; delete or clearly re-scope anything with zero live references, fix anything found to be actively wrong (not just unreferenced).

Phase 7 — Close the GPU-passthrough / win11-cape automation gap

This blocks #1524's golden-image checklist from being able to pass at all, so it's in scope here rather than deferred to that issue:

• Provision /etc/modprobe.d/vfio-gpu-passthrough.conf (or equivalent vfio-pci binding) as an install-homeserver.sh step — it's currently referenced as existing by win11-kvm.xml's own comment but doesn't exist on the live host and isn't written by any script.
• Add the missing win11-cape VM-definition/start step to install-homeserver.sh, mirroring win11-sandbox's existing thin-clone-from-golden-image pattern — today only win11-ghosts and win11-sandbox are ever defined; win11-cape.qcow2 sits on disk with no automated path to a running VM.
• Confirm IOMMU is enabled by design on this hardware (87 groups present with no explicit kernel cmdline flag) and document how, so a future BIOS/kernel change doesn't silently disable it without anyone noticing.

Phase 8 — Golden-image / detonation sanity

• Don't re-litigate the checklist itself — it's already tracked in Preflight: Release 0.1.0 #1524 (all three images boot clean, GPU passthrough, full detonation run, verify_vm_detection.py clean pass, dashboard reflects live data). Run that checklist against the freshly reinstalled hosts as this issue's own closing step and close it there, once Phase 7 above has made it possible for win11-cape to actually exist as a VM.

Non-goals

• Not re-doing Preflight: Release 0.1.0 #1524's dashboard bug-hunt pass or Keycloak end-to-end login checklist — that issue owns those, this one only feeds it a freshly reinstalled pair of hosts to run them against.
• Elasticsearch data is explicitly not carried over. The reinstall is a deliberately fresh ES store — no alerts, payloads, or historical event data migrate across. analysis/backup-honeypot.sh's ES-snapshot step is out of scope entirely for this issue (including the curl: 400 bug uncovered while fixing the permission bug in Phase 0) — not worth fixing here since nothing depends on it for this reinstall. Revisit only if a future need for historical ES data retention comes up, as its own issue.
• Not adding a third host/role — "configurable paths" means making the existing home/VPS defaults explicit and consistent, not building support for hypothetical future deployment shapes.

Acceptance criteria

• Nightly homeserver backup timer is green again and has been for at least one real cycle.
• A verified secrets + golden-image backup exists off both original hosts before either was wiped, checksummed, with at least one spot-checked restore.
• Both hosts are reinstalled from bare OS using this repo's own scripts, with no manual undocumented steps required beyond what docs/HOMESERVER-DISK-LAYOUT.md's "storage stays manual" already documents.
• install-homeserver.sh and install-vps.sh share their retry/step/logging framework and agree on which paths are configurable, with working .conf.example files for both.
• The stale-doc fixes (Dockge wording x2, win11-kvm.xml's GPU comment) and the orphaned-script pass are merged.
• win11-cape has an automated VM-creation path and GPU passthrough is provisioned by the install script, not assumed to pre-exist.
• Preflight: Release 0.1.0 #1524's golden-image/detonation checklist passes against the rebuilt hosts, including win11-cape.

[Xore]

Backup executed (2026-08-18) — findings worth carrying forward

The Phase 1 backup (secrets + golden images) has actually been run now, staged on the backup host's attached USB drive (details, real hosts/paths intentionally kept off this public issue — see the operator runbook written to that drive). All three golden images, VPS secrets, and every stack .env are backed up and checksum-verified. Two concrete findings from doing this for real:

1. Golden-image checksums can go stale, not just missing. win11-cape.qcow2's on-disk .sha256 predated the qcow2's own last-modified time by ~4.5 hours — i.e. it was computed before the image was last rebuilt. This initially looked like backup corruption (checksum mismatch, byte-identical file size) until re-hashing the source file directly confirmed the recorded checksum was simply outdated, not the copy. Worth hardening: whatever regenerates a golden image should also regenerate its .sha256 at the same time, not leave it to whoever next happens to notice it's missing (the fix install-homeserver.sh already has for missing checksums, per 0.1.0 release gate: full clean-reinstall + end-to-end smoke test on homeserver + VPS #787, doesn't catch a stale one).
2. honeypot-stack/ has a real, live .env on the homeserver (2188 bytes, actively populated) — not just a stale pre-feat(deploy): migrate home APIARY stacks to Arcane directory-aware Git sync #1502-rebrand path reference as assumed elsewhere in the repo's own comments (install-homeserver.sh around L1247-1249). The Phase 6 orphaned-script/stale-docs pass should check what's actually reading this before treating the directory as dead.

Also worth a process note for whoever runs Phase 1 for real again: launching a long-running background transfer over SSH needs systemd-run, not a plain nohup/setsid backgrounded shell — the latter got killed when the SSH session closed mid-transfer during this run. And rsync forks into multiple cooperating processes even for a purely local copy; don't mistake that for duplicate jobs and kill the "extras" — that aborts the whole transfer (recovered fine here, source was never touched, just cost a restart of the destination copy).

[Xore]

Phase 0 fix applied and verified live (2026-08-18)

Restored the executable bit on the live analysis/backup-honeypot.sh and manually triggered backup-honeypot.service to confirm the fix actually works, rather than just assuming it would.

Result: partial success, and it isolates the real risk.

• The config/secrets archive (stack-config-state.tar.gz, 19.6MB — every stack's compose file + .env references) now succeeds — confirmed by a real run, file present in a fresh /opt/backups/honeypot/<timestamp>/. This is the part that actually matters for "secrets have been getting lost" — it's fixed and verified, not just theorized.
• The Elasticsearch snapshot step in the same script now fails separately (curl: (22) The requested URL returned error: 400) — this is honeypot capture/analysis data, not secrets, and is a distinct, pre-existing bug uncovered by fixing the first one. Very likely related to the already-known uncommitted local hotfix on this same script (see the "Homeserver" section higher up this issue) — needs its own look at the ES snapshot request body / index list against current main, not assumed to be the same root cause as the permission bug. Tracking this as part of the existing Phase 0 "reconcile the local hotfix diff" item rather than opening a new issue.

The nightly timer will now succeed on the config/secrets side going forward; the ES-snapshot half stays broken until that's separately fixed.

[Xore]

Decision: the reinstall intentionally starts with a fresh Elasticsearch store — no alerts, payloads, or historical event data carried over from the old cluster. analysis/backup-honeypot.sh's ES-snapshot step (including the curl: 400 bug found while fixing the permission bug above) is therefore out of scope for this issue and not being fixed here. Updated the issue body and checklist accordingly — the config/secrets side of that script is what actually matters here and is already fixed and verified.