Infra: Add uptime monitoring, harden deploy script

- Add UptimeRobot + Pushover monitoring for getcmdr.com
- Build new image before stopping old container so failed builds don't take down the site
- Log deploy output to `/var/log/cmdr/deploy-website.log`
- Verify container is running after deploy
- Document journalctl access for `deploy-cmdr` user

Author: vdavid

docs/tooling/monitoring.md

@@ -0,0 +1,22 @@
+# Uptime monitoring
+
+We use [UptimeRobot](https://uptimerobot.com/) (free tier) to monitor `getcmdr.com`.
+
+## What's monitored
+
+- **HTTP monitor** on `https://getcmdr.com` — checks every 5 minutes from multiple regions.
+
+## Alerts
+
+- **Email**: Goes to the account owner's email (default alert contact).
+- **Pushover**: Push notifications to phone via [Pushover](https://pushover.net/). Configured as an alert contact in UptimeRobot.
+
+## Links
+
+- **Dashboard**: https://uptimerobot.com/dashboard (login required)
+- **Public status page**: https://stats.uptimerobot.com/MHKbVOfrcB
+
+## Notes
+
+- Free tier allows 50 monitors at 5-minute intervals.
+- If we need to monitor more endpoints later (for example, the license server), add them in the UptimeRobot dashboard.

infra/deploy-webhook/README.md

@@ -10,11 +10,36 @@ Webhook listener for GitHub Actions to trigger deployments without requiring SSH
 4. Webhook verifies the HMAC-SHA256 signature
 5. If valid, runs `deploy-website.sh`
 
+The deploy script builds the new Docker image **before** stopping the old container. If the
+build fails, the existing site stays up.
+
 ## Files
 
 - `hooks.json` — Webhook configuration (reads secret from env var)
 - `deploy-website.sh` — The actual deployment script
 
+## Logs
+
+Deploy output is appended to `/var/log/cmdr/deploy-website.log` on the server.
+
+To view recent deploy logs:
+
+```bash
+ssh hetzner "tail -50 /var/log/cmdr/deploy-website.log"
+```
+
+## Granting journalctl access to the deploy user
+
+The `deploy-cmdr` user can't read systemd journal by default. To fix that without
+giving sudo, add it to the `systemd-journal` group:
+
+```bash
+sudo usermod -aG systemd-journal deploy-cmdr
+```
+
+After that, `deploy-cmdr` can run `journalctl` to see service logs (read-only, no
+sudo needed).
+
 ## Security
 
 The webhook uses HMAC-SHA256 signature verification. Only requests signed with the correct

infra/deploy-webhook/deploy-website.sh

@@ -3,20 +3,42 @@ set -e
 
 # Deploy website script
 # Triggered by GitHub Actions via webhook after CI passes
+#
+# Safety: builds the new image BEFORE stopping the old container.
+# If the build fails, the existing site stays up.
 
+LOG_DIR="/var/log/cmdr"
+LOG_FILE="$LOG_DIR/deploy-website.log"
+mkdir -p "$LOG_DIR"
+
+# Redirect all output to both the log file and stdout
+exec > >(tee -a "$LOG_FILE") 2>&1
+
+echo ""
 echo "=== Starting website deployment ==="
-echo "Time: $(date)"
+echo "Time: $(date --iso-8601=seconds)"
 
 cd /opt/cmdr
 
 echo "Pulling latest code..."
 git pull origin main
 
-echo "Rebuilding website container..."
+echo "Building new image (old site stays up during build)..."
 cd apps/website
-docker compose down
 docker compose build --no-cache
+
+echo "Swapping containers..."
+docker compose down
 docker compose up -d
 
-echo "=== Deployment complete ==="
-echo "Time: $(date)"
+echo "Verifying container is running..."
+sleep 2
+if docker compose ps --status running | grep -q getcmdr-static; then
+    echo "=== Deployment succeeded ==="
+else
+    echo "=== ERROR: Container not running after deploy ==="
+    docker compose logs --tail 20
+    exit 1
+fi
+
+echo "Time: $(date --iso-8601=seconds)"