v1.8.1 — resume.sh bounces dnsmasq + CoreDNS + stuck pods

Fix

After v1.8.0, ./scripts/resume.sh could leave the cluster with broken cluster DNS on some resumes — most visibly argocd-repo-server stuck in CrashLoopBackOff because it couldn't helm pull from argoproj.github.io (lookup failed via CoreDNS → dnsmasq → upstream).

Three issues after a Colima daemon reload:

1. dnsmasq comes back listening on :53 but unresponsive. CoreDNS forwards to it for *.local and the forward chain stalls, cascading into every pod that needs external DNS (repo-server helm charts, cert-manager webhooks, etc.).
2. CoreDNS holds a stale upstream cache even after dnsmasq recovers.
3. Ghost pods from the pre-stop state land stuck Terminating or CrashLoopBackOff on the broken DNS.

resume.sh now:

• docker restart dnsmasq
• kubectl -n kube-system rollout restart deploy coredns + wait
• Force-remove Terminating ghosts (--force --grace-period=0)
• Delete CrashLoopBackOff pods so they reschedule against the now-healthy DNS

Observed end-to-end today: these three fixes take argocd-repo-server from 5 restarts + CrashLoopBackOff to 1/1 Running in ~40 seconds.

Full changelog

See CHANGELOG.md.

v1.8.0 — Offline-capable demo apps (bundled app-repos)

Highlights

Fresh clones on Linux, WSL2, macOS, and Windows now deploy Headscale and Industry 4.0 end-to-end — no manual Gitea repo creation, no one-off docker build steps. The Application manifests in gitops-repo/apps/ continue to point at platform/* Gitea repos, but the source for those repos now lives inside this repo under app-repos/ and gets mirrored into Gitea automatically on every bootstrap.

What's new

Added — app-repos/

Demo-app sources bundled in:

• app-repos/headscale/ — manifests + helper scripts + README
• app-repos/industry40/ — Dockerfile + index.html + manifests + build.sh + README

Added — bootstrap phase 07b

scripts/07b-mirror-app-repos.sh iterates every subdirectory of app-repos/, creates the matching Gitea repo under platform/ if missing, and force-pushes the local content. Idempotent.

Added — bootstrap phase 09c

scripts/09c-build-demo-images.sh runs app-repos/*/build.sh where present. Apps using public images (like Headscale) are silently skipped; industry40 builds + ctr imports its nginx image offline.

Changed — scripts/resume.sh

Per-node containerd health probe (docker exec <node> ctr version with timeout) recycles any k3d node whose containerd socket has gone unresponsive. Fixes the mysterious connection refused on /run/k3s/containerd/containerd.sock failures some nodes hit after a Docker-daemon reload — image builds and pulls that previously required a manual docker restart k3d-gitops-local-agent-X now self-heal on resume.

Adding your own demo app

Drop a directory under app-repos/:

app-repos/my-app/
├── manifests/            # k8s YAMLs — path: manifests in the Application
├── Dockerfile            # optional
├── build.sh              # optional — 09c runs it
└── README.md

Then add a matching Application in gitops-repo/apps/my-app.yaml pointing at http://gitea:3000/${GITEA_ORG}/my-app.git, a repo entry in gitops-repo/manifests/argocd/values.yaml, and bootstrap handles the rest.

Full changelog

See CHANGELOG.md.

v1.7.1 — User manual: Headscale + resume.sh sections

What's new

Portal user manual at http://portal.local/manual.html now has two new sections:

• 10. Headscale VPN (mobile demos) — first-run setup, phone registration flow (iOS + Android), subnet-router recipe for reaching 172.20.0.0/24 from mobile, cross-network options, admin commands.
• 11.1 Resume after stop — points users at the new ./scripts/resume.sh as the preferred restart flow.

TOC updated; subsequent section numbers shifted by one.

Full changelog

See CHANGELOG.md.

v1.7.0 — Headscale control plane + resume.sh

Highlights

Free, self-hosted VPN mesh for the landing zone via Headscale (BSD-3) — point the official Tailscale mobile apps at http://headscale.local, mesh your phone with the cluster, demo mobile apps against backends running on your laptop. No Tailscale account, no paid tier, no phoning home.

Also ships scripts/resume.sh — fills the gap left by v1.5.1 where the Colima daemon-DNS pin only ran during fresh bootstrap, never on resume.

What's new

Added — Headscale app (app-of-apps pattern)

New Gitea repo platform/headscale carries everything the app needs; the only reference from gitops-infra is gitops-repo/apps/headscale.yaml:

• ConfigMap'd config.yaml (embedded DERP STUN on UDP 3478, Magic DNS *.landingzone.local, SQLite persistence)
• PVC for the DB + private keys
• Deployment (headscale/headscale:0.26.1, ARM64-native) + Services (ClusterIP for API, LoadBalancer for DERP UDP) + Ingress at http://headscale.local
• Helper scripts: scripts/create-user.sh, scripts/create-preauth.sh
• README covering the registration flow for iOS / Android / laptop Tailscale clients, the subnet-router trick for reaching 172.20.0.0/24 from mobile, and the Cloudflare Tunnel path for cross-internet demos

Plus wiring in this repo:

• gitops-repo/apps/headscale.yaml — the Application manifest
• gitops-repo/manifests/argocd/values.yaml — new gitea-headscale repo credentials
• scripts/setup-hosts.sh + scripts/04-create-k3d-cluster.sh — headscale.local in the hosts + CoreDNS lists
• Portal card under Your Applications

Added — scripts/resume.sh

Idempotent resume flow after a colima stop / ./teardown.sh cycle:

1. Start Colima if not running.
2. Merge dns: ["8.8.8.8", "1.1.1.1"] into /etc/docker/daemon.json (fills the v1.5.1 gap where already-bootstrapped clusters never got the pin).
3. docker compose ... start the compose stack.
4. k3d cluster start gitops-local, wait for nodes Ready.
5. Re-patch each k3d node's /etc/resolv.conf (belt-and-braces — daemon.json catches new containers, node-level patch catches existing ones).
6. Re-patch CoreDNS NodeHosts for gitea.local.
7. Clear ghost Pending pods from the pre-stop state.

Using Headscale

After the app syncs:

# Create a user + pre-auth key (free unlimited)
bash scripts/create-user.sh alice
bash scripts/create-preauth.sh alice --reusable --expiry 168h

# On your phone: install the Tailscale app,
# → Settings → Change server → http://headscale.local
# → Sign in with auth key → paste the preauth key
# → Phone now has a 100.x.x.x Tailscale IP

To reach portal.local / gitea.local:3000 / etc. from the phone, register the laptop as a subnet router (see the app repo's README for the tailscale up --advertise-routes recipe).

For phones on cellular (outside your LAN), Headscale needs a public URL. Deploy Cloudflare Tunnel as a sibling Application — still 100% free for dev.

Upgrading an existing cluster

# If the cluster is stopped, use the new resume script:
./scripts/resume.sh

# Then pull + re-push gitops-infra to get Headscale wired in:
git pull
bash scripts/07-push-gitops-repo.sh

# Flush argocd-repo-server cache and let ArgoCD sync:
kubectl -n argocd rollout restart deploy argocd-repo-server

Full changelog

See CHANGELOG.md.

v1.6.0 — industry40 demo app via app-of-apps

Highlights

First end-to-end demonstration of the canonical app-of-apps pattern on the landing zone — a new demo application (`industry40`) serving an HTML history of the four industrial revolutions at http://industry40.local.

What the pattern looks like in practice

```
gitops-infra (this repo)
├── apps/
│ ├── root-app.yaml ← one Application that syncs apps/*
│ ├── industry40.yaml ← points at platform/industry40
│ └── ...
└── manifests/
└── argocd/values.yaml ← declares the platform/industry40 repo
so argocd-repo-server has creds

platform/industry40 (new Gitea repo — NOT in this tree)
├── Dockerfile + index.html ← the app itself
├── build.sh ← docker build + ctr import
├── manifests/ ← namespace, deployment, service, ingress
└── README.md
```

No part of the application — not the Dockerfile, not the HTML, not the Kubernetes manifests — lives in gitops-infra. The only cross-reference is the `Application` CR at `apps/industry40.yaml`.

What's new

Added

• `industry40` application (served at http://industry40.local) — nginx-delivered single-file HTML covering Industry 1.0 through 4.0 (and a note on 5.0), with timeline, Nine Pillars grid, and references. Self-contained, themed to match the portal with dark/light toggle.
• Portal card under Your Applications linking to the new app.
• `industry40.local` baked into `scripts/setup-hosts.sh` and the CoreDNS NodeHosts patch (cold bootstraps get the hostname out of the box).

Changed

• `AppProject dev` destinations simplified from a narrow per-namespace list to a single `namespace: '*'`. The old list blocked any new app in a namespace that wasn't pre-declared (`InvalidSpecError`). This is a dev landing zone, not a multi-tenant production cluster — restrict later if you shard users.

To deploy on an existing cluster after upgrading

```bash
git pull

Re-render gitops-infra (adds the new Application + repo credentials)

bash scripts/07-push-gitops-repo.sh

Flush argocd-repo-server so the project + repos refresh immediately

kubectl -n argocd rollout restart deploy argocd-repo-server
```

Then create the `platform/industry40` Gitea repo, push the app contents (see README at platform/industry40), run `bash build.sh` once to build + import the image, and the root app-of-apps picks it up.

Full changelog

See CHANGELOG.md.

v1.5.1 — Colima Docker daemon DNS pinning

Fix

Pod restarts on Colima-hosted k3d no longer trigger cascading failures.

Problem

Colima's default DNS (`192.168.5.2` on the lima gateway) is unreachable from inside nested Docker networks. `scripts/04-create-k3d-cluster.sh` already patched `/etc/resolv.conf` on each k3d node at creation — but that patch is container-local and Docker wipes it on any container restart.

If you `docker restart k3d-gitops-local-agent-1` (or Colima restarts the daemon for any reason), every pod on that node fails with `ImagePullBackOff` (can't resolve registry.k8s.io) or "connection refused to 10.43.0.1:443" (kube-proxy iptables go stale). Pods like kube-state-metrics, cert-manager-cainjector, or metallb-controller start crashlooping and generate hundreds of events per hour.

Solution

`scripts/00-prerequisites.sh` now writes `/etc/docker/daemon.json` inside the Colima VM during the prerequisites phase:

```json
{"dns": ["8.8.8.8", "1.1.1.1"]}
```

and reloads the Docker daemon. Every new container (including k3d nodes after a restart) inherits public DNS automatically. The k3d-node-level patch in `04-create-k3d-cluster.sh` stays as belt-and-braces for already-running clusters.

Also

• New troubleshooting section in the portal user manual (`http://portal.local/manual.html\`) covers the "stale kube-proxy iptables after node restart" failure with the copy-paste recovery recipe.

Recovering an already-affected cluster

```bash

Restart the affected k3d node (non-destructive; pods reschedule)

docker restart k3d-gitops-local-agent-1

Re-patch its DNS (Docker wipes /etc/resolv.conf on restart)

docker exec k3d-gitops-local-agent-1 sh -c \
'printf "nameserver 8.8.8.8\nnameserver 1.1.1.1\noptions ndots:0\n" > /etc/resolv.conf'

Delete pods that crashlooped — they reschedule cleanly

kubectl delete pod -A --field-selector=status.phase!=Running,status.phase!=Succeeded
```

Or, from v1.5.1 onwards, run `bash scripts/00-prerequisites.sh` once and every subsequent container restart will pick up the daemon-level DNS.

Full changelog

See CHANGELOG.md.

v1.5.0 — User manual on the landing portal

Highlights

After bootstrap, the landing portal now links to a full user manual served alongside portal.local — no GitHub round-trip needed.

What's new

Added

• http://portal.local/manual.html — a self-contained 16-section user manual styled to match the portal's dark/light theme. Includes a sticky TOC sidebar with scroll-spy active highlighting, monospace code blocks, callout boxes, and tables. Sections:
  • Overview — what the platform is, shortest path for new users
  • Prerequisites — per-OS tables (Linux / WSL2, macOS + Colima sizing tip, Windows 11)
  • Quick start — one-command bootstrap for every host, zero .env edits
  • Architecture — diagram of the docker network + k3d cluster
  • Bootstrap phases — the full 16-phase table with what each script does
  • Service URLs & credentials — canonical reference + one-liner to dump .env
  • Vaultwarden import — web-UI walkthrough + bw CLI commands
  • Single sign-on (Keycloak) — test user, login flow, adding users
  • Deploy your own application — short-form recipe (with link to docs/ADD_YOUR_APP.md)
  • Lifecycle — restart, teardown, backup & restore
  • Observability — Grafana dashboards, Prometheus retention, Loki LogQL
  • Security — NetworkPolicies, pre-commit hooks, Sealed Secrets
  • AI agent federation — agent matrix + skill list
  • Platform-specific notes — Colima DNS patch, inotify tuning, .local mDNS, MSYS path translation
  • Troubleshooting — six common issues with copy-paste fixes
  • Extending further — domain agents, Grafana dashboards, webhooks, pre-commit hooks in multiple repos
• Documentation card on the portal home page linking to the manual.

Changed

• gitops-repo/manifests/portal/src/Dockerfile copies manual.html into the nginx document root.

Upgrading existing clusters

After pulling v1.5.0:

git pull
bash scripts/07-push-gitops-repo.sh    # pushes new Dockerfile + assets to Gitea
bash scripts/09b-build-portal.sh        # rebuilds portal image + imports into k3d
kubectl -n portal rollout restart deploy portal

Then open http://portal.local — the new Documentation section with the User Manual card will appear.

Full changelog

See CHANGELOG.md.

v1.4.2 — Grafana admin password mismatch fix

Fix

The Grafana admin password stored in the monitoring-grafana Kubernetes Secret was the literal string `CHANGE_ME_from_env` instead of the value from `.env` — so any attempt to log into Grafana with the documented password failed.

Three gitops-repo manifests held a literal `CHANGE_ME_from_env` where a `${VAR}` placeholder belonged, and `GRAFANA_ADMIN_PASSWORD` was missing from the envsubst whitelist in `07-push-gitops-repo.sh`. Fixed:

• `gitops-repo/manifests/monitoring/values.yaml` → `${GRAFANA_ADMIN_PASSWORD}`
• `gitops-repo/apps/monitoring.yaml` → `${GRAFANA_ADMIN_PASSWORD}`
• `gitops-repo/.gitea/workflows/validate.yaml` → `${GITEA_ADMIN_USER}:${GITEA_ADMIN_PASSWORD}`
• `scripts/07-push-gitops-repo.sh` ENVSUBST_VARS → adds `${GRAFANA_ADMIN_PASSWORD}`
• `scripts/10-configure-oidc.sh` ENVSUBST_VARS → adds `${GRAFANA_ADMIN_PASSWORD}`

Upgrading existing clusters

ArgoCD's repo-server caches helm manifest renders for ~3 minutes. After the upgrade push, flush the cache:

```bash
kubectl -n argocd rollout restart deploy argocd-repo-server
```

Then let ArgoCD re-sync the monitoring Application and restart Grafana to pick up the new password:

```bash
kubectl annotate application monitoring -n argocd argocd.argoproj.io/refresh=hard --overwrite
kubectl -n monitoring rollout restart deploy monitoring-grafana
```

Full changelog

See CHANGELOG.md.

v1.4.1 — Vaultwarden import helper

Highlights

New bootstrap phase installs the Bitwarden CLI and generates a ready-to-import JSON of every platform credential — so right after bootstrap finishes, you can load the whole stack into Vaultwarden with a few clicks.

What's new

Added

• Phase 11 — scripts/11-vaultwarden-import.sh
  Runs automatically after the Keycloak OIDC phase:

  1. Waits for Vaultwarden at https://localhost:8443.

  2. Installs bw via the pre-built zip from vault.bitwarden.com (single native binary, ~50 MB, installs in seconds). Falls back to brew install bitwarden-cli if the download fails.

     • Why not brew first? The brew formula compiles from source on Apple Silicon and can take 10+ minutes.

  3. Renders ./vaultwarden-import.json from the current .env:

     Kind	Items
     Logins (8)	ArgoCD, Gitea, Keycloak admin, Keycloak dev SSO user, Grafana, Prometheus, Portal, Vaultwarden admin panel
     Secure notes (3)	Gitea PostgreSQL creds, Keycloak PostgreSQL creds, OIDC client IDs + secrets

  4. Prints import instructions for both the web UI and the bw CLI.

• .gitignore now blocks vaultwarden-import*.json and bitwarden-export*.json from accidental commits.

Changed

• scripts/bootstrap.sh — phase 11 wired in between 10-configure-oidc.sh and the completion banner; the banner directs the user to the import file.

How to use

After ./bootstrap.sh finishes:

1. Open  https://localhost:8443  →  accept self-signed cert
2. Create account — set a strong master password
3. Tools  →  Import data
4. File format:  Bitwarden (json)
5. Select file: vaultwarden-import.json
6. Import data

# Then delete the plaintext:
rm vaultwarden-import.json

Or use the CLI:

export NODE_TLS_REJECT_UNAUTHORIZED=0
bw config server https://localhost:8443
bw login <email-you-registered>
export BW_SESSION=\$(bw unlock --raw)
bw import bitwardenjson vaultwarden-import.json
rm vaultwarden-import.json

Full changelog

See CHANGELOG.md.

v1.4.0 — macOS / Colima support + hands-off bootstrap

Highlights

Fully hands-off bootstrap on macOS (Apple Silicon + Intel) via Colima. git clone && ./bootstrap.sh now succeeds on a fresh machine with no manual .env edits, no TTY required for sudo (GUI admin prompt fallback), and no hard-coded secrets leaking into the landing-zone git history.

The BankOffer + CareerForge test workloads have been extracted to the sibling repo my-testing-apps so this repo is a clean base platform.

Quick start

macOS (Intel or Apple Silicon)

brew install colima docker docker-compose git
colima start --cpu 6 --memory 10 --disk 60
git clone https://github.com/bogdandragosvasile/gitops-landing-zone.git
cd gitops-landing-zone
./bootstrap.sh

Linux / WSL2

git clone https://github.com/bogdandragosvasile/gitops-landing-zone.git
cd gitops-landing-zone
./bootstrap.sh

Windows 11

Same as before — from an elevated PowerShell: .\bootstrap.ps1.

What's new

Added

• macOS + Apple Silicon support via Colima, with platform/arch auto-detection.
• bootstrap.sh / teardown.sh top-level unix entry points.
• scripts/gen-env.sh — renders .env with URL-safe random secrets on first bootstrap.
• scripts/01b-ensure-certs.sh — auto-generates Vaultwarden TLS certs.
• Colima VM sysctl tuning (fs.inotify.max_user_watches=524288) to prevent Promtail too many open files.
• k3d-node DNS patch on macOS (Colima's internal resolver is unreachable from the nested bridge).
• allow-intra-keycloak-http and allow-traefik-prometheus-ingress NetworkPolicies.
• Bcrypt-based offline ArgoCD admin password rotation.
• docs/ADD_YOUR_APP.md + gitops-repo/apps-examples/my-app.yaml.example.
• README "Platform-specific notes" + "Tested platforms" sections.

Changed

• BankOffer + CareerForge extracted to my-testing-apps.
• scripts/07-push-gitops-repo.sh renders into a temp dir, leaving the source tree as ${VAR} templates (secrets no longer leak into git history).
• scripts/10-configure-oidc.sh explicitly applies the keycloak-configure Job with the proper envsubst whitelist instead of hoping the PostSync hook runs.
• scripts/02-start-gitea.sh also brings up vaultwarden + dnsmasq.
• scripts/setup-hosts.sh falls back to osascript with administrator privileges on macOS when no TTY.
• Fixed metallb allow-apiserver-webhook pod-selector labels (matched nothing on the current chart → every IPAddressPool apply returned 502).

Fixed

• Password auth failures when .env passwords contained + (form-urlencoded decoded as space).
• keycloak-postgres-secret hardcoded password ignoring .env regeneration.
• Top-level bootstrap.sh no longer hard-fails when .env is missing — it triggers gen-env.sh automatically.
• scripts/09b-build-portal.sh import bug (ctr -n k8s.io images import + correct k3d node filter).

Tested

Host	Status
macOS 14 + Colima (M1 Pro, 16 GB)	✅ End-to-end verified
Ubuntu 22.04 + Docker Engine	✅ Supported (same scripts)
WSL2 + Docker Desktop	✅ Supported (same scripts)
Windows 11 + Docker Desktop	✅ Supported (bootstrap.ps1)

Full changelog

See CHANGELOG.md.