Auto-PR: fix(ci): align drifted chart versions; tighten CHANGELOG consistency check#21
Merged
Merged
Conversation
Merged
23 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aligns Helm chart metadata with the latest per-service changelog entries and tightens CI validation so chart version drift is caught against the newest changelog release entry rather than any historical occurrence.
Changes:
- Updates WordPress, Vaultwarden, and Nextcloud chart versions to match their latest
CHANGELOG.mdheaders. - Strengthens
.github/workflows/validation.ymlto compareChart.yamlversion:against the first## [X.Y.Z]changelog entry. - Adds clearer failure handling when a changelog has no parseable release header.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
wordpress/helm/Chart.yaml |
Aligns chart version with latest WordPress changelog entry. |
vaultwarden/helm/Chart.yaml |
Aligns chart version with latest Vaultwarden changelog entry. |
nextcloud/helm/Chart.yaml |
Aligns chart version with latest Nextcloud changelog entry. |
.github/workflows/validation.yml |
Tightens version consistency validation for all top-level Helm charts. |
romandidomizio
previously approved these changes
May 13, 2026
Member
romandidomizio
left a comment
romandidomizio
left a comment
There was a problem hiding this comment.
PR 21 Review — fix(ci): Version Consistency Alignment (fix/nik-version-consistency → main)
Main merged in. Changes reviewed across: .github/workflows/validation.yml, nextcloud/helm/Chart.yaml, vaultwarden/helm/Chart.yaml, wordpress/helm/Chart.yaml.
✅ Changes Look Correct
Version consistency check improvement (validation.yml)
The original grep -qi "$chart_version" was a loose substring match that could pass silently when a chart version still appeared in older CHANGELOG entries despite the latest entry having drifted. The new logic correctly extracts the top-most ## [X.Y.Z] header and does a strict equality check. The explanatory comments make the intent clear. Well done.
Chart version bumps
All three chart versions now align with their respective CHANGELOG latest entries:
nextcloud:1.0.0→1.1.0(matches## [1.1.0] - 2025-11-04) ✅vaultwarden:1.0.0→1.3.1(matches## [1.3.1] - 2025-10-27) ✅wordpress:3.2.6→3.3.8(matches## [3.3.8] - 2025-11-07) ✅
✅ Copilot Comment Resolved
Line 204: long error message exceeded 200-char yamllint threshold (Copilot)
Fixed in this review pass — the error message has been split across msg= assignments (commit fcac559). The fix preserves the full diagnostic text and resolves the yamllint line-length warning.
📋 Pre-existing Issue (not introduced by this PR)
nextcloud/helm/Chart.yaml — appVersion: "latest"
This was already "latest" in main before this PR. It's a pre-existing floating appVersion that should be pinned to a specific Nextcloud release in a follow-up. Not a blocker for this PR.
Summary
Clean, purposeful PR. The validation logic improvement prevents a whole class of false-passing version drift (where the old version still appeared in older CHANGELOG entries). All chart versions are now correctly aligned. The one Copilot comment has been addressed in this review. Approving.
ncimino
approved these changes
May 19, 2026
…check The version-consistency check in .github/workflows/validation.yml used `grep -qi $chart_version $changelog_file` (presence-anywhere), which silently passed even when Chart.yaml drifted away from the latest CHANGELOG entry. Concrete case discovered during PR #19 review: wordpress/helm/Chart.yaml was 3.2.6 while wordpress/CHANGELOG.md's top entry was [3.3.8] — the older 3.2.6 string still appeared in past entries, so the check kept passing. Tighten the check to compare Chart.yaml's `version:` against the first `## [X.Y.Z]` header in CHANGELOG.md (the Keep-a-Changelog newest-on-top convention). Three charts had pre-existing drift and would fail the new check on main, so align them in this commit: nextcloud/helm/Chart.yaml 1.0.0 → 1.1.0 vaultwarden/helm/Chart.yaml 1.0.0 → 1.3.1 wordpress/helm/Chart.yaml 3.2.6 → 3.3.8 These are metadata-only bumps: Chart.yaml now matches the version each chart's own CHANGELOG already declares as latest. No template or values changes. Not addressed in this commit (separate decisions needed): - WeOwnVer format inconsistency between docs/VERSIONING_WEOWNVER.md (SEASON.MONTH.WEEK.ITERATION) and validation.yml's `versioning` job (SEASON.WEEK.DAY.VERSION). None of the current chart versions cleanly fit either schema. - The `versioning` job only validates the first Chart.yaml found alphabetically (`*/helm/Chart.yaml | head -1` on line 212), not all of them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ncimino
force-pushed
the
fix/nik-version-consistency
branch
from
1961aad to
5cb0bf2
Compare
ncimino
approved these changes
May 19, 2026
ncimino
added a commit
that referenced
this pull request
May 22, 2026
Resolves the 3 failing CI jobs and all 30 inline review comments on PR #24 (and the predecessor PR #14, whose comments were ported via the branch rename `dra` → `feature/shahid-velero-restic`). CI failures fixed ----------------- * Helm Template Validation — chart no longer declares an upstream `velero` subchart dependency (the chart provides its OWN Velero + Restic templates; the two would collide and `helm template` could not run because `helm dependency build` is not invoked in CI). * Helm Lint — removed the duplicate `cluster-backup.restoreName` template definition in `_helpers.tpl`. * Documentation Validation / Markdown Lint — `cluster-backup/README.md` now satisfies MD031/MD022/MD032 (blank lines around fences / headings / lists). Added `cluster-backup/CHANGELOG.md` with a `[1.0.0] - 2026-05-22` entry that matches `Chart.yaml`'s `version: 1.0.0` — required by the tightened version-consistency check landed in #21. * trailing-whitespace / end-of-file-fixer — stripped trailing whitespace from every cluster-backup yaml + shell file and ensured every file ends with `\n`. Security & correctness (per Copilot / @romandidomizio review) ------------------------------------------------------------ * `automountServiceAccountToken: false` flipped to `true` on BOTH the Velero server and the Restic node-agent ServiceAccounts. Both controllers MUST call the Kubernetes API (Velero for cross-namespace resource enumeration / Backup-Restore CRDs; Restic for PodVolumeBackup / PodVolumeRestore / ResticRepository / coordination leases). Disabling token automount without a projected-token workaround silently broke the controllers. * Removed `helm/templates/velero-secret.yaml`. The chart was rendering a placeholder Secret named `cluster-backup-cloud-credentials` which collided with the real Secret that `deploy.sh` creates out-of-band (`helm upgrade` would fail with "resource already exists" on every re-install). The Secret is now solely owned by `deploy.sh`. Removed the corresponding `checksum/secret` annotation from the Velero Deployment and Restic DaemonSet. * Added `helm/templates/velero-service.yaml`. The chart previously shipped a `ServiceMonitor` selecting a Service that did not exist, and `NOTES.txt` instructed users to `kubectl port-forward svc/<fullname>-velero` — also missing. Service is `ClusterIP`, metrics-only. * `helm/templates/networkpolicy.yaml` egress no longer uses `to: []` for any rule. DNS is constrained to the kube-system namespace; external S3 traffic is allowed to `0.0.0.0/0` MINUS RFC1918, link-local, and loopback ranges (so restic / velero can never accidentally exfiltrate to in-cluster IPs over an S3-shaped path); Kubernetes API access is constrained to the `default` namespace ClusterIP. * `helm/templates/restic-daemonset.yaml` no longer mounts host `/` via `hostPath`. Restic only needs read access to `/var/lib/kubelet/pods` (the standard kubelet pod-volume root), so that's the only host mount that remains. Mounting host-root was incompatible with Pod Security `restricted` and not required for restic operation. * `helm/templates/backup-schedules.yaml` passes a full `dict "Chart" $.Chart "Release" $.Release "Values" $.Values "Template" $.Template ...` context into the schedule-naming and schedule-config helpers, removing any ambiguity about what `cluster-backup.fullname` and `cluster-backup.labels` resolve to when called from within a `range` loop body. * `helm/templates/_helpers.tpl`: simplified `cluster-backup.serviceAccountName` and `cluster-backup.resticServiceAccountName` to return deterministic names. The previous form gated on `.Values.velero.{server,restic} .serviceAccount.create`, which is not a path that exists in `values.yaml` — it was a remnant of the now-removed `velero` subchart pattern. The helpers always returned `"default"`, which conflicted with the ClusterRoleBinding subject the chart actually installed; Velero would have been denied every API call. * `helm/templates/NOTES.txt` rewritten: `kubectl create backup …` / `kubectl create restore …` are NOT valid kubectl subcommands for Velero CRDs and were misleading users into thinking the controller was broken when in reality they were running a non-existent kubectl verb. NOTES now uses the Velero CLI (`velero backup get`, `velero schedule get`, etc.) and points readers at the install docs. Same fix applied across README.md, deploy.sh, verify.sh, test-local.sh, and (post-deployment) usage messages. Script hardening ---------------- * `deploy.sh`: - Switched to `#!/usr/bin/env bash` + `set -euo pipefail`. - All temporary files now live in a single `mktemp -d`-created directory with mode 0700, cleaned up via an `EXIT|INT|TERM` trap. The previous version wrote `/tmp/s3-credentials` and `/tmp/cluster-backup-values.yaml` at fixed paths — world-readable on some hosts and racy under concurrent runs. - S3 secret-key prompt uses `read -rs` (no echo). The credential is piped into `kubectl create secret … --from-file=cloud=/dev/stdin --dry-run=client -o yaml | kubectl apply -f -`, so the secret never lives on disk and never appears in argv (`ps`). - All configuration accepts env-var inputs (`TENANT`, `CLUSTER`, `ENVIRONMENT`, `S3_*`) so the script is usable in CI/automation without an interactive TTY. * `test-local.sh`: - MinIO image pinned (`minio/minio:RELEASE.2024-08-17T01-24-54Z`). Was `minio/minio:latest`, which made local test runs non-reproducible and exposed them to silent upstream breaking changes. - MinIO credentials no longer hardcoded as `minioadmin / minioadmin` in the manifest. Access key defaults to `localdev-access`; secret key is a freshly-generated 24-char random value per run (overridable via `TEST_MINIO_{ACCESS,SECRET}_KEY` env vars). Rendered into the Secret over a stdin pipe — same no-argv-leak pattern as deploy.sh. - Service now correctly exposes BOTH the S3 API (NodePort 30000) AND the web console (NodePort 30001). The original Service mapped only 9000 to NodePort 30000 yet the "MinIO Console" line pointed at 30000 — that's the S3 API, not the console. Web console address is now `http://localhost:30001`. - Replaced fixed `sleep 30` waits with `wait_for_velero_phase` which polls `.status.phase` of the Backup / Restore CR and emits `kubectl describe` + `velero backup logs` diagnostics on timeout. - Uses the Velero CLI for backup / restore creation. * `verify.sh`: - Same shebang + strict-mode + Velero-CLI updates as deploy.sh. - `test_backup_creation` polls phase with diagnostics instead of blind-sleeping; also skips gracefully if the Velero CLI isn't installed locally (verify is run on hosts that may not have it). * `create-tenant-cluster.sh`: - All upstream components are version-pinned via env-var defaults: `DOKS_K8S_VERSION`, `INGRESS_NGINX_VERSION`, `CERT_MANAGER_VERSION`, `METRICS_SERVER_VERSION`. The previous version used `doctl kubernetes cluster create … --version latest` and `metrics-server/releases/latest/`, making provisioning silently non-reproducible. - Team-member namespace is normalized to RFC1123 (`AnnaF` → `annaf`, etc.) before being passed to `kubectl create namespace`. K8s rejects uppercase-containing namespace names with an opaque error, and the previous version of this script would silently fail at `create_team_member_namespace` for any contributor whose handle wasn't already lowercase-alphanumeric. - `create_team_member_access` no longer reads `.secrets[0].name` to harvest a long-lived ServiceAccount token (the BoundServiceAccount- Token feature in K8s ≥1.24 means that field is empty on modern clusters, so the function was returning an empty kubeconfig). Switched to `kubectl create token` (TokenRequest API) with an explicit 24h duration. Kubeconfig is written with `umask 077` so the file containing the bearer token is mode 0600. Not in scope for this commit ---------------------------- * The Restic DaemonSet still ships with a `SYS_ADMIN` capability add and a non-root securityContext (uid 65534). Real-world restic operation typically requires either root or a relaxed podSecurityContext to read pod volumes owned by arbitrary uids; tightening this further requires a per-environment decision about the Pod Security Standard for the `velero` namespace. Flagged in `cluster-backup/CHANGELOG.md` under "Notes for operators". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
23 tasks
ncimino
added a commit
that referenced
this pull request
May 25, 2026
Resolves the 3 failing CI jobs and all 30 inline review comments on PR #24 (and the predecessor PR #14, whose comments were ported via the branch rename `dra` → `feature/shahid-velero-restic`). CI failures fixed ----------------- * Helm Template Validation — chart no longer declares an upstream `velero` subchart dependency (the chart provides its OWN Velero + Restic templates; the two would collide and `helm template` could not run because `helm dependency build` is not invoked in CI). * Helm Lint — removed the duplicate `cluster-backup.restoreName` template definition in `_helpers.tpl`. * Documentation Validation / Markdown Lint — `cluster-backup/README.md` now satisfies MD031/MD022/MD032 (blank lines around fences / headings / lists). Added `cluster-backup/CHANGELOG.md` with a `[1.0.0] - 2026-05-22` entry that matches `Chart.yaml`'s `version: 1.0.0` — required by the tightened version-consistency check landed in #21. * trailing-whitespace / end-of-file-fixer — stripped trailing whitespace from every cluster-backup yaml + shell file and ensured every file ends with `\n`. Security & correctness (per Copilot / @romandidomizio review) ------------------------------------------------------------ * `automountServiceAccountToken: false` flipped to `true` on BOTH the Velero server and the Restic node-agent ServiceAccounts. Both controllers MUST call the Kubernetes API (Velero for cross-namespace resource enumeration / Backup-Restore CRDs; Restic for PodVolumeBackup / PodVolumeRestore / ResticRepository / coordination leases). Disabling token automount without a projected-token workaround silently broke the controllers. * Removed `helm/templates/velero-secret.yaml`. The chart was rendering a placeholder Secret named `cluster-backup-cloud-credentials` which collided with the real Secret that `deploy.sh` creates out-of-band (`helm upgrade` would fail with "resource already exists" on every re-install). The Secret is now solely owned by `deploy.sh`. Removed the corresponding `checksum/secret` annotation from the Velero Deployment and Restic DaemonSet. * Added `helm/templates/velero-service.yaml`. The chart previously shipped a `ServiceMonitor` selecting a Service that did not exist, and `NOTES.txt` instructed users to `kubectl port-forward svc/<fullname>-velero` — also missing. Service is `ClusterIP`, metrics-only. * `helm/templates/networkpolicy.yaml` egress no longer uses `to: []` for any rule. DNS is constrained to the kube-system namespace; external S3 traffic is allowed to `0.0.0.0/0` MINUS RFC1918, link-local, and loopback ranges (so restic / velero can never accidentally exfiltrate to in-cluster IPs over an S3-shaped path); Kubernetes API access is constrained to the `default` namespace ClusterIP. * `helm/templates/restic-daemonset.yaml` no longer mounts host `/` via `hostPath`. Restic only needs read access to `/var/lib/kubelet/pods` (the standard kubelet pod-volume root), so that's the only host mount that remains. Mounting host-root was incompatible with Pod Security `restricted` and not required for restic operation. * `helm/templates/backup-schedules.yaml` passes a full `dict "Chart" $.Chart "Release" $.Release "Values" $.Values "Template" $.Template ...` context into the schedule-naming and schedule-config helpers, removing any ambiguity about what `cluster-backup.fullname` and `cluster-backup.labels` resolve to when called from within a `range` loop body. * `helm/templates/_helpers.tpl`: simplified `cluster-backup.serviceAccountName` and `cluster-backup.resticServiceAccountName` to return deterministic names. The previous form gated on `.Values.velero.{server,restic} .serviceAccount.create`, which is not a path that exists in `values.yaml` — it was a remnant of the now-removed `velero` subchart pattern. The helpers always returned `"default"`, which conflicted with the ClusterRoleBinding subject the chart actually installed; Velero would have been denied every API call. * `helm/templates/NOTES.txt` rewritten: `kubectl create backup …` / `kubectl create restore …` are NOT valid kubectl subcommands for Velero CRDs and were misleading users into thinking the controller was broken when in reality they were running a non-existent kubectl verb. NOTES now uses the Velero CLI (`velero backup get`, `velero schedule get`, etc.) and points readers at the install docs. Same fix applied across README.md, deploy.sh, verify.sh, test-local.sh, and (post-deployment) usage messages. Script hardening ---------------- * `deploy.sh`: - Switched to `#!/usr/bin/env bash` + `set -euo pipefail`. - All temporary files now live in a single `mktemp -d`-created directory with mode 0700, cleaned up via an `EXIT|INT|TERM` trap. The previous version wrote `/tmp/s3-credentials` and `/tmp/cluster-backup-values.yaml` at fixed paths — world-readable on some hosts and racy under concurrent runs. - S3 secret-key prompt uses `read -rs` (no echo). The credential is piped into `kubectl create secret … --from-file=cloud=/dev/stdin --dry-run=client -o yaml | kubectl apply -f -`, so the secret never lives on disk and never appears in argv (`ps`). - All configuration accepts env-var inputs (`TENANT`, `CLUSTER`, `ENVIRONMENT`, `S3_*`) so the script is usable in CI/automation without an interactive TTY. * `test-local.sh`: - MinIO image pinned (`minio/minio:RELEASE.2024-08-17T01-24-54Z`). Was `minio/minio:latest`, which made local test runs non-reproducible and exposed them to silent upstream breaking changes. - MinIO credentials no longer hardcoded as `minioadmin / minioadmin` in the manifest. Access key defaults to `localdev-access`; secret key is a freshly-generated 24-char random value per run (overridable via `TEST_MINIO_{ACCESS,SECRET}_KEY` env vars). Rendered into the Secret over a stdin pipe — same no-argv-leak pattern as deploy.sh. - Service now correctly exposes BOTH the S3 API (NodePort 30000) AND the web console (NodePort 30001). The original Service mapped only 9000 to NodePort 30000 yet the "MinIO Console" line pointed at 30000 — that's the S3 API, not the console. Web console address is now `http://localhost:30001`. - Replaced fixed `sleep 30` waits with `wait_for_velero_phase` which polls `.status.phase` of the Backup / Restore CR and emits `kubectl describe` + `velero backup logs` diagnostics on timeout. - Uses the Velero CLI for backup / restore creation. * `verify.sh`: - Same shebang + strict-mode + Velero-CLI updates as deploy.sh. - `test_backup_creation` polls phase with diagnostics instead of blind-sleeping; also skips gracefully if the Velero CLI isn't installed locally (verify is run on hosts that may not have it). * `create-tenant-cluster.sh`: - All upstream components are version-pinned via env-var defaults: `DOKS_K8S_VERSION`, `INGRESS_NGINX_VERSION`, `CERT_MANAGER_VERSION`, `METRICS_SERVER_VERSION`. The previous version used `doctl kubernetes cluster create … --version latest` and `metrics-server/releases/latest/`, making provisioning silently non-reproducible. - Team-member namespace is normalized to RFC1123 (`AnnaF` → `annaf`, etc.) before being passed to `kubectl create namespace`. K8s rejects uppercase-containing namespace names with an opaque error, and the previous version of this script would silently fail at `create_team_member_namespace` for any contributor whose handle wasn't already lowercase-alphanumeric. - `create_team_member_access` no longer reads `.secrets[0].name` to harvest a long-lived ServiceAccount token (the BoundServiceAccount- Token feature in K8s ≥1.24 means that field is empty on modern clusters, so the function was returning an empty kubeconfig). Switched to `kubectl create token` (TokenRequest API) with an explicit 24h duration. Kubeconfig is written with `umask 077` so the file containing the bearer token is mode 0600. Not in scope for this commit ---------------------------- * The Restic DaemonSet still ships with a `SYS_ADMIN` capability add and a non-root securityContext (uid 65534). Real-world restic operation typically requires either root or a relaxed podSecurityContext to read pod volumes owned by arbitrary uids; tightening this further requires a per-environment decision about the Pod Security Standard for the `velero` namespace. Flagged in `cluster-backup/CHANGELOG.md` under "Notes for operators". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ncimino
added a commit
that referenced
this pull request
May 27, 2026
… DO droplets (#27) * feat(sandbox-docker): add copier template for AIO Sandbox on DO droplets Adds a new copier project for deploying agent-infra/sandbox (browser, shell, filesystem, VSCode, Jupyter, MCP servers in one container) on a DigitalOcean droplet. Mirrors the keycloak-docker / anythingllm-docker pattern: Caddy reverse proxy, Infisical runtime secret injection, OpenTofu droplet + firewall + reserved IP, skinny volume backups with GFS retention. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Auto-PR: Merge branch 'feature/wordpress-docker-copier-template' into feature/wp-hardening-shd (#19) * feat: codify PHP limits and block .user.ini per Task D152 & #264 * chore(helm): implement multi-site values strategy for burnedout and ptoken * chore: bump helm chart version to 3.2.7 * fix: resolve trivy security scan and linting issues * udated version bump * fix: resolve yamllint line endings and comment indentation * fix: resolve yamllint line endings * docs: apply markdownlint autofixes for PR #15 CI - ADR-004: add blank line before bulleted list (MD032) - workflows/README.md: switch *emphasis* to _emphasis_ for style consistency (MD049) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(wordpress/helm): address PR #15 review items 10-12 - ingress.yaml: parameterize spec.tls[0].secretName from .Values.ingress.tls[0].secretName so per-site overrides (burnedout-tls, ptoken-tls) actually take effect. Falls back to "wordpress-tls" when .Values.ingress.tls is a map or unset (preserves default and TLS hardening map). - php-config-configmap.yaml: gate "auto_prepend_file = wordfence-waf.php" behind .Values.wordpress.wordfence.enabled (default false) to avoid PHP warnings when the plugin is not installed. - ingress.yaml: gate the file-blocking server-snippet annotation behind .Values.ingress.serverSnippet.enabled (default true) so the chart can deploy on hardened controllers that set allow-snippet-annotations: false. - values.yaml: add wordpress.wordfence.enabled and ingress.serverSnippet.enabled. - Chart.yaml: bump 3.2.7 -> 3.3.0 (SemVer + WeOwnVer valid). - wordpress/CHANGELOG.md: document 3.3.0. Verified: helm template renders correctly with default, values-burnedout, and values-ptoken; helm lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ingress): expand edge block list and add security response headers Addresses PR #19 review items 11 and 12: - Block xmlrpc.php, wp-config backups, readme.html, license.txt, debug.log, and .svn at the nginx edge (defense-in-depth) - Add X-Frame-Options, X-Content-Type-Options, HSTS, Referrer-Policy, Permissions-Policy, and Content-Security-Policy via configuration-snippet - CSP is configurable per site via ingress.securityHeaders.contentSecurityPolicy - Document wp-login.php rate limiting as controller-level config Chart version 3.3.0 → 3.3.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(values): resolve dead config, env var duplicates, and size mismatch - Move APACHE_HTTP_PORT from dead wordpress.extraEnvVars to top-level extraEnvVars (deployment.yaml only reads the top-level key) - Remove duplicate WORDPRESS_ENABLE_REDIS (already injected by template when redis.enabled=true) and WORDPRESS_CONFIG_EXTRA (already generated from wordpressExtraWpConfigContent — duplicate silently overwrote the more complete template version) - Align proxy-body-size 64m -> 128m to match PHP upload_max_filesize - Fix comment: global.domain -> wordpress.domain - Remove dead hosts field from per-site tls overrides (template builds hosts from wordpress.domain, not tls list) - Document enableMultisite/redirectFromWWW as not yet wired Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: m.shahid <m.shahid538@yahoo.com> Co-authored-by: Nik <nik.cimino@gmail.com> Co-authored-by: romandidomizio <rodi1364@colorado.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * - SearXNG Deployment to 'searxng.weown.app'. - Add SearXNG deployment and team integration docs - Adds the production deployment assets, ingress routing, browser integration guide, workstation browser policy playbook, and secret-safe SearXNG settings templating so the service can be reviewed without committing production secrets. * refactor(searxng): restructure as copier template matching keycloak-docker pattern Resolves all 15 review comments from PR #20 code review: BLOCKING (4 resolved): - Remove committed production IPs from inventory.ini (no inventory file; copier template uses dynamic doctl-based discovery) - Remove committed RFC1918 IPs from k8s-external-ingress.yaml (k8s manifests removed; Caddy handles TLS directly on droplet) - Fix TLS bypass: old docker-compose bound port 80 publicly; now Caddy handles all TLS termination with auto Let's Encrypt - Pin container images: configurable image variables replace :latest tags SECURITY (4 resolved): - Secret key no longer world-readable on disk: Infisical Machine Identity injects SEARXNG_SECRET_KEY at runtime via `infisical run` - Disable Google autocomplete (privacy leak): set autocomplete to "" - TLS cipher restrictions: Caddy enforces modern TLS by default (no nginx ingress annotations needed) - Remove StrictHostKeyChecking=accept-new: no inventory.ini OPERATIONAL (5 resolved): - Docker installation via get.docker.com (upstream docker-ce, not mismatched docker.io + docker-compose-plugin) - Ansible deploy uses proper changed_when patterns - Health checks defined for all 3 services (searxng, valkey, caddy) - Valkey actually configured: settings.yml includes redis.url pointing to valkey:6379 for rate-limiting and bot detection - Firefox policies.json: slurp-merge-write pattern preserves existing enterprise policies instead of overwriting TEMPLATE STRUCTURE (new): - copier.yaml with _subdirectory: template (copier >= 9.0.0) - template/terraform/ — main.tf (droplet + reserved IP + firewall), monitoring.tf (CPU/mem/disk alerts), variables.tf, outputs.tf, backend.tf, versions.tf, cloud-init.yaml - template/docker/ — compose.prod.yaml, Caddyfile, searxng/settings.yml - template/scripts/ — deploy.sh, backup.sh (skinny backups with DO Spaces offload), restore.sh - template/ansible/ — deploy.yml, configure-browser-search.yml - template/ — .gitignore, README.md, CHANGELOG.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(wordpress-docker): use Minimus private image with PDO extensions (D117) Replace reg.mini.dev/wordpress:latest with reg.mini.dev/1923/wordpress-fluentsmtp:latest in the copier template default and all generated site configs. The base Minimus WordPress image strips PDO (WordPress core uses mysqli). FluentSMTP requires PDO for email logging — without it, logins trigger a fatal error. The private image includes php-pdo-auto and php-pdo-mysql-auto plus 5 other PHP extensions. Ref: burnedout.xyz PDO postmortem 2026-04-14, decision D117. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): align drifted chart versions; tighten CHANGELOG consistency check The version-consistency check in .github/workflows/validation.yml used `grep -qi $chart_version $changelog_file` (presence-anywhere), which silently passed even when Chart.yaml drifted away from the latest CHANGELOG entry. Concrete case discovered during PR #19 review: wordpress/helm/Chart.yaml was 3.2.6 while wordpress/CHANGELOG.md's top entry was [3.3.8] — the older 3.2.6 string still appeared in past entries, so the check kept passing. Tighten the check to compare Chart.yaml's `version:` against the first `## [X.Y.Z]` header in CHANGELOG.md (the Keep-a-Changelog newest-on-top convention). Three charts had pre-existing drift and would fail the new check on main, so align them in this commit: nextcloud/helm/Chart.yaml 1.0.0 → 1.1.0 vaultwarden/helm/Chart.yaml 1.0.0 → 1.3.1 wordpress/helm/Chart.yaml 3.2.6 → 3.3.8 These are metadata-only bumps: Chart.yaml now matches the version each chart's own CHANGELOG already declares as latest. No template or values changes. Not addressed in this commit (separate decisions needed): - WeOwnVer format inconsistency between docs/VERSIONING_WEOWNVER.md (SEASON.MONTH.WEEK.ITERATION) and validation.yml's `versioning` job (SEASON.WEEK.DAY.VERSION). None of the current chart versions cleanly fit either schema. - The `versioning` job only validates the first Chart.yaml found alphabetically (`*/helm/Chart.yaml | head -1` on line 212), not all of them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): split long error message to resolve yamllint line-length warning * DR initial commit with restic + velero * fix(cluster-backup): address all reviewer + CI feedback from PR #14/#24 Resolves the 3 failing CI jobs and all 30 inline review comments on PR #24 (and the predecessor PR #14, whose comments were ported via the branch rename `dra` → `feature/shahid-velero-restic`). CI failures fixed ----------------- * Helm Template Validation — chart no longer declares an upstream `velero` subchart dependency (the chart provides its OWN Velero + Restic templates; the two would collide and `helm template` could not run because `helm dependency build` is not invoked in CI). * Helm Lint — removed the duplicate `cluster-backup.restoreName` template definition in `_helpers.tpl`. * Documentation Validation / Markdown Lint — `cluster-backup/README.md` now satisfies MD031/MD022/MD032 (blank lines around fences / headings / lists). Added `cluster-backup/CHANGELOG.md` with a `[1.0.0] - 2026-05-22` entry that matches `Chart.yaml`'s `version: 1.0.0` — required by the tightened version-consistency check landed in #21. * trailing-whitespace / end-of-file-fixer — stripped trailing whitespace from every cluster-backup yaml + shell file and ensured every file ends with `\n`. Security & correctness (per Copilot / @romandidomizio review) ------------------------------------------------------------ * `automountServiceAccountToken: false` flipped to `true` on BOTH the Velero server and the Restic node-agent ServiceAccounts. Both controllers MUST call the Kubernetes API (Velero for cross-namespace resource enumeration / Backup-Restore CRDs; Restic for PodVolumeBackup / PodVolumeRestore / ResticRepository / coordination leases). Disabling token automount without a projected-token workaround silently broke the controllers. * Removed `helm/templates/velero-secret.yaml`. The chart was rendering a placeholder Secret named `cluster-backup-cloud-credentials` which collided with the real Secret that `deploy.sh` creates out-of-band (`helm upgrade` would fail with "resource already exists" on every re-install). The Secret is now solely owned by `deploy.sh`. Removed the corresponding `checksum/secret` annotation from the Velero Deployment and Restic DaemonSet. * Added `helm/templates/velero-service.yaml`. The chart previously shipped a `ServiceMonitor` selecting a Service that did not exist, and `NOTES.txt` instructed users to `kubectl port-forward svc/<fullname>-velero` — also missing. Service is `ClusterIP`, metrics-only. * `helm/templates/networkpolicy.yaml` egress no longer uses `to: []` for any rule. DNS is constrained to the kube-system namespace; external S3 traffic is allowed to `0.0.0.0/0` MINUS RFC1918, link-local, and loopback ranges (so restic / velero can never accidentally exfiltrate to in-cluster IPs over an S3-shaped path); Kubernetes API access is constrained to the `default` namespace ClusterIP. * `helm/templates/restic-daemonset.yaml` no longer mounts host `/` via `hostPath`. Restic only needs read access to `/var/lib/kubelet/pods` (the standard kubelet pod-volume root), so that's the only host mount that remains. Mounting host-root was incompatible with Pod Security `restricted` and not required for restic operation. * `helm/templates/backup-schedules.yaml` passes a full `dict "Chart" $.Chart "Release" $.Release "Values" $.Values "Template" $.Template ...` context into the schedule-naming and schedule-config helpers, removing any ambiguity about what `cluster-backup.fullname` and `cluster-backup.labels` resolve to when called from within a `range` loop body. * `helm/templates/_helpers.tpl`: simplified `cluster-backup.serviceAccountName` and `cluster-backup.resticServiceAccountName` to return deterministic names. The previous form gated on `.Values.velero.{server,restic} .serviceAccount.create`, which is not a path that exists in `values.yaml` — it was a remnant of the now-removed `velero` subchart pattern. The helpers always returned `"default"`, which conflicted with the ClusterRoleBinding subject the chart actually installed; Velero would have been denied every API call. * `helm/templates/NOTES.txt` rewritten: `kubectl create backup …` / `kubectl create restore …` are NOT valid kubectl subcommands for Velero CRDs and were misleading users into thinking the controller was broken when in reality they were running a non-existent kubectl verb. NOTES now uses the Velero CLI (`velero backup get`, `velero schedule get`, etc.) and points readers at the install docs. Same fix applied across README.md, deploy.sh, verify.sh, test-local.sh, and (post-deployment) usage messages. Script hardening ---------------- * `deploy.sh`: - Switched to `#!/usr/bin/env bash` + `set -euo pipefail`. - All temporary files now live in a single `mktemp -d`-created directory with mode 0700, cleaned up via an `EXIT|INT|TERM` trap. The previous version wrote `/tmp/s3-credentials` and `/tmp/cluster-backup-values.yaml` at fixed paths — world-readable on some hosts and racy under concurrent runs. - S3 secret-key prompt uses `read -rs` (no echo). The credential is piped into `kubectl create secret … --from-file=cloud=/dev/stdin --dry-run=client -o yaml | kubectl apply -f -`, so the secret never lives on disk and never appears in argv (`ps`). - All configuration accepts env-var inputs (`TENANT`, `CLUSTER`, `ENVIRONMENT`, `S3_*`) so the script is usable in CI/automation without an interactive TTY. * `test-local.sh`: - MinIO image pinned (`minio/minio:RELEASE.2024-08-17T01-24-54Z`). Was `minio/minio:latest`, which made local test runs non-reproducible and exposed them to silent upstream breaking changes. - MinIO credentials no longer hardcoded as `minioadmin / minioadmin` in the manifest. Access key defaults to `localdev-access`; secret key is a freshly-generated 24-char random value per run (overridable via `TEST_MINIO_{ACCESS,SECRET}_KEY` env vars). Rendered into the Secret over a stdin pipe — same no-argv-leak pattern as deploy.sh. - Service now correctly exposes BOTH the S3 API (NodePort 30000) AND the web console (NodePort 30001). The original Service mapped only 9000 to NodePort 30000 yet the "MinIO Console" line pointed at 30000 — that's the S3 API, not the console. Web console address is now `http://localhost:30001`. - Replaced fixed `sleep 30` waits with `wait_for_velero_phase` which polls `.status.phase` of the Backup / Restore CR and emits `kubectl describe` + `velero backup logs` diagnostics on timeout. - Uses the Velero CLI for backup / restore creation. * `verify.sh`: - Same shebang + strict-mode + Velero-CLI updates as deploy.sh. - `test_backup_creation` polls phase with diagnostics instead of blind-sleeping; also skips gracefully if the Velero CLI isn't installed locally (verify is run on hosts that may not have it). * `create-tenant-cluster.sh`: - All upstream components are version-pinned via env-var defaults: `DOKS_K8S_VERSION`, `INGRESS_NGINX_VERSION`, `CERT_MANAGER_VERSION`, `METRICS_SERVER_VERSION`. The previous version used `doctl kubernetes cluster create … --version latest` and `metrics-server/releases/latest/`, making provisioning silently non-reproducible. - Team-member namespace is normalized to RFC1123 (`AnnaF` → `annaf`, etc.) before being passed to `kubectl create namespace`. K8s rejects uppercase-containing namespace names with an opaque error, and the previous version of this script would silently fail at `create_team_member_namespace` for any contributor whose handle wasn't already lowercase-alphanumeric. - `create_team_member_access` no longer reads `.secrets[0].name` to harvest a long-lived ServiceAccount token (the BoundServiceAccount- Token feature in K8s ≥1.24 means that field is empty on modern clusters, so the function was returning an empty kubeconfig). Switched to `kubectl create token` (TokenRequest API) with an explicit 24h duration. Kubeconfig is written with `umask 077` so the file containing the bearer token is mode 0600. Not in scope for this commit ---------------------------- * The Restic DaemonSet still ships with a `SYS_ADMIN` capability add and a non-root securityContext (uid 65534). Real-world restic operation typically requires either root or a relaxed podSecurityContext to read pod volumes owned by arbitrary uids; tightening this further requires a per-environment decision about the Pod Security Standard for the `velero` namespace. Flagged in `cluster-backup/CHANGELOG.md` under "Notes for operators". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cluster-backup): repair Schedule rendering + dedupe component label Two issues that surfaced once kubeconform actually parsed the rendered output (the previous CI run was already past my first commit but failed on Kubeconform Validation): 1. Duplicate `app.kubernetes.io/component` label The common `cluster-backup.labels` helper emitted `app.kubernetes.io/component: backup-system`, and every per-resource template also appended its own `app.kubernetes.io/component: <role>` right below the include. The K8s API server tolerates duplicate map keys (last write wins), but strict YAML parsers — including `kubeconform` — reject the manifest. Removed the component-label line from the shared helper; per-resource components remain. 2. Schedule CR `.spec.schedule` rendered as `"map[…]"` instead of cron The `cluster-backup.backupScheduleConfig` helper accessed `.schedule | quote` at the top level, but its caller passes the whole schedule object under `.schedule` — so `.schedule` was the FULL map, and Go's `fmt.Sprint` rendered it as `"map[enabled:true excludeNamespaces:[…] includeNamespaces:[] retention:30d schedule:0 2 * * *]"`. Velero would have rejected the Schedule as invalid syntax at apply time. Reworked the helper to extract fields explicitly via `.schedule.schedule`, `.schedule.retention`, `.schedule.includeNamespaces`, `.schedule.excludeNamespaces`. This is the same access pattern the caller already implied; the helper had just been written incorrectly from the start. The CR now renders with the expected `schedule: "0 2 * * *"`. Also fixed an adjacent issue in `backup-schedules.yaml`: the `{{- $ctx := dict … -}}` line had its trailing newline stripped, which fused the previous Schedule's last YAML line directly to the next `---` document separator (e.g. `defaultVolumesToRestic: true---`). Replaced with non-stripped `{{ $ctx := merge … }}` on its own line. Verified with `helm template … | kubeconform -strict -summary -ignore-missing-schemas -`: 21 resources, Valid: 11, Invalid: 0, Errors: 0, Skipped: 10 (skipped resources are Velero CRDs with no published JSON schema: BackupStorageLocation, VolumeSnapshotLocation, Schedule). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cluster-backup): resolve Trivy CRITICAL/HIGH findings Local `trivy config --severity CRITICAL,HIGH cluster-backup/` is now clean (0 findings). Two categories: Real code fixes --------------- 1. **KSV-0053 (HIGH) — restic ClusterRole granted `pods/exec`.** Restic backs up pod volumes by reading the kubelet bind-mount; it never execs into user pods. Removed `pods/exec` from the resource list and tightened the verb set on `[pods, pods/log, namespaces, nodes]` from `get list watch create update patch delete` down to just `get list watch` — restic does not own any of those resources. This also drops KSV-0042 (delete pod logs) + part of KSV-0048 (manage workloads/pods) from the restic role specifically. 2. **KSV-0005 (HIGH) — restic securityContext was internally inconsistent.** The previous values.yaml had `runAsNonRoot: true, runAsUser: 65534` + `capabilities.add: [SYS_ADMIN]`. uid 65534 cannot read pod volumes owned by arbitrary uids, so restic would have failed at runtime. Per Velero's documented restic deployment posture (https://velero.io/docs/v1.12/restic/), the node-agent must run as root with SYS_ADMIN. Changed to `runAsNonRoot: false, runAsUser: 0, runAsGroup: 0, fsGroup: 0`, keeping `readOnlyRootFilesystem: true` and the default seccomp profile. The PSS for the `velero` namespace must therefore be `baseline` or `privileged` (NOT `restricted`) — documented in cluster-backup/CHANGELOG.md. Targeted .trivyignore additions ------------------------------- The remaining findings are inherent to a cluster-backup tool and cannot be fixed without breaking its function. Each entry in .trivyignore is explained — Trivy ignore-rules are the right vehicle for "this is intentional, here's why": - KSV-0041 (CRITICAL): Velero must manage Secrets to back them up. - KSV-0056 (HIGH x3): Velero must manage Services/Endpoints/Ingresses/ NetworkPolicies to back up and restore network topology. - KSV-0005 / KSV-0022 (HIGH): SYS_ADMIN is required for restic's mount/setns ops; this is Velero's documented deployment posture. - KSV-0012 (MEDIUM): companion of KSV-0005 (run-as-root). - KSV-0023 (MEDIUM): hostPath on `/var/lib/kubelet/pods` is how restic reads pod volumes. The earlier broader mount of host `/` was already removed in d0c637b. - KSV-0042 (MEDIUM): Velero must delete pods to restore them. - KSV-0048 (MEDIUM): Velero must create/update Deployments, StatefulSets, DaemonSets, Jobs, CronJobs. - KSV-0049 (MEDIUM): Velero must manage ConfigMaps. - KSV-0125 (MEDIUM): `velero/velero:v1.12.2` is the official upstream image on docker.io. Mirroring to a private registry is tracked as a separate follow-up. All ignored rules are scoped by .trivyignore comments to the cluster-backup chart only. Roman's review verified ----------------------- All 17 items from @romandidomizio's CHANGES_REQUESTED review have been addressed in this branch (d0c637b + ac2e9f6 + this commit). Spot-checked all of them; no outstanding source-level issues. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(wordpress-docker): replace .env secrets with Infisical runtime injection Removes all on-disk .env.prod secret management and replaces it with the Infisical Universal Auth (Machine Identity) pattern, matching the keycloak-docker reference implementation. Changes: - copier.yaml: enable_infisical defaults true; prompts for client_id and client_secret instead of legacy infisical_token - variables.tf.jinja: replace infisical_token + enable_infisical bool with infisical_client_id + infisical_client_secret (sensitive); mysql_password / mysql_root_password gated on not enable_infisical - main.tf.jinja: templatefile() passes correct vars per enable_infisical - cloud-init.yaml.jinja: installs Infisical CLI, writes infisical-auth.sh (machine identity), starts stack via infisical run, cron backup authenticated via infisical-auth.sh + infisical run - terraform.tfvars.example.jinja: removes mysql passwords from Required section; replaces infisical_token with client_id + client_secret - ansible/deploy.yml.jinja: removes pre_tasks (.env.prod check), removes Upload .env task, converts docker_compose_v2 calls to shell + infisical run - scripts/deploy.sh.jinja: removes .env.prod check + scp; uses infisical run - scripts/backup.sh.jinja: removes source .env; uses infisical run for DB dump - scripts/restore.sh.jinja: removes source .env + dot-env restore; uses infisical run for DB restore and compose up - docker/.env.prod.example: replaced with Infisical pointer (no passwords) Compliance: §3.10 Infisical checklist, §3.1 NIST PR.DS, §3.2 CIS 3.11, §3.4 ISO A.8.24, §3.5 SOC2 CC6.7, §3.8 Docker Compose checklist * feat(wordpress-docker): add pull-prod workflow + fix backup/restore scripts - Add scripts/pull-prod.sh: pulls production DB + wp-content to local dev stack in one command; reads DB password from docker inspect (not .env) to avoid silent dump failures when credentials diverge - Fix backup.sh: use docker inspect for DB password instead of sourcing .env — same root cause that caused 0-byte dumps in burnedout-xyz - Fix restore.sh: add COMPOSE_PROJECT_NAME=burnedout-local, auto URL replacement (siteurl/home) after prod DB import, use mariadb client - Fix compose.local.yaml: add Caddy service (WP image is FPM-only; missing Caddy made the stack silently unreachable on port 8080) - Add Caddyfile.local: HTTP-only local Caddy config for FPM proxy - Propagate all fixes to template (pull-prod.sh.jinja, backup.sh.jinja, restore.sh.jinja, compose.local.yaml.jinja) - Expand README.md (site + template) with Day-to-Day Operations section covering when/how to use each script Fixes: silent 0-byte mysqldump when .env password diverges from running container; localhost:8080 returning empty due to missing Caddy * chore: add Ansible template and documentation files - RESTORE-RUNBOOK-PROMPT.md: runbook documentation for burnedout.xyz restore - scripts/manage-droplets.sh: droplet management script (fixed SC2155 shellcheck) - wordpress-docker/template/ansible/: Ansible configuration, inventory template, and requirements * fix(wordpress-docker): address code review findings Critical fixes: - restore.sh: fix PROJECT_NAME ("burnedout" not "burnedoutxyz") and APP_DIR ("/opt/burnedout" not "/opt/burnedout-xyz") — remote restore was completely broken, targeting nonexistent containers and paths - restore.sh: use docker inspect for DB credentials instead of sourcing .env — matches backup.sh pattern, works with or without Infisical - RESTORE-RUNBOOK: replace broken base image (reg.mini.dev/wordpress:latest) with correct private image (reg.mini.dev/1923/wordpress-fluentsmtp:latest) Medium fixes: - backup.sh: stop including .env in backup archive — production secrets should not be stored in tarballs (credential exposure vector) - RESTORE-RUNBOOK: fix false claim that compose.local.yaml sets WORDPRESS_CONFIG_EXTRA (only compose.prod.yaml does) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Auto-PR: feat: add observability infrastructure, SearXNG copier template, and fleet tooling (#26) * feat: add observability infrastructure, SearXNG copier template, and fleet tooling Add SigNoz self-hosted observability (signoz-docker/) and OTel Collector agent (otel-agent/) for per-container metrics, log aggregation, and traces across the fleet. Add SearXNG copier template (searxng-docker/) with full IaC, Infisical integration, skinny backups, and browser search playbook. Add fleet scripts for DO agent enablement and OTel deployment. Add AnythingLLM MCP configuration playbook. Fix cloud-init `runcmds:` typo to correct `runcmd:` across all templates. Fix manage-droplets.sh SSH key rotation parsing bug. Add project CLAUDE.md for AI guidance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(otel): SigNoz Cloud fleet agent Phase 1 (burnedout-xyz verified) Pivot observability to SigNoz Cloud us2 with per-droplet OTel collector, Infisical runtime secret injection, bootstrap/deploy fleet scripts, and deprecation notes for self-hosted signoz-docker template. * fix(pr26): address Copilot + reviewer findings on observability work Resolves all blocking review feedback on PR #26 surfaced by Copilot (rounds 1 + 2) and an independent code review of the diff. Covers correctness bugs, security defaults, and operational gaps. Scope is limited to closing review items — no new features. Correctness bugs (deploy-blocking): - signoz scripts (deploy/backup/restore + ansible/deploy.yml.jinja): use `replace('-', '_')` to match `main.tf.jinja` -> `templatefile` -> cloud-init path/volume normalization. Previously the no-separator form (`replace('-', '')`) caused /opt/signozobservability vs /opt/signoz_observability divergence and silent volume-name mismatches in backup/restore. - cloud-init OTel gateway DSN: switch `password=$${CLICKHOUSE_PASSWORD}` to `password=$${env:CLICKHOUSE_PASSWORD}` so the OTel collector actually expands the env var (bare `${VAR}` is not interpolated by the collector). - README.md.jinja: fix broken code fence at line 92 ("```nished Phase 1..."). - README.md.jinja: replace undefined `monitoring_*_threshold` references with the actual copier-defined names (`cpu_alert_threshold`, etc.). - monitoring.tf.jinja: hardcoded `value = 8` for load-5 alert is now driven by a new `load_alert_threshold` copier variable (default 8 = 2x default vCPUs). - CHANGELOG.md.jinja: drop the stub `[{{ version }}] - {{ date }}` line that referenced undefined copier variables. - anythingllm/ansible/configure-allm.yml: rewrite to use plain Ansible templating (`{{ var }}`) instead of broken copier-style escapes (`{{ '{{' }} var {{ '}}' }}`). Wrap Docker Go-template (`{{.State.Running}}`) in `{% raw %}...{% endraw %}` so Ansible passes it through. Add `no_log: true` to tasks that handle merged MCP server configs (which can include credentials from `extra_mcp_servers`). - otel-agent OTLP transport docs: fix gRPC/HTTP mismatch in comments across config.yaml, compose.yaml, deploy-otel-fleet.sh, otel-agent/README.md (the actual exporter is otlphttp). - scripts/enable-do-agent.sh: drop unused `ALREADY=0` counter. Security + IaC fixes: - terraform backend.tf (signoz + searxng): remove invalid `var.*` references in the s3 backend block (backend cannot interpolate variables — init runs before vars are evaluated). Add the working DO Spaces pattern from keycloak-docker/sites/sso.weown.dev/ (skip_* flags) + a new `init.sh.jinja` that bridges terraform.tfvars -> `tofu init -backend-config=`. Also drop the unsupported `lock`/`lock_timeout` block. Moves `spaces_*` variable declarations to variables.tf.jinja and adds placeholders to terraform.tfvars.example.jinja. - New `ssh_source_cidrs` copier variable (signoz + searxng) drives the SSH firewall rule. Default keeps backward-compat `["0.0.0.0/0", "::/0"]` with a loud help warning to pin to admin IP/32 or VPN range in production. - signoz cloud-init Infisical CLI install: switch from the deprecated install-cli.sh channel (capped at v0.38 — broken `infisical run`) to the current artifacts-cli.infisical.com apt repo, mirroring the idempotent block in scripts/bootstrap-otel-agent.sh. - searxng copier.yaml: pin `searxng_image` to a dated release tag (was `:latest` — drift risk on every pull); help text directs operators to Docker Hub for newer tags. - searxng .gitignore: rewrite the `searxng/settings.yml` ignore rule to the actual generated path `docker/searxng/settings.yml`. - restore.sh.jinja: validate `BACKUP_NAME` against a strict allowlist regex `^[A-Za-z0-9._-]+$` before forwarding it into the remote ssh `bash -c` command, closing the shell-injection vector flagged by both reviewers. - otel-agent deploy paths (deploy-otel-fleet.sh + otel-agent/deploy.yml): reject `http://` URLs explicitly in the OTEL_URL normalization case; only `https://` is accepted (config.yaml sets `insecure: false`). - otel-agent/README.md: add a "Threat model — host-level access by design" section that explicitly documents the root+docker.sock+host-root-mount topology, what `:ro` mitigates vs doesn't, and the mitigations the design relies on. Operational quality: - copier.yaml (signoz + searxng): remove `s3` from the `backup_remote_storage` choices — the backup/restore scripts only implement `do-spaces`, so picking `s3` was a silent no-op. - copier.yaml + variables.tf.jinja + README.md.jinja (signoz): reword `clickhouse_retention_days` description to mark it as informational (actual TTL is set via SigNoz UI), since the value was never wired into any ClickHouse config. - compose.prod.yaml.jinja + cloud-init.yaml.jinja (signoz + searxng): add `/var/log/caddy:/var/log/caddy` bind mount so Caddy access logs survive container recreation and the otel-agent filelog/caddy receiver can pick them up from the host. Repo hygiene: - Add `.claude/` to top-level `.gitignore` (Claude Code local workspace). Deferred (require design decisions; tracked for follow-up): - Infisical Machine Identity persistence in user_data / TF state. Documented as known risk in CHANGELOG; full remediation requires architectural change (cloud-init bootcmd + tmpfs, systemd credentials, or external bootstrap delivery). Rotate the Machine Identity after destroy. - `lifecycle { ignore_changes = [user_data] }` masks user_data updates (intentional, to avoid droplet recreation on every var bump). Future: document `tofu taint` workflow in signoz-docker README. - Docker-socket-proxy in front of otel-agent (significant scope, follow-up PR). - The legacy `install-cli.sh` Infisical install pattern in `anythingllm-docker/` and `keycloak-docker/` cloud-init.yaml.jinja (out of this PR's scope; same fix applies — separate PR). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pr26): address Copilot round-3 + own review pass Resolves the 22 new Copilot comments on merge commit 59a3d0a plus additional issues found in an own-review pass over the diff. New Copilot findings (all addressed): - signoz-docker/template/README.md.jinja: quick-start used `tofu init` directly, but the S3 backend requires `-backend-config` flags via the new `./init.sh`. Rewrote step 2 to use the init script + documented why (variables not allowed in backend blocks). - signoz-docker/template/README.md.jinja: step 3 referenced the old positional-arg `deploy-otel-fleet.sh weown-ai <ip>` syntax which was removed in the SigNoz Cloud pivot. Rewrote to use the new flag-based selectors and added context that fleet OTel agents ship to SigNoz Cloud by default (not to this self-hosted gateway). - signoz-docker/template/scripts/{deploy,restore}.sh.jinja: replaced `143.198.xxx.xxx` example IPs (real DO range) with RFC 5737 documentation ranges (`198.51.100.42`) so the public repo does not normalize references to real infrastructure. - signoz-docker/template/terraform/templates/cloud-init.yaml.jinja: `docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'` would have been interpreted by Copier's Jinja pass as attribute lookups on the root context (likely failing rendering). Wrapped in `{% raw %}...{% endraw %}`. - signoz-docker/template/terraform/templates/cloud-init.yaml.jinja: retention echo claimed "daily 30d / monthly 12mo / yearly forever" but the implementation only deletes local backups older than 30 days. Rewrote the message to match the actual behavior. - scripts/bootstrap-otel-agent.sh: Infisical Machine Identity secrets were passed inline as env-var assignments in the ssh command string, which exposes them in the local `ps` listing during the SSH connection. Rewrote to pipe `export VAR=...` (via `printf %q` for safe quoting) through ssh stdin (process substitution) so the secrets never appear in argv. - otel-agent/README.md: my prior "Threat model" section claimed that `:ro` on `/var/run/docker.sock` prevents Docker API writes. That is incorrect — `:ro` is a bind-mount flag affecting the socket inode, NOT the Docker API protocol. Any process that can `connect(2)` to the socket can issue write API calls (containers/create, exec, --privileged, etc.) regardless of mount mode. Rewrote the section to call out the socket access as effectively root-on-host, list what we actually rely on (image pinning, memory cap, no host-side secrets at rest), and document the docker-socket-proxy mitigation path for future work. - otel-agent/compose.yaml + otel-agent/README.md "Safety" section: brought the "all mounts read-only ⇒ safe" claim in line with the corrected threat model — fs mounts are tamper-safe, but the docker socket isn't, and we say so explicitly. - signoz-docker/template/docker/compose.prod.yaml.jinja + signoz-docker/template/terraform/templates/cloud-init.yaml.jinja: added a comment block on the ZooKeeper service explaining that `ALLOW_ANONYMOUS_LOGIN: "yes"` is intentional for this deprecated self-hosted fallback (port unpublished, only ClickHouse reaches it on `signoznet`); enabling SASL would require synchronized ClickHouse credentials and a new Infisical secret. Operators adopting this for production should enable auth before exposing the stack. Own-review pass: - scripts/deploy-otel-fleet.sh: updated the OTEL_URL normalization comment to match the actual behavior (https:// or scheme-less; reject plain http:// because `tls.insecure: false`). The previous comment said "otlphttp requires https:// or http://" which contradicted the reject-http logic. Not addressed (deferred — same call as round 2): - Infisical Machine Identity persisted in user_data / Terraform state (architectural change required; tracked in CHANGELOG). - Copilot's two "configure-allm.yml jinja escapes" comments at line 33 are stale — that file was fixed in a prior commit and currently uses plain `{{ var }}` form. No further action needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(anythingllm): move configure-allm.yml under anythingllm-docker/ The configure-allm.yml playbook uses `docker exec` + `docker inspect`, so it is Docker-specific. The `anythingllm/` directory is reserved for the helm/k8s deployment path; Docker-stack tooling belongs under `anythingllm-docker/` next to the copier template + cloud-init that provision the Docker-based instances this playbook configures. Used `git mv` to preserve file history. The empty `anythingllm/ansible/` directory has been removed. Updated the CHANGELOG entry to reflect the new location and the rationale for the move. No content changes — only relocation. Callers (currently none in-repo) referencing `anythingllm/ansible/configure-allm.yml` should switch to `anythingllm-docker/ansible/configure-allm.yml`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Nik <nik.cimino@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: m.shahid <m.shahid538@yahoo.com> Co-authored-by: Nik Cimino <ncimino@users.noreply.github.com> * Auto-PR: chore(s004): add baseline copier templates for monitoring and infrastructure scaffolding (#31) * feat: add observability infrastructure, SearXNG copier template, and fleet tooling Add SigNoz self-hosted observability (signoz-docker/) and OTel Collector agent (otel-agent/) for per-container metrics, log aggregation, and traces across the fleet. Add SearXNG copier template (searxng-docker/) with full IaC, Infisical integration, skinny backups, and browser search playbook. Add fleet scripts for DO agent enablement and OTel deployment. Add AnythingLLM MCP configuration playbook. Fix cloud-init `runcmds:` typo to correct `runcmd:` across all templates. Fix manage-droplets.sh SSH key rotation parsing bug. Add project CLAUDE.md for AI guidance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(s004): sync live caddy stdout configuration and private registry updates * chore(s004): add baseline copier templates for monitoring and infrastructure scaffolding * fix(s004-deployment): address Copilot + reviewer findings — make deploy actually work Resolves 8 new Copilot comments on the merge commit + 12 findings from an independent reviewer pass on s004-deployment/. The directory was committed as a skeleton with several deployment-blocking bugs. CRITICAL — deployment-blocking bugs: - Volume name mismatch (backup.sh, restore.sh, cloud-init backup script): scripts referenced `${PROJECT_NAME}_storage` where `PROJECT_NAME=s004anythingllm` (no underscore), so the resolved name was `s004anythingllm_storage`. But `docker/compose.prod.yaml` (and cloud-init's embedded compose) explicitly name volumes `s004_anythingllm_storage`. Every backup tarred an empty anonymous volume; every restore wrote into a phantom volume the app never reads. Hardcoded the actual volume names (`s004_anythingllm_storage`, `s004_anythingllm_caddy_data`) in all three sites. - Empty Infisical credentials in cloud-init: the `infisical-auth.sh` helper, daily cron wrapper, and runcmd `infisical run` all had literal `INFISICAL_CLIENT_ID=''` / `--projectId=`. terraform/main.tf already plumbs `infisical_client_id`, `infisical_client_secret`, `infisical_project_id`, `infisical_environment` into `templatefile()`, so just reference them as `${infisical_*}`. - Empty `--projectId=` in scripts/deploy.sh + scripts/restore.sh: workstation scripts now read INFISICAL_PROJECT_ID (and optional INFISICAL_ENV, default `prod`) from the environment with a `:?` guard that prints a clear error. - Hardcoded `s004.ccc.bot` domain in cloud-init compose env + Caddyfile: switched to `${domain}` so var.domain takes effect. - Image registry inconsistency: cloud-init pulled `mintplexlabs/anythingllm:latest`, compose used `reg.mini.dev/anythingllm:latest`. Both now use `${anythingllm_image}` from var.anythingllm_image (default `reg.mini.dev/anythingllm:1.7.2`, the WeOwn mirror). - 23 unescaped shell variables in cloud-init `templatefile()` body (`${JWT_SECRET}`, `${ADMIN_EMAIL}`, `${BACKUP_NAME}`, `${SPACES_BUCKET}`, `${BASH_REMATCH[1]}`, etc.) would have made `tofu plan` error before producing any output. Escaped every uppercase shell var as `$${VAR}`. HIGH — lessons from PR #26 applied here: - SSH 0.0.0.0/0 firewall rule → introduced `var.ssh_source_cidrs` (default `["0.0.0.0/0", "::/0"]` with help text directing production to pin to admin/VPN CIDR). Same fix as signoz-docker. - Deprecated Infisical install channel (`infisical.com/install-cli.sh`, capped at v0.38 with broken `infisical run` session handling) → rewrote to use the current `artifacts-cli.infisical.com` apt repo with the same idempotent legacy-purge logic from `bootstrap-otel-agent.sh`. - Caddy access logs written to `/var/log/caddy/` but no host bind mount → added `- /var/log/caddy:/var/log/caddy` to both `docker/compose.prod.yaml` and the cloud-init embedded compose, so otel-agent's filelog/caddy receiver can read them. - `restore.sh` shell-injection via unvalidated `$BACKUP_NAME` → `[[ ! "$BACKUP_NAME" =~ ^[A-Za-z0-9._-]+$ ]]` guard before the heredoc. - Real DigitalOcean IP `143.198.xxx.xxx` in usage examples → RFC 5737 `198.51.100.42`. MEDIUM — code quality: - `terraform.tfvars.example` had unrendered Copier jinja stubs (`{{ project_name }}`, `{{ enable_skinny_backups | lower }}`, etc.) — operators copying the file to terraform.tfvars would get literal jinja that terraform rejects. Replaced every stub with a real example value matching the variable's default in variables.tf. - `.gitignore` ignored `.terraform.lock.hcl`, breaking provider reproducibility. Removed the ignore + added an explanatory comment. - `versions.tf` header still said `# {{ project_name }}` → `s004-anythingllm`. - `monitoring.tf` declared three alert resources that always created regardless of `var.enable_monitoring`. Added `count = var.enable_monitoring ? 1 : 0`. - `scripts/restore.sh` had dead REMOTE/BACKUP_NAME assignments at lines 20-21, overwritten by the arg-count block. Removed. Not addressed (would expand scope): - Infisical Machine Identity rendered into user_data → terraform state. Same trade-off as signoz-docker; needs a remote encrypted backend + ideally one-time-use bootstrap secret retrieval. - `lifecycle { ignore_changes = [user_data] }` means cloud-init fixes require `tofu taint` to deploy. Documented behavior, not changed here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(infra): adopt Path C + Layer 2 in s004-deployment; document migration Resolves the two architectural issues deferred from PR #26 / PR #31: 1. Infisical Machine Identity persisted in terraform state / DO metadata (Layer 2 bootstrap-secret rotation). 2. `lifecycle { ignore_changes = [user_data] }` silently swallowing cloud-init updates (Path C: thin cloud-init + ansible app layer). Reference implementation lives in s004-deployment/. Repo-wide pattern and per-project migration checklist live in docs/INFRA_BOOTSTRAP_PATTERN.md. Layer 2 — bootstrap-secret rotation (s004-deployment/): - New rotate-bootstrap-secret.sh embedded in cloud-init `write_files`. Runs once via runcmd at first boot. Flow: 1. Log in to Infisical with v1 (from terraform.tfvars → state → cloud-init). 2. Decode JWT to extract identityId (base64url-safe, handles padding). 3. POST /api/v1/auth/universal-auth/identities/{id}/client-secrets → v2. 4. Atomically swap .infisical-auth.env to v2 after verifying it auths. 5. POST .../client-secrets/{v1Id}/revoke to disable v1. 6. Touch .rotation-complete (idempotent on re-run). Net effect: v1 in terraform state + DO droplet metadata is revoked within minutes of provisioning. v2 only ever lives on the droplet filesystem. Best-effort with structured logging — if the Machine Identity lacks self-management permission, the script logs cleanly to /var/log/s004anythingllm-rotation.log and the operator follows a manual runbook in s004-deployment/README.md. Path C — thin cloud-init + ansible (s004-deployment/): - Slimmed terraform/templates/cloud-init.yaml. It now handles ONLY first-boot bootstrap: package + Docker install, Infisical CLI install (artifacts-cli apt repo), .infisical-auth.env write, Layer 2 rotation, .bootstrap-complete marker, unattended-upgrades. Removed: compose.yaml, Caddyfile, backup.sh, daily-backup cron, infisical helper, docker pulls, docker compose up. - New ansible/deploy.yml owns the app layer. Uploads compose + Caddyfile + backup.sh, renders the daily backup cron, pulls images, runs `docker compose up -d --remove-orphans`, waits for AnythingLLM health. Pre-tasks assert INFISICAL_PROJECT_ID is set and .bootstrap-complete exists. - scripts/deploy.sh rewritten as a thin wrapper around `ansible-playbook ansible/deploy.yml -i 'host,'`. Idempotent — re-runnable any time compose/Caddy/scripts change, without touching terraform. Layer 1 — DO Spaces remote state backend (already standard elsewhere): - Added terraform/backend.tf + terraform/init.sh + spaces_* vars. State is no longer a plain local JSON file. Same pattern as keycloak-docker/sites/sso.weown.dev/ and signoz-docker (PR #26). Required for Layer 2 to actually reduce risk — without it the v1 in state lives on the operator's workstation indefinitely. Documentation + migration plan: - New docs/INFRA_BOOTSTRAP_PATTERN.md (cross-cutting). Explains both problems, both solutions, failure modes, compliance mapping (NIST CSF 2.0, CIS Controls v8, ISO/IEC 27001:2022), per-project migration checklist for signoz-docker / searxng-docker / anythingllm-docker / keycloak-docker / wordpress-docker, a "what Layer 2 does NOT solve" section, and a future-hardening section (Vault wrapping, cloud-IAM mediation, systemd-credentials + TPM). - Each *-docker README has a MIGRATION PENDING / MIGRATION PARTIAL banner pointing at the pattern doc + s004-deployment reference impl. Created searxng-docker/README.md (was missing). - s004-deployment/README.md: rewrote Quick Start to reflect Path C workflow, added "Updating the deployment" matrix, added manual rotation runbook. - s004-deployment/CHANGELOG.md: Unreleased entry. Migration sequencing recommendation (in INFRA_BOOTSTRAP_PATTERN.md): each *-docker project gets its own focused follow-up PR, order by deployment criticality (anythingllm → wordpress → keycloak → signoz → searxng). Each is ~2-4h plus an operator-scheduled tofu taint + tofu apply against existing droplets. Not changed in this commit: the *-docker template implementations themselves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(infra): per-project Migration Status sections + audited status table Replaces the one-line "MIGRATION PENDING" banners with a per-project Migration Status section in each *-docker/README.md. Each section is a table showing the project's current Layer 1 / Layer 2 / Path C / Infisical CLI state with citations to the files that need work, plus project-specific notes. The shared 6-step migration checklist + rationale stays in docs/INFRA_BOOTSTRAP_PATTERN.md (single source of truth). Audit findings (2026-05-25): - anythingllm-docker: Layer 1 missing, Layer 2 missing, Path C not adopted (no ansible playbook exists), legacy Infisical install. **Highest priority** — source of live deployments. - wordpress-docker: Layer 1 missing, Layer 2 missing, Path C partial (ansible exists with compose + Caddy + Wordfence upload, cloud-init also embeds the app layer), legacy Infisical install. - keycloak-docker: Layer 1 partial (template has backend.tf.jinja but no init.sh.jinja; the rendered sites/sso.weown.dev/ subdir has both), Layer 2 missing, Path C partial (site.yml.jinja with community.docker.docker_compose_v2 — most ansible-shaped of any template, but cloud-init still embeds app layer), legacy Infisical install. - signoz-docker: Layer 1 done (PR #26), Layer 2 missing, Path C partial (ansible exists, cloud-init also embeds), legacy Infisical install (with open ZooKeeper anon-login accepted-risk note). - searxng-docker: Layer 1 done (PR #26), Layer 2 missing, Path C partial, legacy Infisical install. Updated docs/INFRA_BOOTSTRAP_PATTERN.md status table to reflect the audit (was less precise — said "Pending" without distinguishing Layer 1 vs Path C state). The migration sequencing recommendation in the doc unchanged: anythingllm → wordpress → keycloak → signoz → searxng. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(infra): migrate anythingllm-docker to Path C + Layer 2; relocate s004; add auto DO tagging Big bundle: completes Option B (migrate the anythingllm-docker template to the canonical bootstrap pattern, then relocate s004 as a generated site of that template), AND adds the automatic DO droplet tagging system across feature scripts + ansible. Template migration (anythingllm-docker/template/): - terraform/backend.tf.jinja + terraform/init.sh.jinja added (Layer 1 — DO Spaces remote state with SSE-C, same pattern as signoz-docker). - terraform/templates/cloud-init.yaml.jinja replaced with the slim Path C version + embedded rotate-bootstrap-secret.sh (Layer 2). Cloud-init now handles only: package install, Docker install, Infisical CLI install (artifacts-cli apt repo — current channel), .infisical-auth.env write, bootstrap-secret rotation, .bootstrap-complete marker, unattended-upgrades. Compose, Caddy, backup script, cron all removed — they live in ansible now. - terraform/main.tf.jinja: `replace('-', '_')` (was `replace('-', '')` — caused the volume-name mismatch we hit in s004), SSH firewall rule uses `var.ssh_source_cidrs`, `lifecycle { ignore_changes = [user_data, tags] }` so runtime tag mutations stick across `tofu apply` cycles. - terraform/variables.tf.jinja: adds ssh_source_cidrs + spaces_* trio. - terraform/terraform.tfvars.example: adds the new vars with placeholders. - terraform/monitoring.tf.jinja: `count = var.enable_monitoring ? 1 : 0` on each alert so the variable actually gates resources. - docker/compose.prod.yaml.jinja: image pinned via copier vars, Caddy /var/log/caddy bind mount so otel-agent's filelog/caddy receiver can read access logs. - docker/Caddyfile.jinja: domain substitution via copier var. - scripts/deploy.sh.jinja: thin ansible-playbook wrapper. Requires INFISICAL_PROJECT_ID env var. - scripts/backup.sh.jinja: hardcoded volume names use `{{ project_name | replace('-', '_') }}_storage` etc. Wraps the `docker ps --format '{{ .Names }}'` lines in {% raw %}{% endraw %} so copier doesn't try to interpret the Go template syntax. - scripts/restore.sh.jinja: same volume fix + BACKUP_NAME regex validation before the SSH heredoc + INFISICAL_PROJECT_ID env requirement. - ansible/deploy.yml.jinja added (Path C app layer): uploads compose + Caddyfile + backup.sh, renders daily cron + logrotate, pulls images, reconciles compose stack, waits for health, then updates DO tags (skinny-backup + commit-<sha>) via the shared helper. - copier.yaml: adds ssh_source_cidrs variable (json type). - README.md.jinja: rewrote Quick Start for Path C workflow. - .gitignore: stop ignoring .terraform.lock.hcl. s004-deployment/ → anythingllm-docker/sites/s004/: - `git mv` preserves history per file. Then overlaid the freshly-rendered template output (via `copier copy --data-file s004-answers.yaml`) so the site reflects exactly what the migrated template produces today, with underscore-consistent paths (/opt/s004_anythingllm/, s004_anythingllm_* volumes) that were previously inconsistent. - All cross-references in INFRA_BOOTSTRAP_PATTERN.md + the 5 *-docker READMEs updated from s004-deployment/... → anythingllm-docker/sites/s004/... Auto DO tagging (new): - scripts/tag-droplet.sh — shared helper for tag mutation. Subcommands: add, remove, replace-prefix, set-commit, list. Chainable in a single call. Resolves droplet name → ID via doctl, mutates tags via `doctl compute droplet-action tag/untag --wait`. Idempotent. - scripts/bootstrap-otel-agent.sh: after each successful bootstrap, looks up the droplet name from the target IP and adds the `otel` tag. - scripts/deploy-otel-fleet.sh: same — adds `otel` after each successful deploy. - anythingllm-docker/ansible/configure-allm.yml: after MCP config changes successfully apply, looks up the droplet from inventory_hostname and adds the `searxng-mcp` tag. - anythingllm-docker/template/ansible/deploy.yml.jinja: on every deploy, invokes tag-droplet.sh with `replace-prefix commit- commit-<sha> add skinny-backup`. Restoring "the last working deployment" becomes read the tag, `git checkout <sha>`, re-run deploy.sh. - terraform/main.tf.jinja `ignore_changes = [tags]` so feature/state tags added at runtime aren't reverted by subsequent `tofu apply` runs. Documentation (docs/INFRA_BOOTSTRAP_PATTERN.md): - New "DO tag taxonomy" section documenting the three layers: project tags (terraform-set), feature tags (script-driven — otel, skinny-backup, searxng-mcp), state tags (commit-<sha>, replaced each deploy). - Project migration status table updated: anythingllm-docker promoted to reference implementation; wordpress now priority 1 follow-up. Verified the migrated template renders cleanly via `copier copy` — produces the 17 files that match anythingllm-docker/sites/s004/ in structure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(anythingllm-docker): Copilot round-4 findings on PR #31 ab65db6 Fixes 7 real findings from Copilot's review of the migration commit (other ~15 were stale — looking at pre-overlay file state or pointed at the deferred Layer 2 limitation already documented). Each fix applied to the TEMPLATE; `sites/s004/` then re-rendered via `copier copy` to keep both in sync. Real fixes: - `template/docker/Caddyfile.jinja`: was `output stdout` while the compose service bind-mounts `/var/log/caddy:/var/log/caddy` — the mount was useless. Switched to `output file /var/log/caddy/{{ project_name }}.log` so the file lands on the host where otel-agent's `filelog/caddy` receiver can read it (and survives container recreation). - `template/README.md.jinja`: backup-storage and migration-procedure examples used `/opt/{{ project_name }}/backups/` (hyphenated, e.g. `/opt/s004-anythingllm/backups/`) but the actual deployment path is `/opt/{{ project_name | replace('-', '_') }}/backups/` (underscore). Updated to underscore form everywhere paths appear. - `template/README.md.jinja`: restore example was `anythingllm-ai_backup_20260115_120000` (stale placeholder from the pre-migration template). Updated to `{{ project_name }}_backup_…`. - `template/scripts/backup.sh.jinja`: remote-mode ran `ssh "$host" "$BACKUP_CMDS"` directly — `SPACES_ACCESS_KEY` / `SPACES_SECRET_KEY` were NOT in the inner shell's env, so the `aws s3 cp` upload step would silently no-op. Wrapped the remote bash in `infisical run --projectId=… --env=…` (sources `/opt/<project>/.infisical-auth.env` the same way restore.sh does). Added `INFISICAL_PROJECT_ID:?` guard so remote mode fails fast. - `template/terraform/templates/cloud-init.yaml.jinja`: re-added `awscli` to the packages list. The slim cloud-init had dropped it during the Path C refactor, but the daily backup cron uses `aws s3 cp` so without awscli the first cron tick fails. - `template/scripts/deploy.sh.jinja`: `ansible-galaxy collection install community.docker` was unpinned — operators on different workstations could end up with different versions. Pinned to `==3.13.0` with a precise version check. Re-rendered `anythingllm-docker/sites/s004/` via `copier copy anythingllm-docker/ <tmp> --data-file s004-answers.yaml --defaults --trust`, then overlaid the rendered output onto sites/s004/. Stale/not-fixed comments (acknowledged): - Several "volume name mismatch" comments were on the OLD code — current scripts use `s004_anythingllm_storage` matching compose. - `terraform/main.tf` "Infisical secret in user_data → tfstate" is the deferred Layer 2 limitation (rotation runs at first boot; see docs/INFRA_BOOTSTRAP_PATTERN.md "What Layer 2 does NOT solve"). - `terraform/versions.tf` "no remote state backend" — backend IS in backend.tf (separate file). Terraform merges multiple `terraform {}` blocks across files. False positive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(anythingllm-docker): Copilot round-5 — fix HCL blocker + 8 other real issues Resolves the 9 actual bugs from Copilot's review of `3e312a4` (other ~19 comments triaged as either stale or false-positives — captured below). CRITICAL — tofu blocker: - `template/terraform/variables.tf.jinja:58` (and tfvars.example): the `ssh_source_cidrs` default was `{{ ssh_source_cidrs }}`, which Copier renders as Python's list-repr `['0.0.0.0/0', '::/0']` (single quotes) — NOT valid HCL. `tofu plan` would error on "Invalid character". Switched to `{{ ssh_source_cidrs | tojson }}` so Copier emits the JSON form with double quotes (which HCL parses as a list). Files-not-being-rendered-by-Copier blocker: - `_templates_suffix` defaults to `.jinja` in Copier, so files without that suffix in template/ were copied as-is — including their `{{ ... }}` template syntax. The rendered s004 site had unrendered placeholders in `.gitignore`, `terraform/versions.tf`, and `terraform/terraform.tfvars.example`. Renamed all three to add the `.jinja` suffix. Real correctness bugs: - `anythingllm-docker/ansible/configure-allm.yml`: the tag step did `ip="{{ inventory_hostname }}"` then looked up the droplet by PublicIPv4. With the documented `-i 'root@<ip>,'` inventory pattern, `inventory_hostname` is `root@<ip>`, not `<ip>`, so the doctl lookup always failed silently. Added `ip="${target##*@}"` to strip the prefix. - `template/scripts/backup.sh.jinja` remote-mode: was `bash -c '$BACKUP_CMDS'` where BACKUP_CMDS contains the literal `'table {{.Names}}...'` directives. Outer single quotes break on the inner single quotes. Restructured to invoke the droplet's own `/opt/<project>/backup.sh` (uploaded by ansible) inside `infisical run`. - `template/scripts/restore.sh.jinja` remote-mode: same quoting issue + sourced `/opt/<project>/infisical-auth.sh` (a script that doesn't exist; cloud-init writes `.infisical-auth.env`). Restructured the same way: source `.infisical-auth.env`, run `infisical login`, then exec `/opt/<project>/restore.sh "$BACKUP_NAME"` under `infisical run`. - `template/terraform/main.tf.jinja:36` tag-comment listed `anythingllm/ansible/configure-allm.yml` (no such file); the playbook lives at `anythingllm-docker/ansible/configure-allm.yml`. Fixed. - `template/docker/compose.prod.yaml.jinja`: `LLM_PROVIDER` and `VECTOR_DB` were hardcoded to "openrouter"/"lancedb" while the template still prompts for `llm_provider` and `vector_db` Copier vars (and passes them through Terraform). Those Copier knobs were dead. Replaced with `{{ llm_provider }}` / `{{ vector_db }}`. Stale/false-positive comments (verified, not fixed): - Caddyfile/compose hardcoded `s004.ccc.bot` — correct for the SITE. - main.tf:25 "Infisical secret in user_data" — deferred Layer 2. - versions.tf:6 "no remote state backend" — false positive; backend IS in backend.tf (separate file). - CHANGELOG anythingllm_image — already `reg.mini.dev/anythingllm:1.7.2`. - cloud-init jq `$$V2` — terraform escape, becomes `$V2` post-templatefile inside single-quoted jq script (jq's `--arg V2` definition). - Volume name mismatch comments — scripts already use the explicit `s004_anythingllm_storage` matching compose `name:` declarations. Re-rendered sites/s004/ from the updated template via copier; all 9 fixes verified in the rendered output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pr31): independent-review findings — restore auto-tagging + harden rotation script Final pre-merge pass from an independent code-review agent caught one HIGH bug that broke the auto-tagging feature this PR was meant to add, plus three LOW hardenings. Applied to the template and re-rendered sites/s004/. HIGH — auto-tagging was silently dead-on-arrival: - `template/ansible/deploy.yml.jinja:202` used `playbook_dir | dirname | dirname | dirname` to find scripts/tag-droplet.sh. But the layout is: <repo-root>/anythingllm-docker/sites/<name>/ansible/deploy.yml So three dirname operations land at <repo-root>/anythingllm-docker/ — not the repo root. With `failed_when: false`, the broken tagging step never failed, so `skinny-backup` and `commit-<sha>` tags silently never appeared on droplets. Added one more `| dirname` to walk to the repo root. The sibling task in anythingllm-docker/ansible/configure-allm.yml is one level shallower and was already correct (dirname × 2). LOW — rotation-log hardening: - cloud-init.yaml.jinja was logging the full Infisical API response on v2-mint failure. Replaced with `jq -r '.message // .error // .'` (head -c 500) so any future API change that echoes request fragments doesn't persist the bearer token. - Pre-chmod the rotation log to 0600 BEFORE writing so it's never world-readable even on failure paths. LOW — config knob clarity: - `INFISICAL_HOST="$${INFISICAL_HOST:-https://app.infisical.com}"` looked overridable, but cloud-init runcmd: has a minimal env so the override couldn't actually be set externally. Dropped the indirection and inlined the SaaS URL. LOW — tag-droplet.sh duplicate-name safety: - Previously did `awk … {print $2; exit}` — first-match-wins on duplicate droplet names. Now counts matches and errors out cleanly if 0 or >1. Reviewer findings explicitly NOT fixed (purely cosmetic): - U+21BA arrow in usage text (Finding 6) - Per-project README status drift risk (Finding 5) Re-rendered sites/s004/ via copier; HIGH fix verified in rendered ansible/deploy.yml, LOW fixes verified in sites/s004/terraform/templates/cloud-init.yaml. This is the merge-ready state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pr31): Copilot round-6 — 3 real bugs + 1 cosmetic Triaged 25 Copilot comments on `0993c43`. Verified ~20 as stale (Copilot re-flagged from older state — current code is correct). 4 actionable: REAL (would break first deploy): - `template/ansible/deploy.yml.jinja:156` — was using `community.docker.docker_image`, which requires the python3-docker Python SDK on the target host. The slim cloud-init only installs Docker itself, not the SDK. Switched to `community.docker.docker_image_pull` (CLI-based — no SDK needed). Available since community.docker 3.4; we pin 3.13.0. REAL (silently-broken config knob): - `template/scripts/backup.sh.jinja` + `restore.sh.jinja` — were hardcoding `REMOTE_STORAGE="do-spaces"` while `copier.yaml` offers …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Automated Pull Request — authored by
weown-bot(ecosystem service account)Opened by: @ncimino
Last pushed by: @ncimino
Branch:
fix/nik-version-consistency→mainContributors on this branch:
📋 Human Review Checklist — NIST CSF 2.0 Functions
Review per the 6 NIST CSF Functions. Frameworks referenced: NIST CSF 2.0, CIS Controls v8 IG1, CSA CCM v4, ISO/IEC 27001:2022, SOC 2, ISO/IEC 42001:2023. See
docs/COMPLIANCE_ROADMAP.md.🏛️ Govern (GV)
.github/CODEOWNERS)🔍 Identify (ID)
.github/SECURITY_ASSESSMENT.md)🛡️ Protect (PR)
--from-literal, never/tmp, always$(mktemp)— ISO A.8.24)restricted(NIST PR.IP, CIS 4)🕵️ Detect (DE)
livenessProbe+readinessProbe) configured🚨 Respond (RS)
.github/INCIDENT_RESPONSE.md)♻️ Recover (RC)
📚 Documentation & Versioning
CHANGELOG.mdupdated (per-directory or repo-level/CHANGELOG.md)#WeOwnVerversion bumped perdocs/VERSIONING_WEOWNVER.md📝 Recent Commits (full bodies for Copilot context)
5cb0bf2 fix(ci): split long error message to resolve yamllint line-length warning
Author: romandidomizio
Date: Wed May 13 13:43:00 2026 -0600
91b059a fix(ci): align drifted chart versions; tighten CHANGELOG consistency check
Author: Nik
Date: Wed May 13 01:00:51 2026 -0600
The version-consistency check in .github/workflows/validation.yml used
grep -qi $chart_version $changelog_file(presence-anywhere), whichsilently passed even when Chart.yaml drifted away from the latest
CHANGELOG entry. Concrete case discovered during PR #19 review:
wordpress/helm/Chart.yaml was 3.2.6 while wordpress/CHANGELOG.md's top
entry was [3.3.8] — the older 3.2.6 string still appeared in past
entries, so the check kept passing.
Tighten the check to compare Chart.yaml's
version:against the first## [X.Y.Z]header in CHANGELOG.md (the Keep-a-Changelog newest-on-topconvention).
Three charts had pre-existing drift and would fail the new check on
main, so align them in this commit:
nextcloud/helm/Chart.yaml 1.0.0 → 1.1.0
vaultwarden/helm/Chart.yaml 1.0.0 → 1.3.1
wordpress/helm/Chart.yaml 3.2.6 → 3.3.8
These are metadata-only bumps: Chart.yaml now matches the version each
chart's own CHANGELOG already declares as latest. No template or values
changes.
Not addressed in this commit (separate decisions needed):
(SEASON.MONTH.WEEK.ITERATION) and validation.yml's
versioningjob(SEASON.WEEK.DAY.VERSION). None of the current chart versions cleanly
fit either schema.
versioningjob only validates the first Chart.yaml foundalphabetically (
*/helm/Chart.yaml | head -1on line 212), not allof them.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
🔍 Copilot AI Review: Copilot is configured to auto-request review for bot-authored PRs. If an auto-created PR opens without an initial Copilot review, push a follow-up commit to the same open PR (
review_on_push: true) to trigger review automatically.👥 Required Reviewers: 1 human approval enforced by branch protection. requested automatically.
📚 Review Guidelines:
.github/copilot-instructions.md(phase-aware compliance directives)🛠️ Workflow Operations:
.github/workflows/README.mdAuto-generated by
.github/workflows/auto-pr-to-main.yml