Skip to content

feat(authentik): adopt live SSO config as an OpenTofu stack - #1458

Merged
Aviator-Coding merged 13 commits into
mainfrom
fm/homeops-authentik-terraform
Aug 27, 2026
Merged

feat(authentik): adopt live SSO config as an OpenTofu stack#1458
Aviator-Coding merged 13 commits into
mainfrom
fm/homeops-authentik-terraform

Conversation

@Aviator-Coding

Copy link
Copy Markdown
Owner

Intent

Bring the reference repo's (joryirving/home-ops terraform/authentik + terraform/tofu.md) Terraform-managed Authentik pattern into this repo as an OpenTofu stack managing our LIVE Authentik instance's configuration as code. Scope is the authentik stack ONLY (not garage/uptimerobot). Files use .tofu, two-space indent, lowercase_underscore names, typed+described variables, pinned ~> provider versions, plus a terraform/tofu.md-style guide adapted to our conventions.

HARD SAFETY GATE (still in force): our Authentik is the live SSO for the whole cluster including the public gateway's ExtAuth. NO tofu apply, and no write-capable Authentik mutation, without explicit current captain approval. A green PR/CI and a clean plan are NOT that approval. tofu plan and state inspection are read-only and fine.

Delivered: (1) terraform/authentik/ mirroring the reference's file organization; (2) read-only inventory of the live instance done FIRST via CNPG psql (4 apps, 4 providers incl. the forward-auth proxy behind ExtAuth; all 14 flows/18 stages/14 policies/36 property mappings are authentik-blueprint-managed and referenced as data sources only, never converted to resources); (3) every resource paired with an import block so adoption never plans a destroy/recreate; client_secret deliberately NOT declared (optional+computed, so import adopts the live value and no plan can rotate it), property_mappings declared explicitly on every provider (plain optional list, omitting it would remove live scopes); (4) state backend on in-cluster Ceph RGW (chosen over the external nas MinIO), hand-made bucket not an ObjectBucketClaim because ceph-bucket is reclaimPolicy Delete, bucket documented as private since state holds client secrets in plaintext; (5) secrets via 1Password, no plaintext secrets in git, nothing under resources/ paths; Renovate needs no customManager because the built-in terraform manager already matches .tofu files, and an overrides.json5 rule pins the provider to the Authentik server release line; (6) CI: path-filtered terraform job in .github/workflows/validate.yaml via scripts/ci/tofu-validate.sh running tofu fmt -check + tofu validate with -backend=false and NO credentials, so CI can never reach live state; (7) docs/authentik/terraform.md runbook with inventory, import strategy, bucket bootstrap, and the apply procedure behind its captain-approval gate.

Later captain decisions incorporated: authorization to mint a READ-ONLY Authentik credential in-cluster (read-only ONLY, nothing write-capable, no tofu apply). Done via ak shell (Django ORM, not hand-written Postgres rows): service account tofu-readonly in the authentik Read-only group, whose role was measured first (104 model permissions, all view_ only, no object permissions, not superuser) and confirmed against the live API (reads 200, POST and PATCH both 403). Key made durable by a PushSecret into 1Password Automation/authentik-terraform through a dedicated single-vault SecretStore, because the shared onepassword ClusterSecretStore lists three vaults by priority and a write resolves through that ordering rather than to a named vault. Vault is Automation, not Home-Lab: 1Password Connect can only see Homelab/Automation/Services and cannot see hyphenated Home-Lab at all, so an item there can never be machine-maintained. Deliberately NOT a self-healing CronJob like grafana-sa-provisioner, whose rationale (Grafana loses its SA on every restart via emptyDir SQLite) does not apply since Authentik persists tokens in Postgres.

Plan evidence obtained: 9 to import, 0 to add, 4 to change, 0 to destroy. All 4 changes are property_mappings list ORDERING with zero net membership change, verified by set comparison per resource; the proxy case shows all 5 as additions only because its read has an "if len(localMappings) > 0" guard that skips the attribute during import. The live plan also surfaced 3 real defects that tofu validate could not, all now fixed: the certificate data source defaulted fetch_certificate/fetch_key to true and was storing the PEM and the PRIVATE KEY in state (and view_certificate is denied to the read-only role, breaking the plan outright); redirect_uri_type must be declared because the API defaults and returns it; client_id must not be marked sensitive because it is a public OAuth2 identifier and the marking created a permanent phantom change on the one attribute whose real change would break every login.

Acceptance: CI green; PR body must contain the read-only inventory summary, the import strategy, and the tofu plan evidence showing no destructive changes to live objects.

What Changed

  • Adds terraform/authentik/ OpenTofu stack (.tofu layout, Ceph RGW state backend, 1Password vals secrets) that adopts the live Authentik hand-made apps/providers via permanent import blocks, leaving blueprint-managed flows/stages/policies/mappings as data sources only; documents conventions in terraform/tofu.md and the captain-gated apply runbook in docs/authentik/terraform.md.
  • Wires a read-only tofu-readonly plan credential path: dedicated Automation-vault SecretStore + PushSecret for authentik-terraform, Renovate pin keeping the goauthentik/authentik provider on the server release line, and a path-filtered Validate terraform job (tofu fmt -check + tofu validate -backend=false, no credentials).
  • Records operator safety in AGENTS.md (no tofu apply without explicit approval; CI cannot prove plan safety) and labels the new tree area/terraform.

Read-only inventory (live instance, 2026-08-26)

Hand-created only (resources owned by this stack):

Application Slug Provider pk Kind
Coder coder 4 OAuth2
open-webui open-webui 36 OAuth2
pgAdmin pg-admin 2 OAuth2
echo echo 37 Proxy (ExtAuth / sklab-externel-auth-provider)

Blueprint-managed and referenced as data sources only (not converted to resources): 14 flows, 18 stages, 14 policies, 36 property mappings, plus stock groups/brand/certs/source/embedded outpost (28 successful blueprintinstance rows).

Import strategy

Every declared resource is paired with an import block in terraform/authentik/imports.tofu so adoption never plans create/destroy of live SSO objects. IDs came from read-only CNPG psql (provider pks / app slugs / outpost-uuid:provider-pk). Load-bearing schema choices: do not declare client_secret (optional+computed; import keeps the live value); always declare property_mappings (optional list; omit would strip scopes); leave grant_types undeclared (optional+computed stock set).

tofu plan evidence (read-only, 2026-08-26)

Plan: 9 to import, 0 to add, 4 to change, 0 to destroy.
  • 0 add / 0 destroy — all nine live objects adopted; no destructive changes.
  • All 4 changes are property_mappings list ordering only, zero net membership (set comparison per resource). Proxy shows five “additions” only because import-time read skips mappings when local state is empty; live API set is identical.
  • Plan-surfaced fixes already in tree: certificate data source fetch_certificate/fetch_key = false (avoid PEM/private key in state; read-only role cannot view_certificate); declare redirect_uri_type; do not mark public client_id sensitive (avoids permanent phantom diff).

Hard gate unchanged: no tofu apply / write-capable Authentik mutation without explicit current captain approval. Green CI and a clean plan are not that approval.

Risk Assessment

✅ Low: Prior apply-path defects are corrected into a coherent fail-closed captain-approved sequence; adoption remains import-only with measured non-destructive plan shape and CI cannot reach live SSO.

Testing

Ran the real CI OpenTofu validator successfully against terraform/authentik, then a new semantic stack test that exercises that script, parses the Validate workflow and HCL into structured models, and checks Automation-only secrets plus the runbook’s inventory/import/non-destructive plan evidence; all checks passed offline. Live tofu plan was not re-run here (no Authentik/RGW credentials; hard safety gate), so plan proof is the runbook’s measured adoption plan artifact rather than a fresh API session.

Evidence: CI tofu-validate.sh success

Success! The configuration is valid. OK: 1 OpenTofu stack(s) formatted and valid

==> terraform/authentik

Initializing provider plugins...
- Reusing previous version of goauthentik/authentik from the dependency lock file
- Using previously-installed goauthentik/authentik v2026.5.1

OpenTofu has been successfully initialized!

You may now begin working with OpenTofu. Try running "tofu plan" to see
any changes that are required for your infrastructure. All OpenTofu commands
should now work.

If you ever set or change modules or backend configuration for OpenTofu,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Success! The configuration is valid.
OK: 1 OpenTofu stack(s) formatted and valid
Evidence: tofu-authentik-stack-test.py full pass report
PASS  tofu_validate_script
PASS  workflow_terraform_job
PASS  stack_hcl_model
PASS  secrets_vals_automation_only
PASS  pushsecret_single_vault
PASS  runbook_acceptance_surface
---
{
  "ok": true,
  "results": {
    "pushsecret_single_vault": {
      "detail": {
        "push_store": [
          {
            "kind": "SecretStore",
            "name": "onepassword-automation"
          }
        ],
        "remote_keys": [
          "authentik-terraform",
          "authentik-terraform",
          "authentik-terraform",
          "authentik-terraform"
        ],
        "vaults": {
          "Automation": 1
        }
      },
      "status": "pass"
    },
    "runbook_acceptance_surface": {
      "detail": {
        "plan_evidence": "Plan: 9 to import, 0 to add, 4 to change, 0 to destroy.",
        "sections_present": [
          "## 1. The inventory this code was written from",
          "## 2. Import strategy",
          "## 6. Planning",
          "## 7. Applying"
        ]
      },
      "status": "pass"
    },
    "secrets_vals_automation_only": {
      "detail": {
        "secrets-apply.vals.yaml": {
          "keys": [
            "AWS_ACCESS_KEY_ID",
            "AWS_SECRET_ACCESS_KEY",
            "TF_VAR_authentik_token",
            "TF_VAR_coder_client_id",
            "TF_VAR_open_webui_client_id",
            "TF_VAR_pgadmin_client_id"
          ],
          "token_field": "ref+op://Automation/authentik-terraform/AUTHENTIK_APPLY_TOKEN"
        },
        "secrets.vals.yaml": {
          "keys": [
            "AWS_ACCESS_KEY_ID",
            "AWS_SECRET_ACCESS_KEY",
            "TF_VAR_authentik_token",
            "TF_VAR_coder_client_id",
            "TF_VAR_open_webui_client_id",
            "TF_VAR_pgadmin_client_id"
          ],
          "token_field": "ref+op://Automation/authentik-terraform/AUTHENTIK_TOKEN"
        }
      },
      "status": "pass"
    },
    "stack_hcl_model": {
      "detail": {
        "backend_bucket": "terraform-state",
        "imports": {
          "authentik_application.echo": "echo",
          "authentik_application.oauth2[\"coder\"]": "coder",
          "authentik_application.oauth2[\"open-webui\"]": "open-webui",
          "authentik_application.oauth2[\"pg-admin\"]": "pg-admin",
          "authentik_outpost_provider_attachment.forward_auth": "a827266f-21ed-4a8b-a080-7b59a75a042e:37",
          "authentik_provider_oauth2.oauth2[\"coder\"]": "4",
          "authentik_provider_oauth2.oauth2[\"open-webui\"]": "36",
          "authentik_provider_oauth2.oauth2[\"pg-admin\"]": "2",
          "authentik_provider_proxy.forward_auth": "37"
        },
        "provider_version": "~> 2026.5.1",
        "resources": [
          "authentik_application.echo",
          "authentik_application.oauth2[\"coder\"]",
          "authentik_application.oauth2[\"open-webui\"]",
          "authentik_application.oauth2[\"pg-admin\"]",
          "authentik_outpost_provider_attachment.forward_auth",
          "authentik_provider_oauth2.oauth2[\"coder\"]",
          "authentik_provider_oauth2.oauth2[\"open-webui\"]",
          "authentik_provider_oauth2.oauth2[\"pg-admin\"]",
          "authentik_provider_proxy.forward_auth"
        ]
      },
      "status": "pass"
    },
    "tofu_validate_script": {
      "detail": {
        "ok": true,
        "stdout_tail": "any changes that are required for your infrastructure. All OpenTofu commands\nshould now work.\n\nIf you ever set or change modules or backend configuration for OpenTofu,\nrerun this command to reinitialize your working directory. If you forget, other\ncommands will detect it and remind you to do so if necessary.\nSuccess! The configuration is valid.\nOK: 1 OpenTofu stack(s) formatted and valid"
      },
      "status": "pass"
    },
    "workflow_terraform_job": {
      "detail": {
        "job": "terraform",
        "path_filter": [
          "terraform/**",
          ".github/workflows/validate.yaml",
          ".mise.toml",
          "scripts/ci/tofu-validate.sh"
        ],
        "runs": "./scripts/ci/tofu-validate.sh"
      },
      "status": "pass"
    }
  }
}
Evidence: Runbook inventory summary (4 apps / blueprint-managed counts)
## 1. The inventory this code was written from

Taken read-only on 2026-08-26, before a line of OpenTofu was written. The code
adopts what is here; it does not propose a different configuration.

### Applications and providers, all hand-created

| Application | Slug | Provider | Provider kind | Notes |
| ----------- | ---- | -------- | ------------- | ----- |
| Coder | `coder` | 4 | OAuth2 | explicit-consent authorization flow |
| open-webui | `open-webui` | 36 | OAuth2 | only app with a launch URL and a logout URI |
| pgAdmin | `pg-admin` | 2 | OAuth2 | two redirect URIs (`pgadmin.` and `pg.`) |
| echo | `echo` | 37 | Proxy | **the ExtAuth provider**, see below |

All four applications sit at `policy_engine_mode = "any"` with **zero policy
bindings**, so any authenticated user can reach any of them. That is the current
state, recorded here as fact. Introducing group-scoped access is a real
authorization change and belongs in its own reviewed change, not folded into
adoption.

### The ExtAuth provider, provider 37

Live name is `sklab-externel-auth-provider`, misspelling included. It runs in
`forward_domain` mode for `external_host = https://auth.sklab.dev` with
`cookie_domain = .sklab.dev` and `intercept_header_auth = true`.

It is bound to `authentik Embedded Outpost`
(`a827266f-21ed-4a8b-a080-7b59a75a042e`), and two Envoy Gateway SecurityPolicies
forward to that outpost's Service:

- `kubernetes/apps/base/network/echo/app/securitypolicy.yaml`
- `kubernetes/apps/base/ai/agentgateway/app/policies/authentik-policy.yaml`
  (covers the agentgateway `internal` **and** `public` https listeners)

This is the single most dangerous object in the stack.

### Everything else is Authentik's own

| Object | Count | Owner |
| ------ | ----- | ----- |
| Flows | 14 | built-in blueprints (`default/flow-*.yaml`) |
| Stages | 18 | built-in blueprints |
| Policies | 14 | built-in blueprints |
| Property mappings | 36 | built-in blueprints (`system/providers-*.yaml`), all carry a `managed` marker |
| Groups | 2 | `authentik Admins`, `authentik Read-only` |
| Brands | 1 | `authentik-default`, stock |
| Certificates | 2 | self-signed + internal JWT, generated at first boot |
| Sources | 1 | `authentik Built-in` |
| Outposts | 1 | embedded, `managed = goauthentik.io/outposts/embedded` |

There are **no** hand-written flows, stages, policies or property mappings on
this instance. 28 `blueprintinstance` rows are `successful` and cover all of the
above, which is why the stack references them as data sources and owns none of
them.

### How the inventory was taken

Through read-only `SELECT`s against Authentik's Postgres database, not the API:

`` `bash
export KUBECONFIG=./kubeconfig
PRIMARY=$(kubectl -n database get pods -l 'cnpg.io/cluster=postgres-17,role=primary' \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n database exec "$PRIMARY" -c postgres -- \
  psql -d authentik -A -F'|' -c "select id, name from authentik_core_provider order by id;"
`` `

Two reasons. There was no Authentik API token to read with (see section 5), and
the database is the only place that yields the exact primary keys the import
blocks need. `client_secret` was deliberately never selected.
Evidence: Runbook import strategy
## 2. Import strategy

Every resource the stack declares is paired with an `import` block in
[`imports.tofu`](../../terraform/authentik/imports.tofu). Without them the first
plan would show nine creates, and applying that would mint new client secrets and
stand up a second forward-auth provider.

| Resource | Import ID | Format |
| -------- | --------- | ------ |
| `authentik_provider_oauth2.oauth2["coder"]` | `4` | numeric provider pk |
| `authentik_provider_oauth2.oauth2["open-webui"]` | `36` | numeric provider pk |
| `authentik_provider_oauth2.oauth2["pg-admin"]` | `2` | numeric provider pk |
| `authentik_provider_proxy.forward_auth` | `37` | numeric provider pk |
| `authentik_application.oauth2["coder"]` | `coder` | slug |
| `authentik_application.oauth2["open-webui"]` | `open-webui` | slug |
| `authentik_application.oauth2["pg-admin"]` | `pg-admin` | slug |
| `authentik_application.echo` | `echo` | slug |
| `authentik_outpost_provider_attachment.forward_auth` | `a827266f-...-7b59a75a042e:37` | `<outpost uuid>:<provider pk>` |

Each of those resources has a passthrough importer in the provider source, so the
ID is passed to the read call unchanged.

Three properties make the adoption non-destructive, and all three are load-bearing:

- **`client_secret` is not declared.** It is optional+computed, so after import
  OpenTofu carries the live secret in state and plans no change. Declaring it
  would require reproducing the current value exactly; getting that wrong rotates
  the secret and breaks every login for that app with no error at plan time.
- **`property_mappings` is declared on every provider.** It is a plain optional
  list, so omitting it would plan the *removal* of the scopes each provider has.
- **`grant_types` is not declared.** It is optional+computed and the live values
  are Authentik's stock set.

The import blocks are left in the tree permanently. OpenTofu ignores an import
block whose target is already in state, and keeping them means the stack can be
rebuilt from scratch if the state file is ever lost.
Evidence: Measured non-destructive tofu plan evidence

Plan: 9 to import, 0 to add, 4 to change, 0 to destroy.

### The adoption plan, as measured 2026-08-26

`` `
Plan: 9 to import, 0 to add, 4 to change, 0 to destroy.
`` `

**0 to add and 0 to destroy is the number that matters.** Nothing is created and
nothing is torn down; all nine live objects are adopted.

The four in-place changes are all the same thing, and all benign: `property_mappings`
list *ordering*, with **zero net membership change**. Verified per resource by
diffing the sets, not by eyeballing the diff:

| Resource | Change | Net membership |
| -------- | ------ | -------------- |
| `provider_oauth2["coder"]` | one id reordered | none |
| `provider_oauth2["open-webui"]` | one id reordered | none |
| `provider_oauth2["pg-admin"]` | one id reordered | none |
| `provider_proxy.forward_auth` | all 5 shown as additions | none, set identical to live API |

The proxy case looks alarming and is not. Its read has an explicit guard,
`if len(localMappings) > 0`, which skips populating `property_mappings` when
there is nothing in state yet - which is exactly the situation during an import.
So prior state reads as empty and the plan shows all five as additions. The live
API returns the same five ids; the sets were compared directly and are equal.

These ordering diffs disappear after the first apply, once state carries a local
ordering for `ListConsistentMerge` to preserve.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

🔧 **Rebase** - 1 issue found → auto-fixed ✅
  • ⚠️ AGENTS.md - merge conflict rebasing onto origin/main

🔧 Fix applied.
✅ Re-checked - no issues remain.

🔧 **Review** - 3 issues found → auto-fixed (3) ✅
  • 🚨 kubernetes/apps/base/security/authentik/app/secretstore-automation.yaml:16 - Namespaced SecretStore sets connectTokenSecretRef.namespace: security. ESO admission rejects that with "namespace not allowed with namespaced SecretStore" (ValidateSecretSelector: namespace must be unset for SecretStore; the store's own namespace is implied). Upstream 1Password SecretStore examples omit namespace. As written, onepassword-automation will not admit, so PushSecret authentik-terraform-credentials cannot push and the durability path for AUTHENTIK_TOKEN/client_ids is broken. Remove the namespace field (keep name/key only).
  • ⚠️ docs/authentik/terraform.md:371 - Section 5 states AUTHENTIK_TOKEN is read-only and "An apply needs a separate, write-capable credential, minted only when an apply is approved," but section 7's apply commands only run vals exec -f secrets.vals.yaml -- tofu apply, which injects the PushSecret-managed read-only token. The measured adoption plan still has 4 property_mappings writes, so apply needs PATCH and will 403 with that token. Document an apply-only injection that cannot be pushed into the durable read-only field (e.g. TF_VAR_authentik_token override / separate write vals file, plus do-not-put-write-token-in-authentik-terraform-credentials).
  • ⚠️ docs/authentik/terraform.md:360 - Pre-apply checklist item 1 requires a plan with "only the nine imports with no creates, no destroys, and no changes," but section 6 documents the live adoption plan as 9 to import / 4 to change (benign property_mappings ordering, including the proxy case). That checklist blocks the only measured first apply and contradicts "ordering diffs disappear after the first apply." Align the gate with section 6: allow the documented ordering-only changes (still forbid creates/destroys and any non-ordering diffs, especially on the proxy/outpost attachment).

🔧 Fix: Drop redundant SecretStore namespace; fix apply docs
2 issues (1 error, 1 warning) still open:

  • 🚨 docs/authentik/terraform.md:388 - Section 7's apply recipe still fails open onto the read-only token. It tells the operator to export TF_VAR_authentik_token=... then vals exec -f secrets.vals.yaml -- tofu ..., and claims "TF_VAR_authentik_token wins over the read-only value vals would otherwise inject." Measured against vals 0.46.0 in this repo: (1) without -i, parent env is not inherited at all; (2) with -i, parent keys are inherited only when the file does not set them - keys present in the file always win. secrets.vals.yaml sets TF_VAR_authentik_token to the durable read-only Automation item, so the documented export is ignored and apply still 403s on every PATCH. Working patterns: vals exec -i -f secrets.vals.yaml -- env TF_VAR_authentik_token=&#34;$WRITE_TOKEN&#34; tofu ... (env overrides after vals populates), or a never-committed write-only vals file that replaces the token and is never pushed into Automation/authentik-terraform. Keep the never-write-the-write-token-into-the-durable-field rule and the captain-approval gate.
  • ⚠️ docs/authentik/terraform.md:296 - Every vals exec -f secrets.vals.yaml -- tofu ... invocation in section 6, section 7, terraform/tofu.md, and terraform/authentik/secrets.vals.yaml omits -i. vals exec without -i runs the child with a stripped PATH (/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.) and no parent env, so the mise-provided tofu from .mise.toml is not found unless it happens to be on that minimal PATH. Add -i on all operator vals exec examples so PATH/mise and op session plumbing survive; this is independent of the token-override fix above (file keys still win even with -i).

🔧 Fix: Fail-closed apply vals; add vals -i
2 issues (1 error, 1 warning) still open:

  • 🚨 docs/authentik/terraform.md:401 - Section 7's post-approval recipe plans with secrets.vals.yaml (TF_VAR_authentik_token = durable read-only AUTHENTIK_TOKEN) then applies that saved plan with secrets-apply.vals.yaml (different TF_VAR_authentik_token = AUTHENTIK_APPLY_TOKEN). OpenTofu/Terraform 1.10+ refuse apply of a saved plan when any TF_VAR differs from the values baked into the plan ("Can't change variable when applying a saved plan"), so the documented captain-approved path cannot complete the first adoption apply. Fix: keep secrets.vals.yaml for read-only review plans with no -out; after AUTHENTIK_APPLY_TOKEN is placed, run both tofu plan -out=... and tofu apply &lt;planfile&gt; through secrets-apply.vals.yaml so the token matches. Still remove AUTHENTIK_APPLY_TOKEN afterwards; never write the write token into AUTHENTIK_TOKEN.
  • ⚠️ docs/authentik/terraform.md:401 - The apply recipe writes -out=tfplan and comments "gitignored", but terraform/.gitignore only has *.tfplan and scripts/ci/tofu-validate.sh only greps \.tfplan$. git check-ignore does not match bare terraform/authentik/tfplan, so a saved plan (embeds variable values / planned state, including secrets) can be committed unnoticed. Ignore the documented filename (e.g. tfplan and/or rename to *.tfplan) and extend the CI leak grep to match it.

🔧 Fix: Match saved-plan token; ignore bare tfplan
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • ./scripts/ci/tofu-validate.sh (tofu fmt -check + init -backend=false + validate; OpenTofu 1.12.6; no credentials)
  • uv run --with python-hcl2 --with pyyaml python3 scripts/ci/tofu-authentik-stack-test.py (workflow terraform job semantics, HCL resource/import model, vals secrets, PushSecret/SecretStore, runbook acceptance surface)
  • Manual evidence extraction of runbook inventory, import strategy, and measured plan (Plan: 9 to import, 0 to add, 4 to change, 0 to destroy.) into the evidence directory
  • Safety check: no tofu apply in CI scripts/workflows; no live Authentik mutation attempted
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@mortyops

mortyops Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor
--- kubernetes/apps/base/security/authentik/app Kustomization: security/authentik SecretStore: security/onepassword-automation

+++ kubernetes/apps/base/security/authentik/app Kustomization: security/authentik SecretStore: security/onepassword-automation

@@ -0,0 +1,22 @@

+---
+apiVersion: external-secrets.io/v1
+kind: SecretStore
+metadata:
+  labels:
+    app.kubernetes.io/name: authentik
+    kustomize.toolkit.fluxcd.io/name: authentik
+    kustomize.toolkit.fluxcd.io/namespace: security
+  name: onepassword-automation
+  namespace: security
+spec:
+  provider:
+    onepassword:
+      auth:
+        secretRef:
+          connectTokenSecretRef:
+            key: token
+            name: onepassword-secret
+      connectHost: http://onepassword-connect.security.svc.cluster.local:8080
+      vaults:
+        Automation: 1
+
--- kubernetes/apps/base/security/authentik/app Kustomization: security/authentik PushSecret: security/authentik-terraform-credentials

+++ kubernetes/apps/base/security/authentik/app Kustomization: security/authentik PushSecret: security/authentik-terraform-credentials

@@ -0,0 +1,40 @@

+---
+apiVersion: external-secrets.io/v1alpha1
+kind: PushSecret
+metadata:
+  labels:
+    app.kubernetes.io/name: authentik
+    kustomize.toolkit.fluxcd.io/name: authentik
+    kustomize.toolkit.fluxcd.io/namespace: security
+  name: authentik-terraform-credentials
+  namespace: security
+spec:
+  data:
+  - match:
+      remoteRef:
+        property: AUTHENTIK_TOKEN
+        remoteKey: authentik-terraform
+      secretKey: AUTHENTIK_TOKEN
+  - match:
+      remoteRef:
+        property: CODER_CLIENT_ID
+        remoteKey: authentik-terraform
+      secretKey: CODER_CLIENT_ID
+  - match:
+      remoteRef:
+        property: OPEN_WEBUI_CLIENT_ID
+        remoteKey: authentik-terraform
+      secretKey: OPEN_WEBUI_CLIENT_ID
+  - match:
+      remoteRef:
+        property: PGADMIN_CLIENT_ID
+        remoteKey: authentik-terraform
+      secretKey: PGADMIN_CLIENT_ID
+  refreshInterval: 1h
+  secretStoreRefs:
+  - kind: SecretStore
+    name: onepassword-automation
+  selector:
+    secret:
+      name: authentik-terraform-credentials
+

Adopts the four applications and four providers that exist on the live
instance today, including the forward-auth proxy provider behind the
cluster's ExtAuth, as OpenTofu resources.

Every resource is paired with an import block so the first plan
reconciles against the real object instead of proposing to create a
second one. Client secrets are deliberately not declared: they are
optional+computed in the provider schema, so import adopts the live
value and no plan can silently rotate them.

Flows, stages, scope mappings, groups, certificates and the brand are
referenced as data sources only. All of them are reconciled by
Authentik's own built-in blueprints, so declaring them as resources
would put OpenTofu and the blueprint reconciler in a fight over the
same objects.

State lives in the in-cluster Ceph RGW object store, which is published
on the internal gateway only. Secrets are resolved from 1Password by
vals at invocation time; no credential is committed.

No apply has been run. This is code and an import strategy only.
Adds a path-filtered `terraform` job on the local ARC runner running
`tofu fmt -check` plus `tofu validate` over every stack under
terraform/, through scripts/ci/tofu-validate.sh.

The job is credential-free by construction: `tofu init` runs with
-backend=false and `tofu validate` never calls the Authentik API, so a
pull request can reach neither the live SSO instance nor the state
bucket. The script also refuses to run if state or rendered tfvars are
ever committed, since state holds adopted client secrets in plaintext.

opentofu is pinned in .mise.toml so CI validates with the version the
runbook tells an operator to plan with.

Renovate needs no custom manager here: the built-in terraform manager
already matches **/*.tofu and maintains .terraform.lock.hcl. What it did
need is the coupling between the provider and the Authentik server it is
generated from, added to .renovate/overrides.json5.
Records the read-only inventory the code was written from, the import
strategy and its exact IDs, why the state bucket is a hand-made Ceph RGW
bucket rather than an ObjectBucketClaim, the 1Password item shape, and
the apply procedure with its approval gate.

Also states plainly what could not be settled without an API token: the
handful of attributes whose plan behaviour depends on how the provider
normalises empty strings. None is destructive, all are listed so the
first real plan is read against a written expectation rather than
interpreted on the spot.

Adds terraform/tofu.md as the conventions guide for the tree and a short
README for the stack itself.
…AGENTS.md

Three facts that are not derivable from the code and are easy to undo by
accident: terraform/ is not Flux-reconciled and its CI is credential-free
so green does not mean apply-safe; almost nothing in our Authentik is
hand-made, so the stack owns 4 apps and 4 providers and references the
28 blueprint-managed objects as data sources; and the state bucket is a
hand-made RGW bucket rather than an ObjectBucketClaim because the
ceph-bucket StorageClass would delete it on a Flux prune.

Also adds the apply gate to ANTI-PATTERNS.
A terraform-only PR would otherwise get no area label. Also replaces a
stray em dash in AGENTS.md left by an earlier entry.
Only a real plan against the instance could find these; `tofu validate`
passes on all three.

The certificate data source defaulted `fetch_certificate` and `fetch_key`
to true, so it additionally called view_certificate/ and
view_private_key/ and stored the PEM and the PRIVATE KEY in state. Only
the certificate's ID is ever used. That is a leak into an
already-secret-bearing state file, and view_certificate/ is denied to the
read-only role the plan runs as, so it also broke the plan outright.

`redirect_uri_type` is returned by the API with a default of
"authorization" even though these three provider rows predate the field,
so omitting it made every plan propose removing it.

The client_id variables were marked sensitive. A client_id is a public
OAuth2 identifier, and Terraform renders a sensitivity-marked attribute
as `~ (sensitive value)` in an import plan even when the value is
identical - putting a phantom "change" on the one attribute whose real
change would break every login. Verified identical across DB, API and
1Password before unmarking.
Adds the durability half of the OpenTofu stack's credential path.

The token is a read-only Authentik service account (tofu-readonly, in the
authentik Read-only group), minted through the ak management shell so it
goes through the same ORM the UI does rather than hand-written rows. The
group was measured before use: 104 model permissions, none other than
view_*, no object permissions, not superuser. Confirmed against the live
API, where reads return 200 and both a create and an update return 403.
It can run plan and can never run apply.

A PushSecret keeps the 1Password item current from the in-cluster Secret
the mint writes. It pushes through a dedicated single-vault SecretStore
rather than the shared onepassword ClusterSecretStore: that store lists
three vaults with priorities, so a write resolves through the ordering
rather than to a vault you named and could land in Homelab.

Vault is Automation, not Home-Lab. 1Password Connect can only reach
Homelab/Automation/Services, verified by querying it directly, and
Home-Lab is a different vault from Homelab that Connect cannot see at
all, so an item there could never be machine-maintained.

Not a self-healing reconciler like grafana-sa-provisioner: that exists
because Grafana loses its service account on every pod restart, while
Authentik persists tokens in Postgres.
…edential

Home-Lab and Homelab are two different vaults and the difference decides
whether a secret can be machine-maintained at all: Connect's credential
sees only Homelab/Automation/Services, so nothing in hyphenated Home-Lab
can ever be read or written by an ExternalSecret or PushSecret. Cost a
full escalation cycle to discover, so it is worth stating once.

Also records that a write through the shared ClusterSecretStore has a
non-deterministic vault target, and how the tofu read-only credential is
minted and kept current.
@Aviator-Coding
Aviator-Coding force-pushed the fm/homeops-authentik-terraform branch from c9edec8 to 29d5925 Compare August 27, 2026 09:28
@Aviator-Coding
Aviator-Coding merged commit 4f3b3aa into main Aug 27, 2026
18 checks passed
@Aviator-Coding
Aviator-Coding deleted the fm/homeops-authentik-terraform branch August 27, 2026 09:41
Aviator-Coding added a commit that referenced this pull request Aug 27, 2026
…#1464)

* feat(github-action): add terraform CI workflows for terraform/* stacks

Adds terraform-publish.yaml (OCI-publishes terraform/ on push to main, mirroring
joryirving/home-ops) and the uncontested scaffold of terraform-diff.yaml
(per-changed-directory tofu fmt/init -backend=false/validate on PRs, matrixed
for any future terraform/* stack). The real plan-against-live-backend layer of
terraform-diff.yaml is intentionally deferred pending a captain decision on
whether PRs should be allowed to reach the live state bucket and Authentik API
at all - see the workflow's header comment and docs/authentik/terraform.md.

area/terraform labeler wiring already exists from #1458; no change needed.

* feat(github-action): run real read-only terraform plans on every PR

Captain decision (terraform-diff-live-plan-scope, option A, "least amount of
manual intervention"): terraform-diff.yaml now runs a real tofu init/plan
against the live state backend and, for authentik, the live Authentik API,
unattended on every same-repo PR touching terraform/**, posting the plan as a
PR comment. This consciously reverses the "CI can never reach the live
instance" boundary validate.yaml and docs/authentik/terraform.md previously
stated as deliberate - both are updated here to explain the new reality.

A stack opts in by providing its own secrets-ci.vals.yaml: a CI-facing mirror
of its secrets.vals.yaml, resolved via 1Password Connect (ref+onepasswordconnect://)
instead of the interactive op CLI (ref+op://), since terraform-diff.yaml has no
interactive session. terraform/authentik/secrets-ci.vals.yaml reuses the exact
same Automation/authentik-terraform item and fields secrets.vals.yaml already
resolves - it needs no new secret material, only a new resolution path. A stack
with no such file keeps getting schema-only checks (-backend=false).

tofu apply is unaffected: it stays fully behind the explicit captain-approval
sequence in docs/authentik/terraform.md section 7. The one new GitHub Actions
secret this requires (OP_CONNECT_TOKEN, a Connect access token scoped to the
Automation vault) and its vault-wide read scope are documented in
docs/authentik/terraform.md section 8 and in the PR description.

* docs(ai): update terraform CI note for terraform-diff.yaml's real plan

Project knowledge bullet was written for the -backend=false-only CI posture;
update it to describe terraform-diff.yaml's real read-only plan and PR comment
alongside validate.yaml's unchanged schema-only check.

* no-mistakes(review): Align terraform CI docs with live plans

* no-mistakes(review): Align leftover terraform CI doc contradictions

* no-mistakes(review): Ignore and scan terraform-diff .planfile artifacts

* no-mistakes(document): Align leftover terraform CI doc inventories

* no-mistakes: apply CI fixes
Aviator-Coding added a commit that referenced this pull request Aug 28, 2026
A namespaced SecretStore may only reference a Secret in its own
namespace; ESO's admission webhook rejects anything else outright. The
1Password Connect token lives in `security`, so the namespaced store
worked there by coincidence (namespace matched) and was rejected
immediately when ai/litellm needed the same push target.

Replaces both namespaced copies with one ClusterSecretStore next to the
existing 1Password store. It still exists for the same reason as before:
the shared `onepassword` store lists three vaults with priorities, which
is right for reads and wrong for writes, since a push through it resolves
by that ordering rather than to a vault you named.

This is the general form of a review finding from #1458, which flagged
the redundant namespace field. That finding was right about the rule and
wrong that the security store was broken; the rule only bites once a
second namespace needs the same store.
Aviator-Coding added a commit that referenced this pull request Aug 28, 2026
* feat(authentik): add LiteLLM OAuth2 provider and application

First resource in this stack that is created rather than adopted, so it
is the only place a plan should show an addition.

Credential handling is deliberately the inverse of the imported
providers. There, client_secret is left undeclared because declaring it
would rotate a live secret; here there is no live secret to protect and
LiteLLM has to be told what it is, so both halves are generated and
surfaced through outputs for the one hop OpenTofu cannot make itself.
Neither value is written into Git.

The redirect URI was read out of the running v1.98.0 image rather than
from documentation: ui_sso.py builds it as
get_custom_url(PROXY_BASE_URL) + "/sso/callback". The provider's scope
mappings are openid/email/profile, which is byte-for-byte LiteLLM's
default GENERIC_SCOPE, so a scope mismatch cannot be the cause of a
failed login.

* feat(ai): wire LiteLLM UI single sign-on through Authentik

Non-secret SSO configuration goes on the LiteLLMProxy CR as plain env so
it stays readable in Git; only the two credential halves pass through the
ExternalSecret.

The two sides are coupled and the coupling is load-bearing: LiteLLM
derives its callback as get_custom_url(PROXY_BASE_URL) + "/sso/callback",
so PROXY_BASE_URL here and the allowed redirect URI in
terraform/authentik/litellm.tofu must move together or every login fails
at the provider with an opaque redirect_uri error. Both files say so.

The client id and secret are generated by OpenTofu rather than typed by
hand, so the value on the Authentik provider and the value LiteLLM
presents cannot drift apart. Getting them from there to here needs one
hop OpenTofu cannot make itself: tofu output -> a hand-made Secret ->
PushSecret -> 1Password -> this ExternalSecret. The push targets a
dedicated single-vault SecretStore for the same reason as the Authentik
one: a write through the shared ClusterSecretStore resolves by vault
priority rather than to a vault you named.

Endpoints were taken from the live .well-known/openid-configuration.
authorize/token/userinfo are instance-global in Authentik; only issuer
and jwks carry the application slug.

* fix(authentik): point tofu state backend at RGW directly, not through Envoy

Measured 2026-08-27: every request from OpenTofu's S3 client through the
envoy-internal gateway fails with SignatureDoesNotMatch, while the same
credentials against the RGW Service succeed. This is a defect in the
merged backend.tofu, which pointed at https://s3.sklab.dev.

The cause is which headers get signed. aws-sdk-go-v2 puts
accept-encoding, amz-sdk-invocation-id and amz-sdk-request in
SignedHeaders; boto3 and minio-go sign none of them. Envoy rewrites one
of those in transit, so RGW recomputes a different signature. That is why
nothing caught it: VolSync (restic/minio-go) and every other S3 client
keep working through the same route, and a boto3 ListObjectsV2 against
this exact bucket returns 200 while tofu gets 403.

Also adds the two RGW state-key fields to the PushSecret. secrets.vals.yaml
already referenced them, so without this a tofu init against the real
backend could not authenticate.

* refactor(security): make the Automation push store cluster-scoped

A namespaced SecretStore may only reference a Secret in its own
namespace; ESO's admission webhook rejects anything else outright. The
1Password Connect token lives in `security`, so the namespaced store
worked there by coincidence (namespace matched) and was rejected
immediately when ai/litellm needed the same push target.

Replaces both namespaced copies with one ClusterSecretStore next to the
existing 1Password store. It still exists for the same reason as before:
the shared `onepassword` store lists three vaults with priorities, which
is right for reads and wrong for writes, since a push through it resolves
by that ordering rather than to a vault you named.

This is the general form of a review finding from #1458, which flagged
the redundant namespace field. That finding was right about the rule and
wrong that the security store was broken; the rule only bites once a
second namespace needs the same store.

* docs(authentik): record the first live apply and the credential paths

Documents what actually happened rather than what was planned: the apply
result, the four post-apply verification checks, the write credential and
the evidence it cannot delete, and the manual hop that carries a
generated credential from tofu output into the cluster.

Also records that the state backend is not reachable through the gateway
and needs a port-forward, with the header-signing reason, so the next
person does not rediscover it against a SignatureDoesNotMatch.

* docs(flux): record the applied tofu state and two traps it exposed

The stack now holds live state and has two least-privilege credentials,
so the AGENTS entry saying nothing had been applied was stale.

Two gotchas worth stating once because both fail silently. The tofu state
backend cannot be reached through the gateway: aws-sdk-go-v2 signs
headers Envoy rewrites, so RGW rejects the signature while every other S3
client in the cluster keeps working. And a kubectl invoked through the
mise shims inherits KUBECONFIG from .mise.toml, pointing at a file a
fresh worktree does not have, which with stderr suppressed shows up as
empty output rather than an error.

Also generalises the ESO write-store rule: the single-vault store is
cluster-scoped because a namespaced SecretStore may only reference a
Secret in its own namespace.

* no-mistakes(review): Ship litellm_role claim and SSO logout URL

* no-mistakes(review): Add LiteLLM logout flow; document blocked apply

* no-mistakes(document): Synced SSO docs; cleared stale SecretStore refs

* no-mistakes: apply CI fixes

* fix(authentik): use flow uuid, and stop managing the deleted open-webui

Two things the second live apply forced, neither catchable by validate.

authentik_flow's terraform id is the SLUG, so pointing a provider's
invalidation_flow at .id returns 400 "not a valid UUID". The stage
binding already used .uuid; the provider reference did not. Fixed and
documented, because the two attributes look interchangeable and are not.

open-webui was deleted from Authentik by the captain on 2026-08-27,
confirmed in the audit log: akadmin issued both DELETEs, while the only
tofu action that day was a PUT from tofu-writer, which holds no delete
permission at all. The deletion was intentional, so its resources, import
blocks and client-id variable are removed and the two instances dropped
from state, rather than letting the next plan recreate the application
with a client secret that would match nothing.

Records the write role now holding fourteen permissions including the
model-level change_flow, and why every future apply should be checked
against a before/after flow snapshot rather than terraform's own summary.

* docs(flux): record three OpenTofu/Authentik traps validate cannot catch

The flow id-vs-uuid distinction, the S3 backend silently falling back to
real AWS when AWS_ENDPOINT_URL_S3 is unset, and the standing obligation
to verify applies against a database snapshot now that the write role
holds model-level change_flow.

* no-mistakes(review): Align SSO docs and tests to delivered state

* no-mistakes(review): Drop open-webui PushSecret; fix SSO docs xrefs

* no-mistakes(review): Retarget CI xrefs; fix inventory counts

* no-mistakes(review): Fix imports header; drop runbook prose test

* no-mistakes(review): Retarget remaining CI docs xrefs to §9

* no-mistakes(document): Synced SSO docs to delivered Authentik state
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant