Skip to content

fix(security): enforce cosign verify on the live root OCIRepository - #3001

Merged
devantler merged 6 commits into
mainfrom
claude/root-oci-verify-2922-2
Aug 6, 2026
Merged

fix(security): enforce cosign verify on the live root OCIRepository#3001
devantler merged 6 commits into
mainfrom
claude/root-oci-verify-2922-2

Conversation

@devantler

@devantler devantler commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

The platform's root artifact — the one every controller, tenant binding and policy arrives through —
is pulled from a mutable tag with nothing checking who produced it. This turns signature verification
on for the running cluster.

An earlier version of this PR did that with a three-entry signer list, which cosign rejects
outright
: it fails closed for the whole set, so the control verified nothing and halted all GitOps
delivery on prod for about five and a half hours (#3005). That is fixed here, and the fix is proven
against the real artifact rather than reasoned about.

What

  • The signer list is collapsed to the one entry cosign supports, with the same three trusted
    publishers alternated inside a single pattern. No signer is added or removed.
  • The same correction is applied to the bootstrap and disaster-recovery copy, which carries the
    identical broken shape on main today — so a DR rebuild would currently fail the same way, at the
    worst possible moment.
  • The validator now fails the build on any signer list longer than one, in both halves, so the
    shape cannot come back silently.

Merge safety

The blocker that made this PR dangerous is resolved and independently verified: the artifact
currently live on latest is confirmed — from the public Sigstore transparency log, needing no
registry access — to have been signed by a publisher this matcher accepts. Evidence is in a comment
below rather than here.

Fixes #2922
Part of #3005

flux-operator owns ocirepository/flux-system and KSail writes no spec key on
it during a routine deploy, so spec.workload.flux.verify covered bootstrap
only and the live root source pulled unverified.

Have the operator write the field via spec.kustomize.patches, and extend
validate-flux-verify to assert the patch exists and targets the root source --
a mistargeted patch renders nothing, exits 0 and warns nobody.
…t signers

The policy is written twice -- bootstrap in the cluster config, running
cluster in the FluxInstance -- because neither covers the other's window.
Nothing compared them, so tightening one would silently leave the other
trusting a publisher it should not.

Also make the mistargeted-patch report independent of patch order, and name
only the differing signers instead of dumping both lists.
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

⚠️MegaLinter analysis: Success with warnings

⚠️ BASH / bash-exec - 5 errors
Results of bash-exec linter (version 5.3.9)
See documentation on https://megalinter.io/9.6.0/descriptors/bash_bash_exec/
-----------------------------------------------

✅ [SUCCESS] scripts/dr-rebuild-supersession-guard.sh
❌ [ERROR] scripts/ghcr-auth-lib.sh
    Error: File:[scripts/ghcr-auth-lib.sh] is not executable

✅ [SUCCESS] scripts/guard-cilium-homogeneous-device-rollout.sh
✅ [SUCCESS] scripts/megalinter-scan-counts.sh
✅ [SUCCESS] scripts/normalize-sarif-paths.sh
❌ [ERROR] scripts/refresh-flux-ghcr-auth-safety.sh
    Error: File:[scripts/refresh-flux-ghcr-auth-safety.sh] is not executable

✅ [SUCCESS] scripts/refresh-flux-ghcr-auth.sh
✅ [SUCCESS] scripts/report-cilium-rollout-gate-suppression.sh
✅ [SUCCESS] scripts/run-ksail-prod-with-pull-auth.sh
✅ [SUCCESS] scripts/summarize-sarif-findings.sh
✅ [SUCCESS] scripts/tests/test-cilium-bandwidth-manager-component.sh
✅ [SUCCESS] scripts/tests/test-cilium-homogeneous-devices-activation.sh
✅ [SUCCESS] scripts/tests/test-cilium-homogeneous-devices-autoscaler-gate.sh
✅ [SUCCESS] scripts/tests/test-cilium-homogeneous-devices-flux-wait.sh
✅ [SUCCESS] scripts/tests/test-cilium-mutual-auth-policy-regressions.sh
✅ [SUCCESS] scripts/tests/test-cilium-mutual-auth-policy.sh
✅ [SUCCESS] scripts/tests/test-cilium-rollout-gate-suppression-signal.sh
✅ [SUCCESS] scripts/tests/test-crossplane-sync-exporter.sh
✅ [SUCCESS] scripts/tests/test-dr-rebuild-supersession-guard.sh
✅ [SUCCESS] scripts/tests/test-kyverno-admission-vpa.sh
❌ [ERROR] scripts/tests/test-normalize-sarif-paths.sh
    Error: File:[scripts/tests/test-normalize-sarif-paths.sh] is not executable

✅ [SUCCESS] scripts/tests/test-openbao-oidc-role.sh
❌ [ERROR] scripts/tests/test-opencost-usage-scraper.sh
    Error: File:[scripts/tests/test-opencost-usage-scraper.sh] is not executable

❌ [ERROR] scripts/tests/test-refresh-flux-ghcr-auth-safety.sh
    Error: File:[scripts/tests/test-refresh-flux-ghcr-auth-safety.sh] is not executable

✅ [SUCCESS] scripts/tests/test-restrict-tenant-secret-stores.sh
✅ [SUCCESS] scripts/tests/test-summarize-sarif-findings.sh
✅ [SUCCESS] scripts/tests/test-verify-published-evidence.sh
✅ [SUCCESS] scripts/update-vendored-operators.sh
✅ [SUCCESS] scripts/validate-alert-coverage.sh
✅ [SUCCESS] scripts/verify-published-evidence.sh
✅ [SUCCESS] scripts/wait-for-platform-flux-revision.sh
⚠️ REPOSITORY / checkov - 19 errors
to review the file.
Check: CKV_K8S_8: "Liveness Probe Should be Configured"
	FAILED for resource: PodTemplate.overprovisioning.overprovisioning
	File: /k8s/providers/hetzner/infrastructure/overprovisioning/pod-template.yaml:29-86
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-7

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_49: "Minimize wildcard use in Roles and ClusterRoles"
	FAILED for resource: Role.github-config.github-config-managed-resources
	File: /k8s/bases/apps/github-config/role.yaml:20-55
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/ensure-minimized-wildcard-use-in-roles-and-clusterroles

		20 | apiVersion: rbac.authorization.k8s.io/v1
		21 | kind: Role
		22 | metadata:
		23 |   name: github-config-managed-resources
		24 |   namespace: github-config
		25 |   labels:
		26 |     app.kubernetes.io/managed-by: ksail
		27 | rules:
		28 |   - apiGroups:
		29 |       - repo.github.m.upbound.io
		30 |       - team.github.m.upbound.io
		31 |       - actions.github.m.upbound.io
		32 |       - enterprise.github.m.upbound.io
		33 |       - github.m.upbound.io
		34 |     resources:
		35 |       - "*"
		36 |     verbs:
		37 |       - get
		38 |       - list
		39 |       - watch
		40 |       - create
		41 |       - update
		42 |       - patch
		43 |       - delete
		44 |   - apiGroups:
		45 |       - external-secrets.io
		46 |     resources:
		47 |       - externalsecrets
		48 |     verbs:
		49 |       - get
		50 |       - list
		51 |       - watch
		52 |       - create
		53 |       - update
		54 |       - patch
		55 |       - delete

Check: CKV_K8S_35: "Prefer using secrets as files over secrets as environment variables"
	FAILED for resource: CronJob.umami.umami-provision-tenants
	File: /k8s/bases/apps/umami/cron-job.yaml:44-318
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-33

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_22: "Use read-only filesystem for containers where possible"
	FAILED for resource: CronJob.umami.umami-provision-tenants
	File: /k8s/bases/apps/umami/cron-job.yaml:44-318
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-21

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_9: "Readiness Probe Should be Configured"
	FAILED for resource: Deployment.observability.crossplane-sync-exporter
	File: /k8s/bases/infrastructure/coroot/components/crossplane-sync-exporter/deployment.yaml:1-111
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-8

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_8: "Liveness Probe Should be Configured"
	FAILED for resource: Deployment.observability.crossplane-sync-exporter
	File: /k8s/bases/infrastructure/coroot/components/crossplane-sync-exporter/deployment.yaml:1-111
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-7

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_40: "Containers should run as a high UID to avoid host conflict"
	FAILED for resource: Job.openbao.vault-config
	File: /k8s/bases/infrastructure/vault-config/job.yaml:36-1200
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-37

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_40: "Containers should run as a high UID to avoid host conflict"
	FAILED for resource: CronJob.openbao.vault-snapshot
	File: /k8s/bases/infrastructure/vault-backup/cron-job.yaml:23-197
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-37

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_40: "Containers should run as a high UID to avoid host conflict"
	FAILED for resource: Job.openbao.vault-snapshot-init
	File: /k8s/bases/infrastructure/vault-backup/job.yaml:23-191
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-37

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_9: "Readiness Probe Should be Configured"
	FAILED for resource: Deployment.opencost.opencost-usage-scraper
	File: /k8s/bases/infrastructure/opencost/components/usage-scraper/deployment.yaml:1-78
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-8

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_8: "Liveness Probe Should be Configured"
	FAILED for resource: Deployment.opencost.opencost-usage-scraper
	File: /k8s/bases/infrastructure/opencost/components/usage-scraper/deployment.yaml:1-78
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-7

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_38: "Ensure that Service Account Tokens are only mounted where necessary"
	FAILED for resource: CronJob.observability.cnpg-degraded-alert
	File: /k8s/bases/infrastructure/controllers/coroot/cron-job-cnpg-degraded-alert.yaml:56-202
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-35

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
Check: CKV_K8S_35: "Prefer using secrets as files over secrets as environment variables"
	FAILED for resource: CronJob.observability.cnpg-degraded-alert
	File: /k8s/bases/infrastructure/controllers/coroot/cron-job-cnpg-degraded-alert.yaml:56-202
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/kubernetes-policies/kubernetes-policy-index/bc-k8s-33

		Code lines for this resource are too many. Please use IDE of your choice to review the file.
github_actions scan results:

Passed checks: 100, Failed checks: 0, Skipped checks: 0

(Truncated to last 6666 characters out of 9747)
⚠️ SPELL / cspell - 3067 errors
"dbname",
        "dedupe",
        "deletecollection",
        "deploymentruntimeconfigs",
        "descheduler",
        "descheduling",
        "desynchronising",
        "devantler",
        "diffable",
        "dispositioned",
        "distroless",
        "dockerconfigjson",
        "docstrings",
        "dorny",
        "dpkg",
        "drainable",
        "emptydir",
        "endgroup",
        "entrys",
        "envsubst",
        "esac",
        "etcdctl",
        "evictable",
        "extensionless",
        "externalsecret",
        "externalsecrets",
        "fanout",
        "featureflagsource",
        "fleetdm",
        "fluxinstance",
        "fromdateiso",
        "gatewayapi",
        "generatable",
        "generatingpolicies",
        "genkey",
        "gethomepage",
        "gitops",
        "gocritic's",
        "golangci",
        "gosec",
        "grjtvs",
        "growfs",
        "growpart",
        "healthchecks",
        "healthz",
        "helmrelease",
        "helmreleases",
        "helmv",
        "homelab",
        "hostnames",
        "httproute",
        "idempotently",
        "imagetools",
        "imagevalidatingpolicy",
        "imranismail",
        "injective",
        "inspectable",
        "iscsi",
        "italicise",
        "ivpol",
        "keylessly",
        "keypair",
        "kprobes",
        "kptr",
        "krew",
        "ksail",
        "ksail's",
        "ksailcd",
        "kubeconfig",
        "kubeconform",
        "kubeconform's",
        "kubelet",
        "kubelet's",
        "kubelets",
        "kubelogin",
        "kubescape",
        "kubespan",
        "kubevirt",
        "kubevuln",
        "kustomization",
        "kustomizations",
        "kyverno",
        "letsencrypt",
        "libc",
        "libgnutls",
        "lintable",
        "livez",
        "loadtester",
        "locationless",
        "luks",
        "lycheeignore",
        "machineconfig",
        "materialise",
        "materialised",
        "materialises",
        "maxage",
        "maxbackup",
        "maxsize",
        "maxx",
        "microtime",
        "misconfig",
        "misconfigs",
        "misordered",
        "mistargeted",
        "mktemp",
        "mlock",
        "mutatingpolicies",
        "mutatingwebhookconfigurations",
        "myapp",
        "najsk",
        "neighbour",
        "nenv",
        "netlink",
        "netpol",
        "netpols",
        "neutralises",
        "neutralising",
        "nextjs",
        "nftables",
        "nilnil",
        "nobuckets",
        "nodepod",
        "nodeport",
        "nolint",
        "nonroot",
        "normalisation",
        "normalised",
        "normalises",
        "normalising",
        "nosec",
        "nullglob",
        "ocirepository",
        "openbao",
        "opencost",
        "openfeature",
        "oras",
        "overprovisioning",
        "parallelised",
        "pasteable",
        "permissioning",
        "pipefail",
        "policyignore",
        "policyreports",
        "portforward",
        "preemptible",
        "preservingly",
        "prioritisable",
        "prioritisation",
        "prioritised",
        "privesc",
        "providerconfigs",
        "pseudonymization",
        "pseudonymized",
        "pseudonymizes",
        "publishapp",
        "publishprovider",
        "pushsecret",
        "pushsecrets",
        "pycache",
        "randomises",
        "rdqwpktr",
        "readyz",
        "reassertions",
        "recognisably",
        "recognise",
        "recognised",
        "recolour",
        "reconverges",
        "refreshfluxghcrauth",
        "regenerable",
        "releaserc",
        "rematerialise",
        "rematerialised",
        "replicaset",
        "repoint",
        "repointed",
        "repoints",
        "repositoryrulesets",
        "resizer",
        "restrictor",
        "retabbed",
        "retarget",
        "rmem",
        "rolebindings",
        "rollouts",
        "rshared",
        "sanitised",
        "sanitiser",
        "sanitising",
        "sarif",
        "scheckov",
        "schedulability",
        "schedulable",
        "scopeable",
        "scopeless",
        "seccomp",
        "secretbox",
        "secretstore",
        "seedable",
        "serialised",
        "serverside",
        "serviceaccount",
        "serviceaccounts",
        "sgdisk",
        "shellcheck",
        "shfmt",
        "shopt",
        "siderolabs",
        "siderolink",
        "sigstore",
        "skmde",
        "slurpfile",
        "sngle",
        "specnull",
        "spiffe",
        "srole",
        "startswith",
        "statefulset",
        "statemanager",
        "stdlib",
        "storageclass",
        "subtest",
        "summarised",
        "surfaceless",
        "syft",
        "syscall",
        "sysctls",
        "talosconfig",
        "talosctl",
        "tanzu",
        "templatesyncignore",
        "thresholded",
        "tlsv",
        "tmpl",
        "tostring",
        "tracepoints",
        "trixie",
        "trueish",
        "trustd",
        "ture",
        "uids",
        "umami",
        "umami's",
        "unablated",
        "unclickable",
        "uncompilable",
        "uncordon",
        "uncordoned",
        "undecoded",
        "undercounts",
        "undispositioned",
        "unevidenced",
        "unexcepted",
        "unfiled",
        "ungated",
        "ungenerated",
        "unifi",
        "uninvoked",
        "unmarshalling",
        "unmarshals",
        "unmodelled",
        "unparseable",
        "unprovisioned",
        "unrecognisable",
        "unrecognised",
        "unrepresentable",
        "unreviewed",
        "unroutable",
        "unrun",
        "unshippable",
        "unskipped",
        "untrackable",
        "unwired",
        "upbound",
        "updatekeys",
        "upjet",
        "upstreaming",
        "urlencode",
        "userinfo",
        "userns",
        "ushfn",
        "validatable",
        "validatealertcoverage",
        "validatingwebhookconfigurations",
        "vcunav",
        "velero",
        "virt",
        "volumesnapshot",
        "vpas",
        "vulnerabilitymanifests",
        "vulnerabilitymanifestsummary",
        "vulns",
        "vxlan",
        "wffc",
        "wgpolicyk",
        "wildcarded",
        "wlid",
        "wmem",
        "workloadconfigurationscans",
        "workloadconfigurationscansummary",
        "worktrees",
        "yannh",
        "yubikey",
        "yzwvjjmcyfnl",
        "zizmor"
    ]
}


You can also copy-paste megalinter-reports/.cspell.json at the root of your repository

(Truncated to last 6666 characters out of 544484)
⚠️ COPYPASTE / jscpd - 183 errors
efresh-flux-ghcr-auth/rollout_safety_test.go [435:83 - 443:5]
Clone found (go)
 - scripts/tests/refresh-flux-ghcr-auth/rollout_safety_test.go [447:52 - 464:5] (18 lines, 123 tokens)
   scripts/tests/refresh-flux-ghcr-auth/rollout_safety_test.go [468:57 - 485:5]
Clone found (go)
 - scripts/tests/refresh-flux-ghcr-auth/rollout_safety_test.go [492:77 - 508:11] (17 lines, 118 tokens)
   scripts/tests/refresh-flux-ghcr-auth/rollout_safety_test.go [519:110 - 535:11]
Clone found (bash)
 - scripts/tests/test-cilium-bandwidth-manager-component.sh [9:1 - 54:2] (46 lines, 202 tokens)
   scripts/tests/test-cilium-homogeneous-devices-activation.sh [9:1 - 54:2]
Clone found (bash)
 - scripts/tests/test-cilium-bandwidth-manager-component.sh [52:5 - 72:2] (21 lines, 67 tokens)
   scripts/tests/test-cilium-homogeneous-devices-activation.sh [80:23 - 100:2]
Clone found (bash)
 - scripts/tests/test-cilium-bandwidth-manager-component.sh [54:1 - 72:2] (19 lines, 65 tokens)
   scripts/tests/test-opencost-usage-scraper.sh [15:1 - 33:2]
Clone found (bash)
 - scripts/tests/test-cilium-mutual-auth-policy.sh [28:29 - 41:10] (14 lines, 78 tokens)
   scripts/tests/test-cilium-mutual-auth-policy.sh [79:33 - 92:10]
Clone found (bash)
 - scripts/tests/test-cilium-mutual-auth-policy.sh [98:33 - 107:54] (10 lines, 57 tokens)
   scripts/tests/test-cilium-mutual-auth-policy.sh [118:32 - 127:54]
Clone found (bash)
 - scripts/tests/test-crossplane-sync-exporter.sh [56:3 - 118:11] (63 lines, 338 tokens)
   scripts/tests/test-opencost-usage-scraper.sh [22:45 - 84:11]
Clone found (bash)
 - scripts/tests/test-refresh-flux-ghcr-auth-safety.sh [177:5 - 185:51] (9 lines, 76 tokens)
   scripts/tests/test-refresh-flux-ghcr-auth-safety.sh [186:5 - 194:51]
Clone found (python)
 - scripts/tests/test_validate_homepage_bookmarks.py [46:57 - 54:54] (9 lines, 58 tokens)
   scripts/tests/test_validate_homepage_bookmarks.py [100:53 - 109:54]
Clone found (go)
 - scripts/validate-dr-signing/main_test.go [1007:13 - 1015:2] (9 lines, 51 tokens)
   scripts/validate-dr-signing/main_test.go [1075:12 - 1083:2]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [244:50 - 249:24] (6 lines, 103 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [756:31 - 761:24]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [276:14 - 282:4] (7 lines, 115 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [603:43 - 609:4]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [302:13 - 311:7] (10 lines, 158 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [828:18 - 837:7]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [311:1 - 316:8] (6 lines, 93 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [838:1 - 843:8]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [388:5 - 393:2] (6 lines, 82 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [574:8 - 579:9]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [389:1 - 399:23] (11 lines, 220 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [666:60 - 677:3]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [393:15 - 404:2] (12 lines, 185 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [494:17 - 505:2]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [440:15 - 451:12] (12 lines, 264 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [649:1 - 662:3]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [503:2 - 509:19] (7 lines, 89 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [688:7 - 694:19]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [503:2 - 509:33] (7 lines, 103 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [1013:58 - 1019:33]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [503:2 - 509:4] (7 lines, 74 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [1126:5 - 1132:4]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [596:30 - 601:8] (6 lines, 50 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [623:44 - 628:8]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [710:22 - 716:2] (7 lines, 135 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [1212:22 - 1218:2]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [951:33 - 959:11] (9 lines, 118 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [963:130 - 971:11]
Clone found (go)
 - scripts/validate-eks-ci-role-policy/main_test.go [1158:47 - 1163:2] (6 lines, 166 tokens)
   scripts/validate-eks-ci-role-policy/main_test.go [1238:46 - 1243:2]
Clone found (go)
 - scripts/validate-flux-verify/instance_test.go [109:62 - 126:31] (18 lines, 57 tokens)
   scripts/validate-flux-verify/instance_test.go [161:57 - 178:31]
Clone found (python)
 - scripts/validate-naming.py [126:52 - 132:25] (7 lines, 53 tokens)
   scripts/validate-naming.py [171:82 - 177:29]
┌────────┬────────────────┬─────────────┬──────────────┬──────────────┬──────────────────┬───────────────────┐
│ Format │ Files analyzed │ Total lines │ Total tokens │ Clones found │ Duplicated lines │ Duplicated tokens │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ bash   │ 34             │ 11018       │ 42989        │ 20           │ 295 (2.68%)      │ 1797 (4.18%)      │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ go     │ 36             │ 29122       │ 177852       │ 97           │ 777 (2.67%)      │ 7487 (4.21%)      │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ python │ 4              │ 807         │ 5606         │ 2            │ 14 (1.73%)       │ 111 (1.98%)       │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ txt    │ 57             │ 3378        │ 109125       │ 64           │ 2417 (71.55%)    │ 80219 (73.51%)    │
├────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
│ Total: │ 131            │ 44325       │ 335572       │ 183          │ 3503 (7.90%)     │ 89614 (26.70%)    │
└────────┴────────────────┴─────────────┴──────────────┴──────────────┴──────────────────┴───────────────────┘
Found 183 clones.
HTML report saved to megalinter-reports/copy-paste/jscpd-report.html
ERROR: jscpd found too many duplicates (7.9%) over threshold (0.0%)
time: 1.32s

(Truncated to last 6666 characters out of 38979)
⚠️ MARKDOWN / markdownlint - 63 errors
ould have a language specified [Context: "```"]
AGENTS.md:101:401 error MD013/line-length Line length [Expected: 400; Actual: 1769]
AGENTS.md:105:401 error MD013/line-length Line length [Expected: 400; Actual: 1126]
AGENTS.md:106:401 error MD013/line-length Line length [Expected: 400; Actual: 1447]
AGENTS.md:159:401 error MD013/line-length Line length [Expected: 400; Actual: 649]
AGENTS.md:161:401 error MD013/line-length Line length [Expected: 400; Actual: 971]
AGENTS.md:188:401 error MD013/line-length Line length [Expected: 400; Actual: 970]
AGENTS.md:191:401 error MD013/line-length Line length [Expected: 400; Actual: 453]
AGENTS.md:210:401 error MD013/line-length Line length [Expected: 400; Actual: 1360]
AGENTS.md:279:401 error MD013/line-length Line length [Expected: 400; Actual: 491]
AGENTS.md:280:401 error MD013/line-length Line length [Expected: 400; Actual: 468]
AGENTS.md:286:401 error MD013/line-length Line length [Expected: 400; Actual: 532]
AGENTS.md:288:401 error MD013/line-length Line length [Expected: 400; Actual: 523]
AGENTS.md:291:401 error MD013/line-length Line length [Expected: 400; Actual: 613]
AGENTS.md:292:401 error MD013/line-length Line length [Expected: 400; Actual: 714]
AGENTS.md:296:401 error MD013/line-length Line length [Expected: 400; Actual: 502]
AGENTS.md:300:401 error MD013/line-length Line length [Expected: 400; Actual: 441]
AGENTS.md:305:401 error MD013/line-length Line length [Expected: 400; Actual: 427]
AGENTS.md:408:401 error MD013/line-length Line length [Expected: 400; Actual: 1139]
AGENTS.md:410:401 error MD013/line-length Line length [Expected: 400; Actual: 1240]
AGENTS.md:415:401 error MD013/line-length Line length [Expected: 400; Actual: 430]
AGENTS.md:426:401 error MD013/line-length Line length [Expected: 400; Actual: 1137]
AGENTS.md:437:401 error MD013/line-length Line length [Expected: 400; Actual: 790]
AGENTS.md:442:401 error MD013/line-length Line length [Expected: 400; Actual: 515]
CLAUDE.md:1 error MD041/first-line-heading/first-line-h1 First line in a file should be a top-level heading [Context: "@AGENTS.md"]
docs/dr/alerting.md:226:28 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/crypto-custody.md:22:389 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/crypto-custody.md:23:264 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/crypto-custody.md:27:35 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/crypto-custody.md:27:161 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/crypto-custody.md:27:239 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/crypto-custody.md:114 error MD024/no-duplicate-heading Multiple headings with the same content [Context: "Custody recommendations"]
docs/dr/crypto-custody.md:245 error MD024/no-duplicate-heading Multiple headings with the same content [Context: "Custody recommendations"]
docs/dr/crypto-custody.md:251 error MD024/no-duplicate-heading Multiple headings with the same content [Context: "What to do if it leaks"]
docs/dr/crypto-custody.md:258 error MD024/no-duplicate-heading Multiple headings with the same content [Context: "What to do if it is *lost* (no..."]
docs/dr/restore-drill.md:42 error MD028/no-blanks-blockquote Blank line inside blockquote
docs/dr/runbook.md:23:102 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/runbook.md:23:487 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/runbook.md:34 error MD028/no-blanks-blockquote Blank line inside blockquote
docs/dr/runbook.md:41 error MD028/no-blanks-blockquote Blank line inside blockquote
docs/dr/runbook.md:50 error MD028/no-blanks-blockquote Blank line inside blockquote
docs/dr/runbook.md:489:92 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/spire-server-ha.md:93 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
docs/dr/velero-cnpg.md:11 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
docs/dr/velero-cnpg.md:56:78 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/velero-cnpg.md:56:166 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/velero-cnpg.md:57:78 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/velero-cnpg.md:57:227 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/velero-cnpg.md:58:78 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/dr/velero-cnpg.md:58:166 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
docs/github-management.md:38:401 error MD013/line-length Line length [Expected: 400; Actual: 419]
docs/github-management.md:40:401 error MD013/line-length Line length [Expected: 400; Actual: 522]
docs/node-autoscaling.md:14 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
docs/oidc-kubectl.md:95 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
docs/runtime-security.md:114 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
docs/rwx-storage.md:9 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
docs/unifi-management.md:14 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
docs/unifi-management.md:62 error MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]
README.md:116:401 error MD013/line-length Line length [Expected: 400; Actual: 540]
README.md:237:32 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]
README.md:237:36 error MD060/table-column-style Table column style [Table pipe does not align with header for style "aligned"]

(Truncated to last 6666 characters out of 6993)
⚠️ REPOSITORY / trivy - 1 error
astructure/vertical-pod-autoscalers/hubble-relay.yaml:7-11
────────────────────────────────────────
   7 ┌ spec:
   8 │   targetRef:
   9 │     apiVersion: apps/v1
  10 │     kind: Deployment
  11 └     name: hubble-relay
────────────────────────────────────────



k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/hubble-ui.yaml (kubernetes)
=========================================================================================
Tests: 117 (SUCCESSES: 116, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

KSV-0037 (MEDIUM): VerticalPodAutoscaler 'hubble-ui' should not be set with 'kube-system' namespace
════════════════════════════════════════
ensure that user resources are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv-0037
────────────────────────────────────────
 k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/hubble-ui.yaml:7-11
────────────────────────────────────────
   7 ┌ spec:
   8 │   targetRef:
   9 │     apiVersion: apps/v1
  10 │     kind: Deployment
  11 └     name: hubble-ui
────────────────────────────────────────



k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/metrics-server.yaml (kubernetes)
==============================================================================================
Tests: 117 (SUCCESSES: 116, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

KSV-0037 (MEDIUM): VerticalPodAutoscaler 'metrics-server' should not be set with 'kube-system' namespace
════════════════════════════════════════
ensure that user resources are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv-0037
────────────────────────────────────────
 k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/metrics-server.yaml:7-11
────────────────────────────────────────
   7 ┌ spec:
   8 │   targetRef:
   9 │     apiVersion: apps/v1
  10 │     kind: Deployment
  11 └     name: metrics-server
────────────────────────────────────────



k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/snapshot-controller.yaml (kubernetes)
===================================================================================================
Tests: 117 (SUCCESSES: 116, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

KSV-0037 (MEDIUM): VerticalPodAutoscaler 'snapshot-controller' should not be set with 'kube-system' namespace
════════════════════════════════════════
ensure that user resources are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv-0037
────────────────────────────────────────
 k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/snapshot-controller.yaml:7-11
────────────────────────────────────────
   7 ┌ spec:
   8 │   targetRef:
   9 │     apiVersion: apps/v1
  10 │     kind: Deployment
  11 └     name: snapshot-controller
────────────────────────────────────────



k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/spire-agent.yaml (kubernetes)
===========================================================================================
Tests: 117 (SUCCESSES: 116, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

KSV-0037 (MEDIUM): VerticalPodAutoscaler 'spire-agent' should not be set with 'kube-system' namespace
════════════════════════════════════════
ensure that user resources are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv-0037
────────────────────────────────────────
 k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/spire-agent.yaml:9-13
────────────────────────────────────────
   9 ┌ spec:
  10 │   targetRef:
  11 │     apiVersion: apps/v1
  12 │     kind: DaemonSet
  13 └     name: spire-agent
────────────────────────────────────────



k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/spire-server.yaml (kubernetes)
============================================================================================
Tests: 117 (SUCCESSES: 116, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

KSV-0037 (MEDIUM): VerticalPodAutoscaler 'spire-server' should not be set with 'kube-system' namespace
════════════════════════════════════════
ensure that user resources are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv-0037
────────────────────────────────────────
 k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/spire-server.yaml:7-11
────────────────────────────────────────
   7 ┌ spec:
   8 │   targetRef:
   9 │     apiVersion: apps/v1
  10 │     kind: StatefulSet
  11 └     name: spire-server
────────────────────────────────────────



k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/tetragon-operator.yaml (kubernetes)
=================================================================================================
Tests: 117 (SUCCESSES: 116, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

KSV-0037 (MEDIUM): VerticalPodAutoscaler 'tetragon-operator' should not be set with 'kube-system' namespace
════════════════════════════════════════
ensure that user resources are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv-0037
────────────────────────────────────────
 k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/tetragon-operator.yaml:7-11
────────────────────────────────────────
   7 ┌ spec:
   8 │   targetRef:
   9 │     apiVersion: apps/v1
  10 │     kind: Deployment
  11 └     name: tetragon-operator
────────────────────────────────────────



k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/tetragon.yaml (kubernetes)
========================================================================================
Tests: 117 (SUCCESSES: 116, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

KSV-0037 (MEDIUM): VerticalPodAutoscaler 'tetragon' should not be set with 'kube-system' namespace
════════════════════════════════════════
ensure that user resources are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv-0037
────────────────────────────────────────
 k8s/providers/hetzner/infrastructure/vertical-pod-autoscalers/tetragon.yaml:9-13
────────────────────────────────────────
   9 ┌ spec:
  10 │   targetRef:
  11 │     apiVersion: apps/v1
  12 │     kind: DaemonSet
  13 └     name: tetragon
────────────────────────────────────────



📣 Notices:
  - Version 0.73.0 of Trivy is now available, current version is 0.71.2

To suppress version checks, run Trivy scans with the --skip-version-check flag

(Truncated to last 6666 characters out of 343700)

✅ Linters with no issues

actionlint, betterleaks, git_diff, golangci-lint, grype, jsonlint, lychee, osv-scanner, prettier, prettier, revive, secretlint, shellcheck, shfmt, syft, trivy-sbom, trufflehog, v8r, v8r, yamllint, zizmor

Notices

📣 MegaLinter 9.5.0 is out! Discover the new features and security recommendations in the release announcement. (Skip this info by defining SECURITY_SUGGESTIONS: false)

See detailed reports in MegaLinter artifacts

MegaLinter is graciously provided by OX Security
Show us your support by starring ⭐ the repository

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

CI is green at the current head and this draft carries no qualifying review, so requesting the
lane-priority reviewer.

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@devantler I will review the changes in #3001.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adds Cosign verification to the Flux root OCIRepository through a FluxInstance patch. It documents bootstrap and runtime ownership of verification settings. The validator now accepts both the KSail configuration and FluxInstance manifest, validates each, and detects signer configuration drift. Tests cover valid, missing, malformed, mistargeted, and inconsistent configurations. CI now runs the validator against both manifests.

Possibly related issues

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the primary change: enforcing Cosign verification on the live root OCIRepository.
Description check ✅ Passed The description directly explains the Cosign verification change, signer constraints, validator updates, and deployment safety.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/validate-flux-verify/instance.go`:
- Around line 309-321: Update findInstance to select only the FluxInstance whose
metadata.name is "flux" and metadata.namespace is "flux-system"; ignore other
resource kinds and identities, and return an error when the expected resource is
missing or appears more than once instead of accepting the first match.
- Around line 104-122: Update verifyWrite and the matching-patch processing to
evaluate every add, replace, and remove operation in declared order, retaining
the final effective /spec/verify state rather than the first value. Ensure
validation rejects an absent final value or a final remove, and add regression
coverage for add-then-remove operations within one patch and across separate
patches.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9c102261-1894-43fa-89b7-9c8611523a42

📥 Commits

Reviewing files that changed from the base of the PR and between 72fe791 and 5d3d662.

📒 Files selected for processing (7)
  • .github/workflows/ci.yaml
  • k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml
  • ksail.prod.yaml
  • scripts/validate-flux-verify/instance.go
  • scripts/validate-flux-verify/instance_test.go
  • scripts/validate-flux-verify/main.go
  • scripts/validate-flux-verify/main_test.go
📜 Review details
🧰 Additional context used
📓 Path-based instructions (4)
k8s/**/*.yaml

📄 CodeRabbit inference engine (AGENTS.md)

k8s/**/*.yaml: Kubernetes base files under k8s/bases/ are immutable; modify them through Kustomize patches: in provider or cluster overlays.
Preserve Flux dependency order: bootstrapinfrastructure-controllersinfrastructureapps.
Use one Kubernetes resource per file, except explicitly whitelisted vendored CDI and KubeVirt operator bundles.
Use kebab-case directories and Kind-led filenames in component folders; qualify filenames with a purpose when multiple resources share a Kind.
Before a manifest PR, run schema-aware KSail validation when available, otherwise build both k8s/clusters/local/ and k8s/clusters/prod/ with kubectl kustomize; use client-side dry-run for changed individual manifests.

Files:

  • k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml
**/*.{yaml,yml}

📄 CodeRabbit inference engine (AGENTS.md)

Use Conventional Commit messages and create pull requests as drafts; keep changes small and focused.

Files:

  • k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml
  • ksail.prod.yaml
ksail*.yaml

📄 CodeRabbit inference engine (AGENTS.md)

Validate KSail configuration and manifests statically; do not start a cluster or mutate ~/.kube/config for maintenance work.

Files:

  • ksail.prod.yaml
.github/workflows/*.yaml

📄 CodeRabbit inference engine (AGENTS.md)

CI validation for Kubernetes-related changes must remain static and must not depend on a running cluster; flux check and other cluster-dependent checks are not maintenance validation.

Files:

  • .github/workflows/ci.yaml
🧠 Learnings (4)
📚 Learning: 2026-08-04T13:06:25.700Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2944
File: scripts/validate-flux-verify/main_test.go:300-305
Timestamp: 2026-08-04T13:06:25.700Z
Learning: For validator acceptance tests under scripts/**/main_test.go, fixed repository-relative paths passed to os.ReadFile do not require //nolint:gosec: golangci-lint does not run gosec on these test-file calls. Apply gosec G304 suppressions only to non-test Go code. Use scripts/validate-dr-signing/main_test.go as the reference analogue.

Applied to files:

  • scripts/validate-flux-verify/main_test.go
📚 Learning: 2026-07-01T21:13:36.950Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2359
File: k8s/bases/apps/actual-budget/helm-release.yaml:62-111
Timestamp: 2026-07-01T21:13:36.950Z
Learning: When reviewing Kustomize/Helm YAML in this repo, keep the base vs provider overlay split: `k8s/bases/apps/**` and `k8s/bases/infrastructure/**` should contain each app’s full, environment-agnostic configuration (including base-level postRenderer Kustomize patches such as deployment strategy, topology spread, probes, and env injection). `k8s/providers/{docker,hetzner}/**` should only add small provider-specific deltas (e.g., `interval`, `persistence.size`) via patch files (like `k8s/providers/<provider>/apps/<app>/patches/helm-release-patch.yaml`). If configuration is identical across providers (e.g., OIDC/OAuth env vars where `${domain}` is resolved per cluster via envsubst), it belongs in the base and must not be duplicated into provider overlays.

Applied to files:

  • k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml
📚 Learning: 2026-07-28T00:33:10.828Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2825
File: .github/workflows/dr-rebuild.yaml:70-73
Timestamp: 2026-07-28T00:33:10.828Z
Learning: For GitHub Actions concurrency configuration in this repo, don’t use `queue: single` on a shared production concurrency group (e.g., `prod-deploy`) that is used for normal production deployments—`queue: single` will cancel/replace any pending run in that group. Normal production deployments must use a uniform `queue: max` for that shared group to ensure serialization without coalescing. If you need DR rebuild coalescing (replacement of pending DR rebuild runs), it must use a separate DR-only concurrency scope/group (not the shared production `prod-deploy` group).

Applied to files:

  • .github/workflows/ci.yaml
📚 Learning: 2026-07-29T07:29:34.804Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2848
File: scripts/validate-publication-order/main.go:315-353
Timestamp: 2026-07-29T07:29:34.804Z
Learning: When reviewing GitHub Actions workflow steps with `run:` blocks, remember that the default non-interactive Bash invocation won’t expand shell aliases unless the script explicitly enables it (e.g., `shopt -s expand_aliases`). Therefore, when assessing potential command-name shadowing (e.g., whether an alias could change what command runs), only treat alias-based behavior as possible if the step explicitly opts into alias expansion via `shopt -s expand_aliases` (or equivalent).

Applied to files:

  • .github/workflows/ci.yaml

Comment thread scripts/validate-flux-verify/instance.go Outdated
Comment thread scripts/validate-flux-verify/instance.go Outdated
…nce identity

Two fail-open holes found in review, both of which let an unverified cluster
pass the gate:

A patch could add /spec/verify and a later operation remove it again -- the
first write was taken as the answer. The state is now folded over every
operation in declared order, so only the effective value counts.

The instance was selected by kind alone, so a second FluxInstance carrying a
correct patch satisfied the check while the deployed one carried none. It is
now selected by identity, and a duplicate is refused rather than guessed.
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Both findings from the previous review are fixed and their threads resolved; CI is green at the new head, which stales the earlier review. Re-requesting at this head.

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@devantler I will review the changes at the requested head.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

CodeRabbit is rate limited at this head (CodeRabbit status: Review rate limited, 14:01:27Z — no review object at 63c61e5a, only empty reply containers from the resolved threads). Advancing to the next lane in priority order.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Codex reports its code-review usage limit is reached (14:12:32Z) — a service failure, not a finding — so advancing to the last lane in priority order.

@devantler

Copy link
Copy Markdown
Contributor Author

@cursor review

@cursor

cursor Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_0f050e59-f980-4482-9963-bb16db4a9d55)

@devantler devantler left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Generated by the Agentic Engineer

Self-review (fallback — CodeRabbit, Codex and Cursor Bugbot unavailable)

Reviewed commit: 63c61e5a251ef35a83ab21e98ec6b4aaecddf26a

Per-lane evidence at this exact head, checked directly on this PR:

  • CodeRabbitCodeRabbit commit status success — Review rate limited at 2026-08-06T14:01:27Z. No review object exists at this head; the two coderabbitai[bot] entries here are zero-length reply containers from the resolved threads, not reviews. (Its review of the previous head 5d3d6621 was real — **Actionable comments posted: 2** — and both findings are fixed and resolved below.)
  • Codex — issue comment at 2026-08-06T14:12:32Z: usage limits for code reviews reached. Service failure, no findings.
  • Cursor Bugbot — check-run conclusion: neutral with output.title: Error, summary Bugbot run failed (9s), plus a cursor[bot] comment at 14:22:18Z stating the usage limit was reached. Zero inline comments, no review object — the run never started. No retry window is stated, so the lane is not merely throttled.

All three are quota/service states rather than verdicts on this change, so the local round applies.

Scope reviewed

The full diff at this head: the FluxInstance kustomize patch, the ksail.prod.yaml comment correction, the CI wiring, and scripts/validate-flux-verify (instance.go, instance_test.go, main.go, main_test.go).

Evidence, not assertion

  • RED before GREEN. TestRealFluxInstanceValidates failed on the shipped manifest with exactly the #2922 defect before the patch was added, and passes after.
  • The new guard is ablated, not assumed. Repointing the patch target is a 2-line change; the guard fires with the specific message and the post-revert control is clean. Separately, kubectl kustomize renders the mistargeted patch at exit 0 with no warning — which is the whole reason the guard exists.
  • Both review fixes are ablated too. Reverting the operation fold to first-write-wins fails exactly the two add-then-remove arms; reverting instance selection to kind-only fails exactly the three identity arms. Neither test set is vacuous.
  • The drift check is ablated. Changing one signer subject in ksail.prod.yaml (2 lines) fails the build naming only the differing entry; post-revert clean.
  • Live baseline recorded for the post-merge check (read-only, admin@prod): ocirepository/flux-system spec.verify ABSENT, generation 3, Ready=True; fluxinstance/flux currently carries 6 patches. After the next prod deploy the expectation is spec.verify present and 7 patches.
  • Full static validation green: ksail --config ksail.prod.yaml workload validate (564 files), both overlay builds, validate-naming.py, go vet, gofmt, and all 31 CI checks at this head with mergeStateStatus: CLEAN.

Findings

No P0/P1.

Two behaviours are deliberate rather than overlooked, recorded here so they are not mistaken for oversights:

  • nit — the drift check is order-sensitive. reflect.DeepEqual over matchOIDCIdentity means reordering the three matchers in one file without the other fails the build, though Flux accepts a match from any entry so the reorder is semantically neutral. Kept: the failure direction is closed, the message says exactly what differs, and "make the two blocks identical" is the intended contract — a comparator that tolerated reordering would be the first step toward tolerating difference.
  • nit — targetsRootSource accepts an absent namespace on the target. A kustomize target without a namespace matches any, which still includes the root source, so treating it as a hit is correct; only a namespace that is present and different is a miss. An absent target name, by contrast, is refused, because that genuinely leaves the subject unstated.

Verdict

Verdict: no P0/P1 findings

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Readiness — all three conditions met at 63c61e5a

1. Programmatically tested. RED before GREEN: TestRealFluxInstanceValidates failed on the shipped manifest with the exact #2922 defect and passes after the patch. Every new guard is ablated rather than assumed — repointing the patch target (2 lines) fires the check and reverts clean; reverting the operation fold to first-write-wins fails exactly the two add-then-remove arms; reverting instance selection to kind-only fails exactly the three identity arms; drifting one signer subject in ksail.prod.yaml (2 lines) fails the build naming only the differing entry. Full static validation green: ksail --config ksail.prod.yaml workload validate (564 files), both overlay builds, validate-naming.py, go vet, gofmt. 32 CI checks green, mergeStateStatus: CLEAN, 0 unresolved threads, 0 non-thread findings.

2. Reviewed. CodeRabbit reviewed the previous head and posted 2 🟠 Major findings — both real fail-open holes in the gate (a patch could add /spec/verify and later remove it; the instance was selected by kind alone so a decoy could satisfy the check). Both fixed, both threads resolved with reasoning, regression coverage added for each. At the current head all three lanes returned quota/service failures — CodeRabbit Review rate limited, Codex usage limit, Bugbot neutral/Error with usage limit reached — so the qualifying local review round applies, clean at this exact SHA with per-lane evidence recorded.

3. Tried and evaluated as a user. Exercised the real behaviour, not traced:

  • Ran the shipped command against the real files: go run ./scripts/validate-flux-verify ksail.prod.yaml k8s/…/flux-instance.yaml → exit 0.
  • Rendered the manifest and confirmed the patch actually materialises: kubectl kustomize k8s/providers/hetzner/infrastructure/controllers/ puts path: /spec/verify with all three matchers into the FluxInstance.
  • Read every failure message as its reader. The first drift message dumped both full matcher lists — ~700 characters of near-identical escaped regex hiding one changed word — so it was rewritten to print only the differing entry. A gate whose output cannot be read is one people route around.
  • Read the live cluster to fix the post-merge baseline (read-only, admin@prod): ocirepository/flux-system spec.verify ABSENT, generation 3, Ready=True; fluxinstance/flux carries 6 patches today.

Why this is Part of #2922 and not Fixes. The control's real effect lands on the next prod deploy, and the issue's own AC-3 insists on a live read-back rather than a green validate. Expected after deploy: spec.verify present and 7 patches. The issue stays open and boarded 📊 Verifying until that is observed.

@devantler
devantler marked this pull request as ready for review August 6, 2026 14:24
@devantler
devantler added this pull request to the merge queue Aug 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 6, 2026
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Evicted from the merge queue — root-caused, and NOT re-queued

This PR was enqueued and then evicted when its merge_group run
31110692734 failed. The
cause is not in this diff.

Both 🚀 Deploy to Prod and 🩹 Heal Prod — the latter re-deploys main, without this change —
failed at the same step with the same error:

The Flux image-verification policy owner did not quiesce before the image-verification policy handoff.

A failure that reproduces while deploying main cannot be attributable to this branch.

The real chain is the 11-day gated Cilium rollout: daemonset/cilium is OnDelete with
updatedNumberScheduled at 0 of 9 (template v1.20.0, all 9 pods still v1.20.0-rc.0), so the
Flux DaemonSet health check can never converge, infrastructure-controllers stays Reconciling,
infrastructure stays DependencyNotReady, and the deploy bridge's handoff quiesce check times
out. Full evidence and the impact analysis are on
#2963.

Per the merge-queue rule in AGENTS.md — root-cause a kick-out before re-queuing, and never
re-queue a known systemic failure — this PR has deliberately not been re-queued. The prior 11
merge-group deploys today succeeded because the blocking health-check windows are bounded and
followed by a retry gap; re-queuing now would only re-roll that dice, and each attempt runs a real
prod deploy.

Publish was skipped in the failed run, so no speculative artifact reached the mutable tag and prod
remained on its previous revision.

The PR itself is unchanged and still pentad-clear at 63c61e5a25 (green checks, zero unresolved
threads, no findings, CLEAN, clean current-head review). It is ready to merge as soon as the
rollout in #2963 is finished or rolled back.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

This PR's premise changed today — spec.verify is now LIVE on the root source, and I did not put it there

#2922 and this PR both rest on "the live root ocirepository/flux-system still has spec.verify
absent". Re-measured against prod just now, that is no longer true:

ocirepository/flux-system  (flux-system)
  verify-subject: ^https://github\.com/devantler-tech/platform/\.github/workflows/ci\.yaml@refs/heads/gh-readonly-queue/main/.+$
  verify-subject: ^https://github\.com/devantler-tech/platform/\.github/workflows/cd\.yaml@refs/heads/main$
  verify-subject: ^https://github\.com/devantler-tech/platform/\.github/workflows/dr-rebuild\.yaml@refs/heads/main$

Those are exactly the three subjects this PR intends to install, in the same order.

Who wrote it, and when

From managedFields on that object (read with --show-managed-fields — the field is omitted without it):

Manager Operation Time Owns
ksail Update 2026-08-06T14:28:48Z (no spec keys)
flux-operator Apply 2026-08-06T14:28:53Z interval, provider, ref, secretRef, url, verify

So verify is owned by flux-operator — the manager this PR argues it must be owned by — and it
landed five seconds after a ksail write, today. The bootstrap/update path evidently ran against
prod at 14:28Z and propagated spec.workload.flux.verify from ksail.prod.yaml.

Two things follow, and I am deliberately not resolving them inside this PR:

  1. The mechanism this PR advocates is confirmed correct — the operator, not a hand-written
    override, is the right writer, and it is demonstrably capable of owning the field.
  2. The stated defect is currently not reproducible. Shipping a fix whose RED condition no longer
    holds would be shipping on a stale premise.

What is still worth having

The control being present today is not the same as it being enforced. This PR's other half —
scripts/validate-flux-verify extended to assert the flux-instance.yaml half on every pull
request — is what stops the field regressing silently, and that value is unchanged. ksail's write
is a bootstrap-path effect; nothing in CI asserts it survives.

Also worth flagging

This 14:28Z ksail write sits oddly against #2963, which measures that the Cilium rollout gate has
been skipping the pipeline's only ksail cluster update since 2026-07-26. Either that step ran by
another route today, or the gate's suppression is narrower than #2963 records. That is a question for
#2963, and I have posted today's widened Cilium evidence there separately.

Not promoting or merging this while its premise reads stale. Next step is to re-derive the RED
condition — is there still a path on which the root source ends up unverified? — and either rescope
this to the CI-enforcement half alone, or close it and keep #2922 open against the real gap.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Correction to my previous comment — spec.verify is live because of this PR, and it is unbacked by Git

My 17:43Z comment reported that the live root OCIRepository now carries the three verify subjects
and that I "did not put it there". The first half is right; the second is wrong, and the conclusion a
reader would draw from it — that this PR is no longer needed — is the opposite of the truth.

Where the live verify came from

This PR's merge group
(run 31110692734, branch
gh-readonly-queue/main/pr-3001-…) started 🚀 Deploy to Prod at 14:26:06Z. A merge-group deploy is
speculative — it applies to real production before the PR merges. It got as far as applying, then
failed at 14:57:00Z on the #2963 quiesce timeout, and the PR was evicted.

The apply is on the object: fluxinstance/flux shows kustomize-controller Apply at 14:28:51Z
— inside that deploy's window — and the live spec.kustomize.patches now contains this branch's
OCIRepository/flux-systemadd /spec/verify patch. The flux-operator write I pointed at
(14:28:53Z, two seconds later) is the operator doing its job: it read the patch this deploy had just
installed and rendered it onto the root source. It was the effect of this PR, not an independent
fix.

Why that makes merging more urgent, not less

On main, flux-instance.yaml was last modified 2026-06-18 (0b821b4d0) and contains zero
occurrences of spec/verify. So right now:

  • production has the control;
  • main does not; and
  • the 🩹 Heal Prod job that exists to undo exactly this failed too, at 15:02:13Z, on the same
    quiesce cause — so nothing rolled it back.

The control is therefore live only as an artifact of a failed deploy, and the next successful deploy
from main removes it.
Closing this PR as "already done" would leave the security control in the
one state that looks fixed and silently is not — present on the cluster, absent from Git, and
scheduled for deletion by the next green deploy.

I have filed the underlying mechanism — a failed candidate deploy leaving prod on never-merged code
with a correlated recovery failure — as #3005.

Status of this PR: unchanged and correctly parked

It stays promoted and pentad-clear (CLEAN, 27 checks green, 0 threads, 0 non-thread findings), and
it is not being re-queued. Per the merge-queue rule in AGENTS.md, re-queuing on a known
systemic failure is futile: every merge-group deploy currently fails at the #2963 Cilium quiesce, and
a re-queue would simply repeat today's sequence — another speculative apply, another failed heal.
The blocker is #2963, and the GitHub Actions incident opened at 15:22Z (still investigating) means
merge-group checks cannot complete at all meanwhile.

Do not close this PR on the grounds that the cluster already has the control. It has it from
here
, and only until the next deploy.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

🛑 STOP — this configuration is live on prod right now and Flux is REFUSING it. Do not merge as written.

Following up on my correction above, I read the root source's conditions rather than just its spec.
The three-subject matcher this PR installs is not merely unproven — it is actively failing, and it
has been since the moment the speculative deploy applied it:

ocirepository/flux-system  (flux-system)
  Ready          False  VerificationError   since 2026-08-06T14:29:01Z
  SourceVerified False  VerificationError   since 2026-08-06T14:29:01Z
  message: failed to verify the signature using provider 'cosign keyless':
           no matching attestations: unsupported: multiple identities are not
           supported at this time                      (repeated 3×, once per matcher entry)

multiple identities are not supported at this time is Flux's own verifier refusing the shape of
the config, not a mismatch in the subjects. Whether the three subjects are individually correct is
irrelevant — a matchOIDCIdentity list with more than one entry is rejected before any of them is
evaluated. The error repeats exactly three times, once per entry, which is what identifies it as a
per-entry structural refusal rather than a failed match.

So the central mechanism of this PR does not work. The RED I built it against was real, the delivery
route through the operator is right, and the pinned-subject reasoning is right — but the shape of
the matcher cannot express three identities.

Live impact, and why it cannot self-heal

The root OCIRepository is the source every controller, tenant binding and policy arrives through. It
has been Ready: False for 4h 45m. Existing workloads keep running, but reconciliation at the
root is refused, and flux-system/apps is still on the previous revision
(…82e0d0bf) while the stored artifact is …e6039d0a.

It is also self-locking. The offending verify block is written by flux-operator from the live
FluxInstance; the FluxInstance is delivered by kustomize-controller from the artifact; and the
artifact is what verification is now refusing. The cluster therefore cannot reconcile away the config
that is blocking it — the fix cannot arrive through the channel it has broken. Recovery needs an
out-of-band change to the live FluxInstance, which is a maintainer action; I have read-only cluster
access by design and have changed nothing.

This is the failure mode filed as #3005, now with a concrete instance: a never-merged candidate did
not merely diverge production from main, it halted delivery, and the 🩹 Heal Prod job that
exists to undo exactly that failed in the same run.

What this PR needs before it goes anywhere near the queue again

  1. Express the allow-list as ONE matchOIDCIdentity entry. The three subjects have a common
    prefix, so a single alternation is the obvious candidate — but I have not verified that Flux
    accepts it, and I am explicitly not guessing on a control whose failure mode is what you see above.
    Prove it against a real Flux verifier before proposing it.
  2. Prove the matcher against a real signature, not against a schema. Everything in this PR passed
    static validation and 27 green checks while being unable to verify a single artifact. ksail workload validate, the rendered-shape assertions, and scripts/validate-flux-verify all
    confirmed the control was present and aimed correctly; none of them could have caught that the
    verifier rejects the shape. That gap is the real lesson here and should be closed in whatever
    replaces this.
  3. Retract the "operational note" in the body — "the effect lands on the next production deploy"
    is now known to be wrong: the effect landed early, via the merge queue, and it broke delivery.

Given (1) is unresolved and the deploy path is separately blocked by #2963, I am leaving this PR
open, promoted, and NOT re-queued. It should probably be converted back to draft or closed in
favour of a corrected single-identity version; I am not doing either unilaterally while its config is
the thing currently sitting on production.

Correcting myself explicitly: my previous comment said "do not close this PR on the grounds that the
cluster already has the control". That still holds — but for a different reason than I gave. The
cluster does not have a working control; it has a broken one, from this branch.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Root cause is upstream, and it is a docs-vs-implementation gap — not our misconfiguration

I said above I would not guess at the remedy. Here is the evidenced version.

Flux documents matchOIDCIdentity as a list with OR semantics. From the OCIRepository
API documentation: it is "a list of OIDC
identity matchers", each with .issuer and .subject as regexps that "should follow the Go regular
expression syntax", and — decisively — "the matchers are evaluated in an OR fashion, i.e. the identity
is deemed to be verified if any one matcher successfully matches against the identity."

So the three-entry list in this PR is exactly what the documented API describes, which is why it
passed every review and validation gate we have.

The underlying library cannot honour it. The error text is not Flux's — it comes from
sigstore/cosign (pkg/cosign/verify.go), whose verification path supports a single identity and
returns unsupported: multiple identities are not supported at this time when given more than one.
Flux accepts the list at the API level, hands the identities down, and cosign refuses the set. The
result is the worst of both: the config is documented-valid, schema-valid, and verifies nothing
— failing closed for all three entries rather than falling back to any one of them.

The only shape that can express our allow-list

Because both fields are Go regexps and exactly one identity is permitted, an allow-list has to be
collapsed into one matcher with an alternation in the subject — the three subjects share a prefix,
so that is expressible without weakening anything:

issuer:  ^https://token\.actions\.githubusercontent\.com$
subject: ^https://github\.com/devantler-tech/platform/\.github/workflows/(ci\.yaml@refs/heads/gh-readonly-queue/main/.+|cd\.yaml@refs/heads/main|dr-rebuild\.yaml@refs/heads/main)$

That is a candidate, still unproven against a real verifier. It is now grounded in documented
semantics rather than a hunch, but the whole lesson of this PR is that documented semantics were not
enough — so it must be demonstrated verifying an actual signed artifact before it goes near the merge
queue. Given a candidate deploy applies to production (#3005), "try it and see" is not a safe
experiment on this resource.

Two follow-ups this creates

  1. Our validator needs a check it cannot currently make. scripts/validate-flux-verify asserts
    presence and target. It should also reject a matchOIDCIdentity with more than one entry — a
    cheap, static guard that would have caught this before it ever reached a cluster, and the only
    part of this whole finding that is fixable without a live verifier.
  2. Worth reporting upstream to Flux, since the API and docs promise OR-evaluation the
    implementation cannot deliver, and a fail-closed security control is a bad place to discover that.
    Opening third-party issues is not something I do autonomously — flagging it as a maintainer call.

Sources: Flux OCIRepository API docs ·
sigstore/cosign verification

@devantler
devantler marked this pull request as draft August 6, 2026 19:23
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Converted back to draft — safety only. I am not taking this PR's fix; the analysis above owns it.

This was sitting OPEN, non-draft, mergeStateStatus: CLEAN, with a current-head green review on
record. That is precisely the trigger condition for the merge rung, so a sweep following the procedure
mechanically could have merged it — putting the three-identity matchOIDCIdentity onto main
permanently, when we now know it verifies nothing and has already halted prod delivery for ~5h
(platform#3005).

The readiness condition that no longer holds is the third one: tried and evaluated as a user. The
change is documented-valid and schema-valid, and it does not work. Draft is the honest state for that,
and it is trivially reversible once the single-matcher collapse lands.

I have deliberately not pushed the regex-alternation fix — a second instance editing this branch
while its owner is mid-analysis is the two-writer collision our contract warns about. Re-promote as
soon as the collapsed matcher is in and verified against the live source's status.conditions (not
its spec — that distinction is what hid this for five hours).

One datum for that verification, measured independently at ~19:2xZ: the live remedy for the cluster
is a single patch removal — fluxinstance/flux has 7 entries under spec.kustomize.patches, exactly
one of which adds /spec/verify. main declares zero. Details on #3005.

…pports

cosign's keyless verification rejects a multi-entry matchOIDCIdentity outright
and fails closed for the whole set, so the three-entry matcher verified nothing
while reading as stricter than a single one. Both halves now alternate the same
three signers inside one subject regex, and the validator fails the build on any
list longer than one so the shape cannot return.
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Validation for db0b8ca7 — including the positive proof the first attempt lacked

The original failure was not "a typo in a matcher". It was that every check available passed on a
control that verified zero artifacts
— schema-valid, docs-valid, correctly targeted, and green in
this PR's own purpose-built validator. So arity alone is not enough evidence; efficacy had to be
demonstrated.

1. Who actually signed the live artifact — measured, not assumed

I have no packages: read on this host, so cosign verify against GHCR returns
UNAUTHORIZED and could not settle it. The public Sigstore transparency log does, with no
registry access at all. Querying Rekor by the live digest
(latest@sha256:e6039d0a…) returns 3 DSSE entries, all carrying the same certificate SAN:

https://github.com/devantler-tech/platform/.github/workflows/ci.yaml@refs/heads/gh-readonly-queue/main/pr-3001-72fe7919f1195705f1e95fc6e609b13dc0d56b51

with OIDC issuer https://token.actions.githubusercontent.com (cert extension
1.3.6.1.4.1.57264.1.1). Two things follow. The signer is ci.yaml under the merge-queue ref —
the first alternative in the new pattern — and the pr-3001-… ref confirms the artifact on
latest was published by this PR's own evicted merge_group run, which is the "prod on
never-merged code" #3005 describes.

2. The matcher accepts that exact subject

Tested with Go's regexp — the same engine cosign uses:

input result
the real SAN above match
the real issuer above match
ci.yaml@refs/heads/main (near miss, never trusted) no match

Plus 7 further negative controls: wrong org, unlisted workflow, feature branch, branch-prefix
extension (mainX), empty merge-queue suffix, and an unanchored prefix. All reject. The pattern is
exactly the union of the three previous subjects — no signer gained or lost.

3. RED/GREEN on the guard

Ran the validator against the exact config that halted prod, and against the fix:

input result
3-entry matcher (what shipped) exit 1 — "has 3 entries, and cosign supports exactly ONE"
1-entry alternation (this PR) exit 0

The guard is asserted in both halves independently, not just the cluster-config one. That matters
because the existing drift check only makes the two copies equal — two identically broken copies
satisfy it. Ablation: with the cluster-config half fixed and the patch half left broken, the failure
message changes from the drift error to the arity error, proving the patch-half check is genuinely
reached rather than shadowed.

4. One existing test was inverted, deliberately

one complete matcher among incomplete ones validates asserted that a 2-entry list should pass,
reasoning that one usable entry is enough. That describes Flux's documented OR semantics, which
cosign does not implement. The arm is now a second matcher entry is rejected even when one entry is complete, with the reasoning recorded inline. A non-vacuity control was added alongside it — the
same three signers alternated in one subject must still validate — so a guard that rejected
everything could not pass.

go test ./scripts/validate-flux-verify/... green (incl. TestRealConfigValidates,
TestRealFluxInstanceValidates, TestRealHalvesAgree, which exercise the real repo files);
go vet clean.

Still outstanding, and not in this PR

Efficacy is proven for the artifact live right now, by hand. Nothing yet proves it on every future
deploy, which is the general form of this defect — I have filed that separately rather than widening
this PR, alongside #3006 (a bad FluxInstance change cannot be corrected through GitOps).

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '=== Changed files ==='
git diff --name-status HEAD^ HEAD

printf '%s\n' '=== Go package outline ==='
ast-grep outline scripts/validate-flux-verify --items all --type function

printf '%s\n' '=== Matcher and verification references ==='
rg -n -C 4 'matchOIDCIdentity|verify|checkNoDrift|validateInstance|configVerifyBlock' \
  scripts/validate-flux-verify \
  ksail.prod.yaml \
  k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml \
  .github/workflows/ci.yaml

printf '%s\n' '=== Current patch diff ==='
git diff --unified=80 HEAD^ HEAD -- \
  .github/workflows/ci.yaml \
  ksail.prod.yaml \
  k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml \
  scripts/validate-flux-verify

Length of output: 145290


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '=== Current effective matcher lines ==='
nl -ba ksail.prod.yaml | sed -n '228,256p'
nl -ba k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml | sed -n '194,218p'

printf '%s\n' '=== Validator matcher checks ==='
nl -ba scripts/validate-flux-verify/main.go | sed -n '108,168p;360,402p'
nl -ba scripts/validate-flux-verify/instance.go | sed -n '176,232p'

printf '%s\n' '=== Tests that compile or match issuer and subject regexes ==='
rg -n -C 5 'regexp\.|Compile|MatchString|token\.actions|githubusercontent|actions\.githubusercontent' \
  scripts/validate-flux-verify

printf '%s\n' '=== Parent configuration issuer values ==='
git show HEAD^:ksail.prod.yaml | sed -n '/matchOIDCIdentity:/,+8p'
git show HEAD^:k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml | sed -n '/matchOIDCIdentity:/,+14p'

Length of output: 1291


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '=== Current effective matcher lines ==='
sed -n '228,256p' ksail.prod.yaml
sed -n '194,218p' k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml

printf '%s\n' '=== Validator matcher checks ==='
sed -n '108,168p;360,402p' scripts/validate-flux-verify/main.go
sed -n '176,232p' scripts/validate-flux-verify/instance.go

printf '%s\n' '=== Tests that compile or match issuer and subject regexes ==='
rg -n -C 5 'regexp\.|Compile|MatchString|token\.actions|githubusercontent|actions\.githubusercontent' \
  scripts/validate-flux-verify || true

printf '%s\n' '=== Parent configuration issuer values ==='
git show HEAD^:ksail.prod.yaml | sed -n '/matchOIDCIdentity:/,+8p'
git show HEAD^:k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml | sed -n '/matchOIDCIdentity:/,+14p'

Length of output: 20334


@devantler I found one blocking finding.

  • High — Correct the OIDC issuer in both matcher copies. ksail.prod.yaml configures ^https://token\.actions\.githubusercontent\.com$. The FluxInstance patch has the same value. Your Rekor evidence reports the live certificate issuer as https://token.actions.githubusercontent.com. These domains differ. Cosign checks the issuer before it checks the subject, so the revised subject alternation still rejects the live artifact. Change both copies to match the measured issuer and add a test that compiles the issuer regex and matches the measured issuer value. The current validator only checks that issuer is non-blank, so it cannot detect this failure.

✏️ Learnings added
Learnt from: devantler
URL: https://github.com/devantler-tech/platform/pull/3001

Timestamp: 2026-08-06T20:59:46.461Z
Learning: For this platform's GitHub Actions keyless Cosign signatures, the measured OIDC issuer is `https://token.actions.githubusercontent.com`. The issuer regex in `ksail.prod.yaml` and `k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml` must use this exact domain; `token.actions.githubusercontent.com` is a different domain and does not match the certificate issuer.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@devantler devantler left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Generated by the Agentic Engineer

Self-review (fallback — CodeRabbit, Codex and Cursor Bugbot unavailable)

Reviewed commit: db0b8ca7c8af0f26d8acda4556e8769a74b6baa3

Per-lane evidence, checked directly on this PR at this head rather than taken from a portfolio digest:

  • CodeRabbit — commit status at head is success / "Review rate limited". That is the provider's own quota signal, not a verdict on this change; the lane will not deliver here.
  • Codex — zero review objects at this head; its only comment on this PR (14:12:32Z) reads "You have reached your Codex usage limits for code reviews."
  • Cursor Bugbot — no check-run at this head. The lane is usage-limited account-wide (measured across 35 PRs today, this one included); re-requesting cannot succeed and only burns comments, so it was not re-triggered.

What I checked, and how

Correctness of the matcher was established positively, which is the whole point — the defect this PR fixes was a control that passed every structural check while verifying nothing.

  1. Who really signed the live artifact. No packages: read on this host, so cosign verify against GHCR returns UNAUTHORIZED. Resolved instead through the public Sigstore transparency log, which needs no registry credential: all 3 Rekor DSSE entries for latest@sha256:e6039d0a… carry SAN …/workflows/ci.yaml@refs/heads/gh-readonly-queue/main/pr-3001-72fe7919… and issuer https://token.actions.githubusercontent.com.
  2. The new matcher accepts exactly that, tested with Go's regexp (cosign's engine): real subject → match, real issuer → match, ci.yaml@refs/heads/main → no match. Plus 7 further negative controls (wrong org, unlisted workflow, feature branch, mainX prefix extension, empty queue suffix, unanchored prefix). The pattern is the exact union of the three previous subjects — no signer gained or lost.
  3. RED/GREEN on the guard, against the literal config that halted prod: 3-entry → exit 1 naming the fix; 1-entry alternation → exit 0. Asserted independently in both halves, with an ablation proving the FluxInstance-half check is genuinely reached and not shadowed by the drift check (drift only proves the two copies are equal, and two identically broken copies satisfy it).
  4. Suite greengo test ./scripts/validate-flux-verify/..., including TestRealConfigValidates, TestRealFluxInstanceValidates and TestRealHalvesAgree, which exercise the real repo files. go vet clean.
  5. Second half is a real fix, not collateral. ksail.prod.yaml on main carries the same broken 3-entry shape today, so a disaster-recovery rebuild would currently fail exactly as prod did — with no cluster left to recover from.

One judgement call, disclosed rather than hidden

Three comments added here narrate history (the inverted test arm, and the two "this halted prod" rationales). That runs against the repo's own describe-the-as-is rule, and I kept them deliberately: the inverted expectation looks wrong without its reason and would otherwise be "corrected" back, and the guard's cost/benefit is unreadable without knowing what it caught. I would accept a reviewer trimming them; I would not accept trimming them to the point where the next reader reinstates the multi-entry form.

Verdict

No P0/P1 findings.

⚠️ This clears the review gate only. Required CI has not run at this head — the repo's CI workflow was never created for db0b8ca7 (the sole run is a queued GitHub-managed dynamic one), because GitHub Actions is in a critical outage. This PR must stay a draft until CI actually runs green: merging enqueues a merge_group run that deploys to the real prod cluster, and that is not something to do on unverified checks — least of all on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

Cosign verify is configured but not enforced: the deploy path never applies it to the live OCIRepository

1 participant