feat: Add Helm chart support, scripts, and docs for externalized config (Phase 5 of 5)

[cwiklik]

Summary

Final phase of the externalize-config epic (#109). Adds the Helm chart infrastructure, operational scripts, and documentation that wire together all four previous phases.

• Helm ConfigMaps: configmap-platform-defaults.yaml and configmap-feature-gates.yaml templates render values.yaml defaults into cluster ConfigMaps. Keys match exactly what the Go loaders read (config.yaml, feature-gates.yaml).
• Deployment wiring: deployment.yaml updated with --config-path and --feature-gates-path CLI args, and matching volume mounts at /etc/kagenti and /etc/kagenti/feature-gates.
• values.yaml: New featureGates and defaults sections — all sidecar images, ports, resources, and per-sidecar enabled flags are version-controlled with the chart.
• webhook-rollout.sh: Extended to 6-step flow — builds image, loads to kind, deploys both ConfigMaps via helm template, patches deployment volumes, waits for rollout, creates authbridge webhook.
• test-precedence.sh: Automated 8-test runner covering all 7 precedence layers. Speeds up kubelet ConfigMap sync to 10s for the test run, then resets.
• demo-precedence.sh: Interactive tmux demo — 6 scenarios with live pod-watch and webhook log panes.
• spiffe-helper opt-out (finalized): Layer 7 is now opt-out (kagenti.io/spire=disabled blocks injection). No label required for injection. spireEnabled in InjectAuthBridge now follows decision.SpiffeHelper.Inject.
• SecurityContext: BuildSpiffeHelperContainer gains RunAsNonRoot: true alongside existing UID/GID 1000 — required for admission controllers enforcing runAsNonRoot.
• Dead code removed: Four package-level ContainerBuilder wrapper functions (introduced for Phase 3 backward compat, superseded by Phase 4) removed.

Test plan

• helm lint charts/kagenti-webhook/ — 0 failures
• helm template kagenti-webhook charts/kagenti-webhook/ — ConfigMap keys and volume paths render correctly
• go build ./... && go vet ./... && go test ./internal/webhook/... — all pass
• ./scripts/webhook-rollout.sh — deploys webhook + ConfigMaps in one command
• ./scripts/test-precedence.sh — all 8 tests pass against a live kind cluster
• ./scripts/demo-precedence.sh — 6 interactive scenarios run correctly in tmux

Part of #109 (Phase 5 of 5 — final)

This PR depends on changes in kagenti UI described here: rossoctl/rossoctl#836

[Copilot]

Pull request overview

Final phase of the externalized-config effort: adds Helm chart wiring (ConfigMaps + volume mounts + CLI flags), operational scripts, and updated docs/tests to support multi-layer injection precedence with hot-reloadable platform defaults and feature gates.

Changes:

• Add Helm chart templates for platform defaults / feature gates ConfigMaps and mount them into the controller with --config-path / --feature-gates-path.
• Extend local operational scripts for rollout + automated/interactive precedence verification.
• Update injector logic/tests/docs for SPIRE opt-out behavior and tighten sidecar security context.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 22 comments.

Show a summary per file

File	Description
kagenti-webhook/testing-plan-precedence.md	Manual testing plan for precedence behavior and hot-reload.
kagenti-webhook/scripts/webhook-rollout.sh	Local build/deploy workflow extended to apply Helm-rendered ConfigMaps and patch deployment.
kagenti-webhook/scripts/test-precedence.sh	Automated precedence test runner (kind-based) including hot-reload validation.
kagenti-webhook/scripts/demo-precedence.sh	Interactive tmux demo for precedence scenarios.
kagenti-webhook/internal/webhook/injector/precedence_test.go	Updates precedence tests for SPIRE opt-out semantics.
kagenti-webhook/internal/webhook/injector/precedence.go	SPIRE opt-out layer added for spiffe-helper after standard chain.
kagenti-webhook/internal/webhook/injector/pod_mutator.go	SPIRE decision now follows precedence decision instead of label helper.
kagenti-webhook/internal/webhook/injector/container_builder.go	Add RunAsNonRoot: true for spiffe-helper; remove deprecated wrapper functions.
kagenti-webhook/README.md	Document precedence chain, feature gates, rollout workflow, and platform defaults.
kagenti-webhook/ARCHITECTURE.md	Update architecture/injection-flow docs for new precedence behavior.
charts/kagenti-webhook/values.yaml	Add featureGates and defaults values to drive generated ConfigMaps.
charts/kagenti-webhook/templates/deployment.yaml	Mount ConfigMaps and pass config paths via CLI flags.
charts/kagenti-webhook/templates/configmap-platform-defaults.yaml	Render values.defaults into config.yaml ConfigMap.
charts/kagenti-webhook/templates/configmap-feature-gates.yaml	Render values.featureGates into feature-gates.yaml ConfigMap.
AuthBridge/k8s/configmaps-webhook.yaml	Namespace-scoped ConfigMaps required for AuthBridge webhook demo.
AuthBridge/k8s/authbridge-deployment.yaml	Full AuthBridge demo manifest (SPIFFE).
AuthBridge/k8s/authbridge-deployment-no-spiffe.yaml	Simplified AuthBridge demo manifest (no SPIFFE).
AuthBridge/k8s/auth-target-deployment-webhook.yaml	Target service manifest for webhook demo.
AuthBridge/k8s/agent-deployment-webhook.yaml	Agent workload manifest for webhook demo (SPIFFE).
AuthBridge/k8s/agent-deployment-webhook-no-spiffe.yaml	Agent workload manifest for webhook demo (SPIFFE opt-out).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pdettori]

Review Summary

Comprehensive final phase of the externalize-config epic. Well-structured PR with Helm chart wiring (ConfigMaps + volume mounts), operational scripts (rollout, automated tests, interactive demo), and updated docs. The opt-out model for spiffe-helper, SecurityContext hardening, and dead code removal are clean.

A few issues in the Helm templates and scripts need attention — one could leave a cluster in a modified state on script failure.

Areas reviewed: Go, Helm/K8s, Shell scripts, YAML manifests, Security
Commits: 4, all signed-off ✓
CI status: passing ✓

Additional notes (not inline)

• The configmap-feature-gates.yaml template has the same nil-guard issue as configmap-platform-defaults.yaml (see inline comment).
• Demo manifests (AuthBridge/k8s/) store Keycloak admin credentials in ConfigMaps rather than Secrets. Acceptable for local demos but worth a # WARNING: demo only comment on each.
• demo-precedence.sh has no cleanup trap — if interrupted, test deployments and namespace are left behind.
• spiffe-helper container in authbridge-deployment.yaml is missing resource limits (all other containers have them).

[cwiklik]

Code review

Found 1 issue:

1. webhook.enableClientRegistration: false in Helm values has no effect. The template conditionally includes the flag only when truthy, but the Go binary defaults --enable-client-registration to true (cmd/main.go:88). When the Helm value is false, the flag is omitted entirely and the Go default takes over, so client registration stays enabled. Fix: always emit the flag with the configured value, e.g. - --enable-client-registration={{ .Values.webhook.enableClientRegistration }}.

https://github.com/kagenti/kagenti-extensions/blob/260ee0cb3b6c5f79e50730d7fff033fd38286473/charts/kagenti-webhook/templates/deployment.yaml#L44-L48

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[mrsabath]

Review Summary

Final phase of the externalize-config epic. The Helm chart wiring (ConfigMaps + volume mounts + CLI flags), operational scripts (rollout, automated tests, interactive demo), and two-stage injection refactor are well-structured. The Go code quality improvements are excellent — error handling fixes, dead code removal, per-request ContainerBuilder (enabling proper hot-reload), and SecurityContext hardening.

Main concern: The opt-in → opt-out behavioral change could silently inject sidecars into existing workloads on upgrade. This needs explicit migration documentation / release notes. The empty feature gates edge case could also silently disable all injection if the featureGates key is accidentally removed from values.yaml.

Areas reviewed: Go, Helm/K8s, Shell scripts, YAML manifests, Security, Docs
Commits: 8, all signed-off ✓
CI status: passing ✓

Highlights

• ✅ ContainerBuilder created per-request with current config snapshot — hot-reload now works correctly
• ✅ RunAsNonRoot: true added to spiffe-helper SecurityContext
• ✅ Error handling fixed in ensureServiceAccount (was silently swallowed)
• ✅ Good error wrapping with file path context in feature_gate_loader.go
• ✅ Dead wrapper functions cleanly removed
• ✅ Review feedback from @pdettori addressed (demo credential warnings, cleanup traps, Podman compat, nil guards)

Additional comments not attached inline

• nit: test-precedence.sh uses untagged busybox (:latest) while demo-precedence.sh pins busybox:1.36. Pin consistently for reproducibility.
• nit: Test coverage for precedence evaluator reduced significantly (290 lines removed, 56 added). Edge cases for the new two-layer model could be added: nil featureGates, concurrent evaluator access.
• suggestion: Multiple demo manifests echo Client Secret: $CLIENT_SECRET to stdout in container startup scripts. Container logs are often collected — even in demos, printing secrets sets a bad precedent.
• suggestion: Keycloak admin credentials in ConfigMaps across demo manifests. The # WARNING: demo only comments help, but using Secrets with secretKeyRef even in demos establishes the correct pattern.

[mrsabath]

Re-Review (after a76d6dd)

The author addressed the majority of feedback from both @pdettori and @mrsabath reviews in commit a76d6dd.

Previous Issues — Resolution

#	Issue	Status
1	Breaking change needs migration docs	FIXED — README has "Upgrading from the opt-in model" section
2	Empty feature gates → all-false struct	FIXED — All-zero detection with fallback to defaults
3	Nested ConfigMap volume mounts	Not addressed (functional, noted)
4	jwt_svid_file_mode = 0644	PARTIALLY FIXED — Fixed in standalone demo, missing in webhook ConfigMap
5	Missing set -euo pipefail	Not addressed
6	Busybox pinning	PARTIALLY FIXED — Pinned in scripts, not in testing-plan doc
7	Test coverage reduced	Not addressed (acceptable for simplified architecture)
8	Client secret echoed to stdout	FIXED — Redacted in both manifests
9	Keycloak admin in ConfigMaps	PARTIALLY FIXED — Warning comments added

Highlights

• ✅ Migration documentation added for opt-in → opt-out behavioral change
• ✅ All-zero FeatureGates detection prevents accidental injection disablement
• ✅ Client secrets redacted from container startup logs
• ✅ jwt_svid_file_mode tightened to 0600 in demo manifest
• ✅ spiffe-helper image pinned from nightly to 0.11.0
• ✅ busybox pinned to 1.36 in test scripts
• ✅ Resource limits added to spiffe-helper in demo manifest

Remaining items (see inline comments)

Lower-severity items that can be addressed in follow-up — none are blocking.

Areas reviewed: Go, Helm/K8s, Shell scripts, YAML manifests, Security, Docs
Commits: 9, all signed-off ✓
CI status: passing ✓