fix: validate authorization header in oauth2-proxy apps

[merll]

📌 Summary

The initial implementation of the Gateway API implements a check whether the Authorization header is added, but not the contents of the token. Each workload that needs authentication by OAuth2-Proxy therefore needs an additional RequestAuthentication for validating the token, and an AuthorizationPolicy for acting on it.

The existing AuthorizationPolicy still needs to be attached to the Gateway resource, since it operates on that level (checking hostnames and redirecting). The two new resources however need to be added to the workloads, since RequestAuthentication can only be scoped to a Gateway or workload, but not to hostnames. Attaching it to the Gateway would make all bearer tokens being validated by these rules, also the ones that are not related to the platform but user workloads with their own authentication mechanisms.

This also forces authentication of internal traffic against these workloads, which by itself is not a disadvantage. The drawback of attaching it to the workload is however that

1. all validated workloads need to be added to the service mesh
2. all workloads which need exemptions from the rules (e.g. by namespace) need to be added to the mesh as well.

Since Tekton Dashboards needs this while the pipeline containers are specifically not supposed to be injected with Istio proxies, the dashboards are moved from the tekton-pipelines to tekton-dashboards namespace.

A few charts needed to be enhanced so that their pods can be labeled.

🔍 Reviewer Notes

🧹 Checklist

• Code is readable, maintainable, and robust.
• Unit tests added/updated

[svcAPLBot]

Comparison of Helm chart templating output:

@@ spec.template.metadata.labels @@
! + one map entry added:
+ istio.io/rev: 1-26-0


@@ spec.template.metadata.labels @@
! + two map entries added:
+ otomi.io/auth: platform
+ otomi.io/auth-policy: platform


@@ spec.template.metadata.labels @@
! + two map entries added:
+ otomi.io/auth: platform
+ otomi.io/auth-policy: platform


@@ spec.rules.0.to.0.operation.notPaths @@
# security.istio.io/v1/AuthorizationPolicy/istio-system/harbor-oauth2-proxy
! + two list entries added:
+ - "/api/*"
+ - "/c/*"


@@ data.original-values @@
! ± value change in multiline text (one insert, one deletion)
  {
    "autoscaleMax": 5,
    "autoscaleMin": 1,
    "env": {
-     "PILOT_ENABLE_ALPHA_GATEWAY_API": "true"
+     "PILOT_ENABLE_ALPHA_GATEWAY_API": "true",
+     "PILOT_JWT_PUB_KEY_REFRESH_INTERVAL": "1m"
    },
    "gatewayClasses": {
      "istio": {
        "deployment": {
  
  [107 lines unchanged)]
  
      }
    },
    "revision": "1-26-0"
  }

@@ data.merged-values @@
! ± value change in multiline text (one insert, one deletion)
  {
    "affinity": {},
    "autoscaleBehavior": {},
    "autoscaleEnabled": true,
  
  [twelve lines unchanged)]
  
    },
    "deploymentAnnotations": {},
    "deploymentLabels": {},
    "env": {
-     "PILOT_ENABLE_ALPHA_GATEWAY_API": "true"
+     "PILOT_ENABLE_ALPHA_GATEWAY_API": "true",
+     "PILOT_JWT_PUB_KEY_REFRESH_INTERVAL": "1m"
    },
    "envVarFrom": [],
    "experimental": {
      "stableValidationPolicy": false
  
  [271 lines unchanged)]
  
    "variant": "",
    "volumeMounts": [],
    "volumes": []
  }


@@ spec.template.spec.containers.discovery.env @@
! + one list entry added:
+ - name: PILOT_JWT_PUB_KEY_REFRESH_INTERVAL
+   value: 1m


@@ data.values @@
! ± value change in multiline text (one insert, one deletion)
  {
    "gateways": {
      "seccompProfile": {},
      "securityContext": {}
  
  [124 lines unchanged)]
  
        "enabled": false,
        "provider": "default"
      },
      "env": {
-       "PILOT_ENABLE_ALPHA_GATEWAY_API": "true"
+       "PILOT_ENABLE_ALPHA_GATEWAY_API": "true",
+       "PILOT_JWT_PUB_KEY_REFRESH_INTERVAL": "1m"
      }
    },
    "revision": "1-26-0",
    "sidecarInjectorWebhook": {
  
  [six lines unchanged)]
  
      "rewriteAppHTTPProbe": true,
      "templates": {}
    }
  }


@@ spec.from @@
# gateway.networking.k8s.io/v1beta1/ReferenceGrant/platform-oauth2-proxy-apps
! - one list entry removed:
- - group: gateway.networking.k8s.io
-   kind: HTTPRoute
-   namespace: tekton-pipelines
! + one list entry added:
+ - group: gateway.networking.k8s.io
+   kind: HTTPRoute
+   namespace: tekton-dashboard

@@ (root level) @@
# security.istio.io/v1/AuthorizationPolicy/platform-auth-require-jwt
! + one document added:
+ ---
+ # Source: raw/templates/resources.yaml
+ apiVersion: security.istio.io/v1
+ kind: AuthorizationPolicy
+ metadata:
+   name: platform-auth-require-jwt
+   labels:
+     app: raw
+     app.kubernetes.io/instance: kubernetes-gateways-artifacts
+     app.kubernetes.io/managed-by: Helm
+     app.kubernetes.io/name: raw
+     app.kubernetes.io/part-of: otomi
+     app.kubernetes.io/version: 0.2.3
+     helm.sh/chart: raw-0.2.3
+ spec:
+   action: ALLOW
+   rules:
+   - from:
+     - source:
+         requestPrincipals:
+         - "*"
+   selector:
+     matchLabels:
+       otomi.io/auth-policy: platform

@@ (root level) @@
# security.istio.io/v1/RequestAuthentication/platform-auth
! + one document added:
+ ---
+ # Source: raw/templates/resources.yaml
+ apiVersion: security.istio.io/v1
+ kind: RequestAuthentication
+ metadata:
+   name: platform-auth
+   labels:
+     app: raw
+     app.kubernetes.io/instance: kubernetes-gateways-artifacts
+     app.kubernetes.io/managed-by: Helm
+     app.kubernetes.io/name: raw
+     app.kubernetes.io/part-of: otomi
+     app.kubernetes.io/version: 0.2.3
+     helm.sh/chart: raw-0.2.3
+ spec:
+   jwtRules:
+   - audiences:
+     - otomi
+     fromHeaders:
+     - name: Authorization
+       prefix: "Bearer "
+     issuer: "https://keycloak.dev.linode-apl.net/realms/otomi"
+     jwksUri: "http://keycloak-keycloakx-http.keycloak:8080/realms/otomi/protocol/openid-connect/certs"
+   selector:
+     matchLabels:
+       otomi.io/auth: platform


@@ spec.template.metadata.labels @@
! + three map entries added:
+ istio.io/rev: 1-26-0
+ otomi.io/auth: platform
+ otomi.io/auth-policy: platform


@@ spec.podMetadata @@
! - one map entry removed:
- annotations:
-   sidecar.istio.io/inject: "true"

@@ spec.podMetadata.labels @@
! + three map entries added:
+ istio.io/rev: 1-26-0
+ otomi.io/auth: platform
+ otomi.io/auth-policy: monitoring


@@ spec.rules @@
! - one list entry removed:
- - backendRefs:
-   - name: po-alertmanager
-     port: 9093
-   filters:
-   - type: RequestHeaderModifier
-     requestHeaderModifier:
-       remove:
-       - authorization
-   matches:
-   - path:
-       type: PathPrefix
-       value: /
! + one list entry added:
+ - backendRefs:
+   - name: po-alertmanager
+     port: 9093
+   matches:
+   - path:
+       type: PathPrefix
+       value: /


@@ spec.endpoints @@
! - two list entries removed:
- - port: http-web
-   enableHttp2: true
-   path: /metrics
- - port: reloader-web
-   path: /metrics
! + two list entries added:
+ - port: http-web
+   enableHttp2: true
+   path: /metrics
+   relabelings:
+   - separator: ":"
+     sourceLabels:
+     - __tmp_pod_ip_port
+     - __tmp_service_namespace
+     targetLabel: __address__
+   - regex: "(\\d+)\\.(\\d+)\\.(\\d+)\\.(\\d+):(\\d+)"
+     replacement: $5
+     sourceLabels:
+     - __address__
+     targetLabel: __tmp_pod_ip_port
+   - replacement: $1.svc.cluster.local
+     separator: .
+     sourceLabels:
+     - __meta_kubernetes_namespace
+     - service
+     targetLabel: __tmp_service_namespace
+ - port: reloader-web
+   path: /metrics
+   relabelings:
+   - separator: ":"
+     sourceLabels:
+     - __tmp_pod_ip_port
+     - __tmp_service_namespace
+     targetLabel: __address__
+   - regex: "(\\d+)\\.(\\d+)\\.(\\d+)\\.(\\d+):(\\d+)"
+     replacement: $5
+     sourceLabels:
+     - __address__
+     targetLabel: __tmp_pod_ip_port
+   - replacement: $1.svc.cluster.local
+     separator: .
+     sourceLabels:
+     - __meta_kubernetes_namespace
+     - service
+     targetLabel: __tmp_service_namespace


@@ (root level) @@
# security.istio.io/v1/AuthorizationPolicy/prometheus-auth-require-jwt
! + one document added:
+ ---
+ # Source: kube-prometheus-stack/templates/extra-objects.yaml
+ apiVersion: security.istio.io/v1
+ kind: AuthorizationPolicy
+ metadata:
+   name: prometheus-auth-require-jwt
+ spec:
+   action: ALLOW
+   rules:
+   - from:
+     - source:
+         requestPrincipals:
+         - "*"
+     - source:
+         namespaces:
+         - grafana
+         - monitoring
+         - "team-*"
+   selector:
+     matchLabels:
+       otomi.io/auth-policy: prometheus

@@ (root level) @@
# security.istio.io/v1/AuthorizationPolicy/monitoring-auth-require-jwt
! + one document added:
+ ---
+ # Source: kube-prometheus-stack/templates/extra-objects.yaml
+ apiVersion: security.istio.io/v1
+ kind: AuthorizationPolicy
+ metadata:
+   name: monitoring-auth-require-jwt
+ spec:
+   action: ALLOW
+   rules:
+   - from:
+     - source:
+         requestPrincipals:
+         - "*"
+     - source:
+         serviceAccounts:
+         - monitoring/po-prometheus
+   selector:
+     matchLabels:
+       otomi.io/auth-policy: monitoring

@@ (root level) @@
# security.istio.io/v1/AuthorizationPolicy/grafana/monitoring-auth-require-jwt
! + one document added:
+ ---
+ # Source: kube-prometheus-stack/templates/extra-objects.yaml
+ apiVersion: security.istio.io/v1
+ kind: AuthorizationPolicy
+ metadata:
+   name: monitoring-auth-require-jwt
+   namespace: grafana
+ spec:
+   action: ALLOW
+   rules:
+   - from:
+     - source:
+         requestPrincipals:
+         - "*"
+     - source:
+         serviceAccounts:
+         - monitoring/po-prometheus
+   selector:
+     matchLabels:
+       otomi.io/auth-policy: monitoring


@@ spec.alerting.alertmanagers @@
! - one list entry removed:
- - name: admin-po-alertmanager
-   apiVersion: v2
-   namespace: team-admin
-   port: http-web
-   scheme: http

@@ spec.alerting.alertmanagers.po-alertmanager @@
! + one map entry added:
+ relabelings:
+ - replacement: "po-alertmanager.monitoring.svc.cluster.local:9093"
+   targetLabel: __address__

@@ spec.alerting.alertmanagers.demo-po-alertmanager @@
! + one map entry added:
+ relabelings:
+ - replacement: "demo-po-alertmanager.team-demo.svc.cluster.local:9093"
+   targetLabel: __address__

@@ spec.alerting.alertmanagers.dev-po-alertmanager @@
! + one map entry added:
+ relabelings:
+ - replacement: "dev-po-alertmanager.team-dev.svc.cluster.local:9093"
+   targetLabel: __address__

@@ spec.podMetadata @@
! - one map entry removed:
- annotations:
-   sidecar.istio.io/inject: "true"
! + one map entry added:
+ labels:
+   istio.io/rev: 1-26-0
+   otomi.io/auth: platform
+   otomi.io/auth-policy: prometheus


@@ spec.rules @@
! - one list entry removed:
- - backendRefs:
-   - name: po-prometheus
-     port: 9090
-   filters:
-   - type: RequestHeaderModifier
-     requestHeaderModifier:
-       remove:
-       - authorization
-   matches:
-   - path:
-       type: PathPrefix
-       value: /
! + one list entry added:
+ - backendRefs:
+   - name: po-prometheus
+     port: 9090
+   matches:
+   - path:
+       type: PathPrefix
+       value: /


@@ data.core.yaml @@
! ± value change in multiline text (two inserts, one deletion)
  adminApps:
  - deps:
    - prometheus
    ingress:
  
  [146 lines unchanged)]
  
    tags:
    - ai
  - ingress:
    - auth: true
-     namespace: tekton-pipelines
+     namespace: tekton-dashboard
      port: 9097
      removeRequestHeaders:
      - authorization
      svc: tekton-dashboard
  
  [500 lines unchanged)]
  
      name: kfp
    - app: kyverno
      disableIstioInjection: true
      name: kyverno
+   - app: tekton-pipelines
+     name: tekton-dashboard
    - disableIstioInjection: true
      disablePolicyChecks: true
      name: tekton-pipelines
    - disableIstioInjection: true
  
  [59 lines unchanged)]
  
      svc: tekton-dashboard
      type: public
    name: tekton
    ownHost: true



@@ spec.template.metadata.labels @@
! + two map entries added:
+ otomi.io/auth: platform
+ otomi.io/auth-policy: platform



@@ spec.http @@
# networking.istio.io/v1beta1/VirtualService/tekton-dev-linodeapl-net
! - one list entry removed:
- - match:
-   - uri:
-       prefix: /
-   rewrite:
-     uri: /
-   route:
-   - destination:
-       host: tekton-dashboard.tekton-pipelines.svc.cluster.local
-       port:
-         number: 9097
-     headers:
-       request:
-         set:
-           # fix for istio (=envoy) incorrectly setting proto to http
- # (@see https://github.com/istio/istio/issues/7964):
- X-Forwarded-Proto: https
-         remove:
-         - authorization
! + one list entry added:
+ - match:
+   - uri:
+       prefix: /
+   rewrite:
+     uri: /
+   route:
+   - destination:
+       host: tekton-dashboard.tekton-dashboard.svc.cluster.local
+       port:
+         number: 9097
+     headers:
+       request:
+         set:
+           # fix for istio (=envoy) incorrectly setting proto to http
+ # (@see https://github.com/istio/istio/issues/7964):
+ X-Forwarded-Proto: https
+         remove:
+         - authorization


@@ spec.template.metadata.labels @@
! + two map entries added:
+ otomi.io/auth: platform
+ otomi.io/auth-policy: monitoring-demo


@@ spec.rules @@
! - one list entry removed:
- - backendRefs:
-   - name: demo-po-grafana
-     port: 80
-     group: 
-     kind: Service
-     weight: 1
-   filters:
-   - type: RequestHeaderModifier
-     requestHeaderModifier:
-       remove:
-       - authorization
-   matches:
-   - path:
-       type: PathPrefix
-       value: /
! + one list entry added:
+ - backendRefs:
+   - name: demo-po-grafana
+     port: 80
+     group: 
+     kind: Service
+     weight: 1
+   matches:
+   - path:
+       type: PathPrefix
+       value: /


@@ spec.podMetadata @@
! - one map entry removed:
- annotations:
-   sidecar.istio.io/inject: "true"

@@ spec.podMetadata.labels @@
! + three map entries added:
+ istio.io/rev: 1-26-0
+ otomi.io/auth: platform
+ otomi.io/auth-policy: monitoring-demo


@@ spec.rules @@
! - one list entry removed:
- - backendRefs:
-   - name: demo-po-alertmanager
-     port: 9093
-   filters:
-   - type: RequestHeaderModifier
-     requestHeaderModifier:
-       remove:
-       - authorization
-   matches:
-   - path:
-       type: PathPrefix
-       value: /
! + one list entry added:
+ - backendRefs:
+   - name: demo-po-alertmanager
+     port: 9093
+   matches:
+   - path:
+       type: PathPrefix
+       value: /


@@ (root level) @@
# security.istio.io/v1/AuthorizationPolicy/monitoring-demo-auth-require-jwt
! + one document added:
+ ---
+ # Source: kube-prometheus-stack/templates/extra-objects.yaml
+ apiVersion: security.istio.io/v1
+ kind: AuthorizationPolicy
+ metadata:
+   name: monitoring-demo-auth-require-jwt
+ spec:
+   action: ALLOW
+   rules:
+   - from:
+     - source:
+         requestPrincipals:
+         - "*"
+     - source:
+         namespaces:
+         - team-demo
+     - source:
+         serviceAccounts:
+         - monitoring/po-prometheus
+   selector:
+     matchLabels:
+       otomi.io/auth-policy: monitoring-demo


@@ spec.endpoints.0 @@
# monitoring.coreos.com/v1/ServiceMonitor/team-demo/po-alertmanager-team-demo
! + one map entry added:
+ relabelings:
+ - targetLabel: __address__
+   replacement: "demo-po-alertmanager.team-demo.svc.cluster.local:9093"

@@ spec.endpoints.0 @@
# monitoring.coreos.com/v1/ServiceMonitor/po-grafana-team-demo
! + one map entry added:
+ relabelings:
+ - targetLabel: __address__
+   replacement: "demo-po-grafana.team-demo.svc.cluster.local:80"


@@ spec.template.metadata.labels @@
! + two map entries added:
+ otomi.io/auth: platform
+ otomi.io/auth-policy: platform


@@ spec.rules @@
# gateway.networking.k8s.io/v1/HTTPRoute/tekton-demo
! - one list entry removed:
- - backendRefs:
-   - name: demo-tekton-dashboard
-     group: 
-     kind: Service
-     port: 9097
-   filters:
-   - type: RequestHeaderModifier
-     requestHeaderModifier:
-       remove:
-       - authorization
-   matches:
-   - path:
-       type: PathPrefix
-       value: /
! + one list entry added:
+ - backendRefs:
+   - name: demo-tekton-dashboard
+     group: 
+     kind: Service
+     port: 9097
+   matches:
+   - path:
+       type: PathPrefix
+       value: /


@@ spec.template.metadata.labels @@
! + two map entries added:
+ otomi.io/auth: platform
+ otomi.io/auth-policy: monitoring-dev


@@ spec.rules @@
! - one list entry removed:
- - backendRefs:
-   - name: dev-po-grafana
-     port: 80
-     group: 
-     kind: Service
-     weight: 1
-   filters:
-   - type: RequestHeaderModifier
-     requestHeaderModifier:
-       remove:
-       - authorization
-   matches:
-   - path:
-       type: PathPrefix
-       value: /
! + one list entry added:
+ - backendRefs:
+   - name: dev-po-grafana
+     port: 80
+     group: 
+     kind: Service
+     weight: 1
+   matches:
+   - path:
+       type: PathPrefix
+       value: /


@@ spec.podMetadata @@
! - one map entry removed:
- annotations:
-   sidecar.istio.io/inject: "true"

@@ spec.podMetadata.labels @@
! + three map entries added:
+ istio.io/rev: 1-26-0
+ otomi.io/auth: platform
+ otomi.io/auth-policy: monitoring-dev


@@ spec.rules @@
! - one list entry removed:
- - backendRefs:
-   - name: dev-po-alertmanager
-     port: 9093
-   filters:
-   - type: RequestHeaderModifier
-     requestHeaderModifier:
-       remove:
-       - authorization
-   matches:
-   - path:
-       type: PathPrefix
-       value: /
! + one list entry added:
+ - backendRefs:
+   - name: dev-po-alertmanager
+     port: 9093
+   matches:
+   - path:
+       type: PathPrefix
+       value: /


@@ (root level) @@
# security.istio.io/v1/AuthorizationPolicy/monitoring-dev-auth-require-jwt
! + one document added:
+ ---
+ # Source: kube-prometheus-stack/templates/extra-objects.yaml
+ apiVersion: security.istio.io/v1
+ kind: AuthorizationPolicy
+ metadata:
+   name: monitoring-dev-auth-require-jwt
+ spec:
+   action: ALLOW
+   rules:
+   - from:
+     - source:
+         requestPrincipals:
+         - "*"
+     - source:
+         namespaces:
+         - team-dev
+     - source:
+         serviceAccounts:
+         - monitoring/po-prometheus
+   selector:
+     matchLabels:
+       otomi.io/auth-policy: monitoring-dev


@@ spec.endpoints.0 @@
# monitoring.coreos.com/v1/ServiceMonitor/team-dev/po-alertmanager-team-dev
! + one map entry added:
+ relabelings:
+ - targetLabel: __address__
+   replacement: "dev-po-alertmanager.team-dev.svc.cluster.local:9093"

@@ spec.endpoints.0 @@
# monitoring.coreos.com/v1/ServiceMonitor/po-grafana-team-dev
! + one map entry added:
+ relabelings:
+ - targetLabel: __address__
+   replacement: "dev-po-grafana.team-dev.svc.cluster.local:80"


@@ spec.template.metadata.labels @@
! + two map entries added:
+ otomi.io/auth: platform
+ otomi.io/auth-policy: platform


@@ spec.rules @@
# gateway.networking.k8s.io/v1/HTTPRoute/tekton-dev
! - one list entry removed:
- - backendRefs:
-   - name: dev-tekton-dashboard
-     group: 
-     kind: Service
-     port: 9097
-   filters:
-   - type: RequestHeaderModifier
-     requestHeaderModifier:
-       remove:
-       - authorization
-   matches:
-   - path:
-       type: PathPrefix
-       value: /
! + one list entry added:
+ - backendRefs:
+   - name: dev-tekton-dashboard
+     group: 
+     kind: Service
+     port: 9097
+   matches:
+   - path:
+       type: PathPrefix
+       value: /

[Copilot]

Pull request overview

This PR extends the Gateway API OAuth2-proxy auth flow by validating the contents of bearer tokens (JWT) at the workload level using Istio RequestAuthentication + AuthorizationPolicy, and adjusts platform components (notably Tekton Dashboard) to support mesh inclusion and labeling.

Changes:

• Add workload-scoped JWT validation/authorization via new authpolicy-jwt.gotmpl and a platform RequestAuthentication selecting labeled workloads.
• Update multiple platform/team apps to add otomi.io/auth / otomi.io/auth-policy pod labels and swap Gateway policies to authpolicy-oauth2-ext.gotmpl.
• Move Tekton Dashboard from tekton-pipelines to a new tekton-dashboard namespace, with a runtime-upgrade hook to remove the old Argo CD Application.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
values/tekton-dashboard/tekton-dashboard.gotmpl	Add auth-related pod labels for Tekton Dashboard pods.
values/tekton-dashboard/tekton-dashboard-teams.gotmpl	Add auth-related pod labels for team dashboards.
values/tekton-dashboard/tekton-dashboard-raw.gotmpl	Stop stripping Authorization header; switch to authpolicy-oauth2-ext.
values/prometheus-operator/prometheus-operator.gotmpl	Add auth labels/injection-related metadata; add relabelings; adjust alerting endpoints loop.
values/otomi-console/otomi-console.gotmpl	Add auth pod labels; switch to authpolicy-oauth2-ext.
values/otomi-api/otomi-api.gotmpl	Switch to authpolicy-oauth2-ext.
values/kubernetes-gateways/kubernetes-gateways-raw.gotmpl	Update oauth2-proxy ReferenceGrant namespaces; add platform JWT AuthorizationPolicy + RequestAuthentication.
values/kubeflow-pipelines/kubeflow-pipelines.gotmpl	Add pod labels to target KFP UI for platform auth/policy.
values/kubeflow-pipelines/kubeflow-pipelines-raw.gotmpl	Switch to authpolicy-oauth2-ext.
values/harbor/harbor.gotmpl	Add auth pod labels (Harbor portal).
values/harbor/harbor-raw.gotmpl	Switch to authpolicy-oauth2-ext; update excluded paths parameter naming/list.
values/argocd/argocd.gotmpl	Add auth pod labels (Argo CD server); switch to authpolicy-oauth2-ext.
values/apl-harbor-operator/apl-harbor-operator.gotmpl	Add pod labels intended for Istio injection selection.
src/common/runtime-upgrades/runtime-upgrades.ts	Wire new Tekton Dashboard removal into v4.16.0 runtime upgrade pre-step.
src/common/runtime-upgrades/ingress-resource-pre-migration.ts	Add removeTektonDashboardApp runtime upgrade helper.
src/cmd/apply-as-apps.ts	Make removeApplication tolerant to 404 via getFinalizers returning undefined.
helmfile.d/snippets/derived.gotmpl	Fix Keycloak backchannel base URL to include :8080.
helmfile.d/snippets/authpolicy-oauth2-ext.gotmpl	Rename exclusionPaths → excludePaths.
helmfile.d/snippets/authpolicy-jwt.gotmpl	New JWT-based Istio AuthorizationPolicy template.
helmfile.d/helmfile-60.teams.yaml.gotmpl	Switch team routes to authpolicy-oauth2-ext; add team JWT auth policies and adjust pod labels.
helmfile.d/helmfile-08.init.yaml.gotmpl	Switch monitoring route to authpolicy-oauth2-ext; add JWT auth policies.
helmfile.d/helmfile-04.init.yaml.gotmpl	Move Tekton Dashboard releases into tekton-dashboard namespace.
core.yaml	Add tekton-dashboard namespace.
charts/team-ns/templates/servicemonitors/service-monitors.yaml	Add relabelings to force stable scrape target addresses.
charts/kubeflow-pipelines/values.yaml	Add mlPipelineUi.podLabels values hook.
charts/kubeflow-pipelines/templates/ml-pipeline-ui/deployment.yaml	Render mlPipelineUi.podLabels into pod labels.
charts/apl-harbor-operator/values.yaml	Add podLabels values hook.
charts/apl-harbor-operator/templates/deployment.yaml	Render podLabels into pod labels.
charts/apl-harbor-operator/Chart.yaml	Bump chart version to 0.1.1.

Comments suppressed due to low confidence (1)

src/common/runtime-upgrades/ingress-resource-pre-migration.ts:102

• removeTektonDashboardApp only deletes the Argo CD Application tekton-pipelines-tekton-dashboard, but this PR also moves the tekton-dashboard-artifacts release to a new namespace. The old Application tekton-pipelines-tekton-dashboard-artifacts will likely remain and keep syncing (or error) unless it is removed as well. Consider deleting both old Applications during the runtime upgrade.

[ferruhcihan]

When I log in to the console via the Keycloak login page, I see a “JWT verification fails” error. Then checked the workloads with the auth-policy: platform label and noticed that some gotmpl files (such as argocd, harbor, otomi-console, and tekton-dashboard) are missing the istio injection label. Meanwhile, this label was added to prometheus, alertmanager, and kubeflow in this PR. After waiting a while and refreshing the page, the console loads correctly.

Is this JWT issue related to that? Should we also add the istio injection label to those gotmpl files?

[CasLubbers]

Looks good just some questions to get a better understanding. Also for future apps we need to add the following labels to have istio-injection working?

podLabels:
  otomi.io/auth: platform
  otomi.io/auth-policy: platform

[merll]

When I log in to the console via the Keycloak login page, I see a “JWT verification fails” error. Then checked the workloads with the auth-policy: platform label and noticed that some gotmpl files (such as argocd, harbor, otomi-console, and tekton-dashboard) are missing the istio injection label. Meanwhile, this label was added to prometheus, alertmanager, and kubeflow in this PR. After waiting a while and refreshing the page, the console loads correctly.

Is this JWT issue related to that? Should we also add the istio injection label to those gotmpl files?

The JWT issue should be fixed, as I added an environment variable to refresh the signatures more frequently in the meantime.

Regarding injection, there are two different approaches: Having the entire namespace labeled so all workloads get injected with a sidecar, or labeling individual workloads. I tried in this PR not to change too many at a time, so where the namespace was set to disableIstioInjection I added individual pods.

The namespaces harbor and otomi have the labels for injection so they do not need to be added individually. Tekton I moved to a different namespace, because the Istio sidecar requires a privileged container, which is specifically disallowed in the tekton-pipelines namespace.

For ArgoCD, the added labels do in fact not have an effect. Since ArgoCD has its own OIDC authentication and would reject an invalid Bearer token by itself, we only add OAuth2-Proxy for a more consistent SSO experience. Due to the resource overhead of the sidecars, I discussed with Jeho to postpone addition to the Istio mesh until we move to ambient mode.

[merll]

Looks good just some questions to get a better understanding. Also for future apps we need to add the following labels to have istio-injection working?

podLabels:
  otomi.io/auth: platform
  otomi.io/auth-policy: platform

These two labels are not for Istio injection - but Istio injection is required for these to have an effect. The otomi.io/auth label is used as a selector for RequestAuthentication (decode and validate bearer token), whereas otomi.io/auth-policy is used to pick a AuthenticationPolicy which might have exemptions from the JWT requirement for internal requests.

We will use platform for the RequestAuthentication for pretty much all cases (but could technically add more). The auth-policy platform means to authenticate all requests, without exceptions, if they are valid.

In addition, we still need an AuthenticationPolicy attached to the Gateway, which directs to OAuth2-Proxy add the Bearer token if not already set. This is then not based on the workload, but the hostname. This makes it in some instances a little complicated (e.g. Harbor).

[Copilot]

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 1 comment.

[Copilot]

excludeNamespaces is passed a glob value ("team-*"), but Istio AuthorizationPolicy source.namespaces only matches exact namespace names (no wildcard/glob expansion). As a result, traffic from team namespaces won’t be exempted from the JWT requirement and can break in-cluster calls to Prometheus (e.g., team Grafana querying the platform Prometheus) unless they always present a JWT.

Consider generating the namespace list from teamConfig (e.g., range over teams and append team-<id>), or switch this exemption to excludeAccounts (serviceAccounts) if you want to allow specific workloads instead of all namespaces.