fix(oauth2-proxy): poll keycloak over the backchannel in wait-for-keycloak

[aweingarten]

📌 Summary

Fixes #3388 — the oauth2-proxy item (and, for this container, the resolution failure in #3420).

Note on scope: #3388 lists three independent items; this PR implements the oauth2-proxy one. The other two (gitops-global ignoring otomi.git, and Loki's object_store/schemaConfig being unsettable through _rawValues) are untouched. Since Fixes will auto-close #3388 on merge, those two should be split into their own issue — happy to open it, or leave this as a plain reference instead if you'd rather keep #3388 open until all three land.

The wait-for-keycloak init container polls the public issuer URL with a vanilla curlimages/curl:

args: ["while [ $(curl -sw '%{http_code}' https://keycloak.<domainSuffix>/realms/otomi -o /dev/null) -ne 200 ]; ..."]

The oauth2-proxy pod is explicitly sidecar-less (sidecar.istio.io/inject: "false"), so that URL is reached without the mesh, and it hits one of two walls depending on the deployment:

• the host is an Istio ServiceEntry with no in-cluster DNS record backing it, so it may not resolve at all (Internal ServiceEntry hosts (<svc>.<domainSuffix>) don't resolve for sidecar-less pods / init containers #3420);
• when it does resolve, the Otomi-signed wildcard cert isn't trusted by a stock curl image, so curl exits 60 (self-signed certificate in certificate chain).

Either way the container loops forever and oauth2-proxy never starts. No chart value exposes the init image, a CA mount, or the curl args, so there's no supported way out.

Worth noting the asymmetry this creates: oauth2-proxy's own config already tolerates the untrusted CA — ssl_insecure_skip_verify = {{ $v._derived.untrustedCA }} — so the gate is currently stricter than the app it gates.

The fix points the poll at _derived.oidcBaseUrlBackchannel (http://keycloak-keycloakx-http.keycloak:8080/realms/otomi), which already exists for exactly this purpose and is used by kubernetes-gateways for its JWKS endpoint. Plain HTTP to a ClusterIP Service: nothing to resolve through the mesh, nothing to trust.

🔍 Reviewer Notes

• The gate condition is unchanged. Both URLs return 200 only once the otomi realm is being served; only the route changes. It does not verify the public ingress path — but the init container never usefully verified that either, since it fails on DNS/TLS before reaching Keycloak.
• I picked this over the alternative (mount the platform CA and pass --cacert, following the custom-ca Secret pattern in argocd-raw/gitea-raw). Mounting a CA fixes the TLS half but leaves the DNS half, and it adds a Secret + volume + mount where a value that already exists does the whole job. Happy to switch if you'd rather the gate stay on the public URL.
• The other two items in Charts hardcode values that can't be overridden via apl-values #3388 (gitops-global ignoring otomi.git; Loki object_store/schemaConfig not settable through _rawValues) are untouched here — they're independent and larger. Splitting that issue may be worthwhile.
• The init container image is still curlimages/curl:latest with no value to override it. That part of the Charts hardcode values that can't be overridden via apl-values #3388 ask stands; it seemed better as its own change.

Template-only change. validate-templates/lint:hf need helmfile, which I don't have locally — leaving those to CI.

🧹 Checklist

• Code is readable, maintainable, and robust.
• Unit tests added/updated — n/a, this is a one-line template change covered by validate-templates

[CasLubbers]

The public URL check here isn't just "is Keycloak up", its an end-to-end smoke test (DNS, cert, ingress) gating oauth2-proxy, which is the platform's SSO entrypoint. Switching to oidcBaseUrlBackchannel fixes the sidecar-less DNS/TLS failure, but it also means the gate now passes even when the public path is broken. That defeats the point of gating on it in the first place.

If the goal is just don't crashloop forever, maybe worth keeping the public URL but adding CA trust + in-mesh DNS path (mount platform CA, skip the ServiceEntry issue) instead of swapping to backchannel. Otherwise, we lose the one check that verifies real users can actually reach Keycloak.

Alternative: make this configurable instead of picking one. Add a value (default true = current public-url behavior) so people who want the real e2e check keep it, and people hitting the DNS/ServiceEntry issue can opt into backchannel. Still need the CA trust fix for the public path though, otherwise the flag is pointless for anyone who leaves it on.

[Copilot]

Pull request overview

This PR fixes oauth2-proxy startup failures caused by the wait-for-keycloak init container polling Keycloak via the public issuer URL from a sidecar-less pod (where ServiceEntry DNS and custom CA trust can break the curl check). It switches the poll target to the existing backchannel in-cluster URL so the init container can reliably reach Keycloak and unblock oauth2-proxy.

Changes:

• Update wait-for-keycloak init container to poll _derived.oidcBaseUrlBackchannel instead of _derived.oidcBaseUrl.
• Add inline template comments documenting why the backchannel URL is required for this sidecar-less init container.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[aweingarten]

You are right, and I have reverted the URL swap. Routing the gate around the public path defeats what the gate is for.

Chasing your CA-trust suggestion turned up the actual root cause, which is narrower than what I originally wrote in the description:

• The pod sets sidecar.istio.io/inject: "false" — that is the whole oauth2-proxy pod, not just this init container. So the main container is in the same position: no sidecar, no mesh DNS, no sidecar-terminated TLS.
• The main container copes because it is handed ssl_insecure_skip_verify = {{ $v._derived.untrustedCA }} a few lines up in the same file. The init curl had no equivalent.
• _derived.untrustedCA is true when the issuer is custom-ca or letsencrypt-staging. custom-ca is the default issuer, so on an ordinary install that curl is verifying a platform-issued cert with no trust for it and loops forever at a TLS error, never reaching a status code at all.

So this was never really a DNS or ServiceEntry problem — my original description overreached there. It is one container missing the trust decision its sibling already makes.

The fix now keeps _derived.oidcBaseUrl and adds -k on exactly the same condition as ssl_insecure_skip_verify:

curl -s{{ if $v._derived.untrustedCA }}k{{ end }}w '%{http_code}' {{ $v._derived.oidcBaseUrl }}

That keeps your end-to-end property — DNS, ingress and the route are all still exercised, and the gate still fails when real users cannot reach Keycloak — while making the init container no more and no less permissive about certificates than the process it is gating for. It also means no new config knob, which I think is better than the toggle option you floated: a flag defaulting to the public URL would leave the default install still broken.

If you would rather have the CA mounted and verified properly instead of -k, I am happy to do that — but note it would then be stricter than the main container, which skips verification outright in this mode, so the init container could block on a cert that oauth2-proxy itself would have accepted. Matching the main container seemed like the more honest gate.