OIDC config via secret fails intermittently

[dfry]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When the OIDC configuration is provided with reference to a secret value, some times the secret is interpreted correctly, other times it is not, resulting in sending the actual string ($argo-oidc:clientid) to the OIDC provider instead of the value from the secret. When this happens, a delete of the argocd server pod will resolve the issue.

To Reproduce

Deploy argo with the below oidc config map elements. After a random period of time, even after a initial successful OIDC login has been performed, when the OIDC token refresh occurs, it will fail with the below warning messages in the logs.

apiVersion: v1
kind: Secret
metadata:
  name: argo-oidc
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: argocd
data:
  clientid:  redacted
  clientsecret: redacted
type: Opaque
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  application.instanceLabelKey: argocd.argoproj.io/instance
  kustomize.buildOptions: '--enable-helm'
  oidc.config: |
    name: Gitlab
    issuer: redacted
    clientID: $argo-oidc:clientid
    clientSecret: $argo-oidc:clientsecret
    requestedScopes: ["openid", "profile", "email", "read_api"]

Expected behavior

Secret references should always be resolved and not treated as plain text.

Screenshots

Version

Argo CD: v2.9.7+fbb6b20
Build Date: 2024-03-01T22:28:14Z
Go Version: go1.21.3
Go Compiler: gc
Platform: linux/amd64
jsonnet: v0.20.0
kustomize: v5.2.1 2023-10-19T20:13:51Z
Helm: v3.13.2+g2a2fb3b
kubectl: v0.24.17

Logs

time="2024-05-15T12:26:15Z" level=warning msg="config referenced '$argo-oidc:clientid', but key does not exist in secret"
time="2024-05-15T12:26:15Z" level=warning msg="config referenced '$argo-oidc:clientsecret', but key does not exist in secret"