argo with SSO login to provider with internal CA throws x509: certificate signed by unknown authority"

[giordyb]

Summary

I am trying to configure ARGO to authenticate via SSO to an internal Keycloak server. The keycloak server certificates are signed by an internal CA.

I added this section in the workflow-controller-configmap:

sso: |
    issuer: https://keycloak-url/auth/realms/master
    clientId:
      name: argo-sso-cred
      key: client-id
    clientSecret:
      name: argo-sso-cred
      key: client-secret
    redirectUrl: https://argo-url/argo/oauth2/callback
    scopes:
      - groups
    rbac:
      enabled: false

and in the argo-server deployment I changed the args as following:

- args:
            - server
            - --auth-mode
            - sso

after applying the configmap and changing the deployment the argo-server it starts but goes on a crash-loop with the following error:

time="2020-11-03T10:57:53Z" level=fatal msg="Get https://keycloak-url/auth/realms/master/.well-known/openid-configuration: x509: certificate signed by unknown authority"                                         │

is there a way to inject the CA in argo server other than having to create a custom docker image?

Diagnostics

What Kubernetes provider are you using?
1.16.8

What version of Argo Workflows are you running?
2.11.6

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

I'm pretty sure you'll need to mount a volume with your certificates. Maybe something like this:

https://medium.com/faun/mount-ssl-certificates-in-kubernetes-pod-with-secret-8aca220896e6

[giordyb]

@alexec that did the trick thank you.

[dsetty25]

I have done the same configurations .but instead of https I am trying with http.

time="2020-12-29T07:15:07.029Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2020-12-29T07:15:07Z" grpc.time_ms=0.17 span.kind=server system=grpc

But I am getting the above error when I hit the argo-server url.

Any suggestions what would have been wrong?

[alexec]

@dsetty25 can you try in Incognito mode please?

[dsetty25]

Hi , I tried in incognito as wellbut same issue exists.

Currently I have added in server-deployment.yaml
args: - server - --auth-mode - sso

And in values.yaml

sso:
# #SSO configuration when SSO is specified as a server auth mode.
# #All the values are requied. SSO is activated by adding --auth-mode=sso
# #to the server command line.
#
# #The root URL of the OIDC identity provider.
issuer: http://<keycloak_ip>/auth/realms/demo
# #Name of a secret and a key in it to retrieve the app OIDC client ID from.
clientId:
name: argo
key: client-id
# #Name of a secret and a key in it to retrieve the app OIDC client secret from.
clientSecret:
name: "argo-server-sso"
key: client-secret
# # The OIDC redirect URL. Should be in the form /oauth2/callback.
redirectUrl: http:///argo/oauth2/callback

And in keycloak ui , I have created client and client credentials.

kubectl create secret generic "argo-server-sso" --from-literal=client-secret=9a9c60ba-647d-480c-b6fa-82c19caad26a
kubectl create secret generic "argo" --from-literal=client-id=argo

After hitting the argo server url,manually I need to click on login option but after that keycloak page appears and then again a popup will come "Failed to login:Unauthorized"
Server logs:
kubectl logs argo-server-5c7f8c5cbb-9fcqk
time="2021-01-20T12:06:26.876Z" level=info authModes="[sso]" baseHRef=/ managedNamespace= namespace=default secure=false
time="2021-01-20T12:06:26.877Z" level=warning msg="You are running in insecure mode. Learn how to enable transport layer security: https://argoproj.github.io/argo/tls/"
time="2021-01-20T12:06:26.877Z" level=info msg="config map" name=argo-workflow-controller-configmap
time="2021-01-20T12:06:28.318Z" level=info msg="SSO configuration" clientId="{{argo} client-id }" issuer="http://10.xx.xx.xx:xxxx/auth/realms/demo" redirectUrl="http://xx/argo/oauth2/callback"
time="2021-01-20T12:06:28.318Z" level=info msg="SSO enabled"
time="2021-01-20T12:06:28.322Z" level=info msg="Starting Argo Server" instanceID= version=v2.12.2
time="2021-01-20T12:06:28.322Z" level=info msg="Creating event controller" operationQueueSize=16 workerCount=4
time="2021-01-20T12:06:28.323Z" level=info msg="Argo Server started successfully on http://localhost:2746"
time="2021-01-20T12:07:21.990Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2021-01-20T12:07:21Z" grpc.time_ms=0.379 span.kind=server system=grpc
time="2021-01-20T12:07:22.009Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2021-01-20T12:07:22Z" grpc.time_ms=0.075 span.kind=server system=grpc

[allapavan1208]

@giordyb Hi, I have same SSO self signed issue and trying to mount the CA but some how it is failing. Can you please share the configuration for mounting the cert and passing cert to argo server startup?

[giordyb]

Hi @allapavan1208 you need to create a configmap (es. myca) with the ca cert and then modify the deployment like this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-server
spec:
  selector:
    matchLabels:
      app: argo-server
  template:
    metadata:
      labels:
        app: argo-server
    spec:
      containers:
        - args:
            - server
            - --auth-mode
            - sso
          image: argoproj/argocli:v2.11.7
          name: argo-server
          env:
            - name: BASE_HREF
              value: /argo-workflow/
          ports:
            - containerPort: 2746
              name: web
          readinessProbe:
            httpGet:
              path: /
              port: 2746
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 20
          volumeMounts:
            - mountPath: /tmp
              name: tmp
            - name: ca-pemstore
              mountPath: /etc/ssl/certs/ca.crt
              subPath: ca.crt
              readOnly: false
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: argo-server
      volumes:
        - emptyDir: {}
          name: tmp
        - name: ca-pemstore
          configMap:
            name: myca

[asaorgad]

@alexec that did the trick thank you.

Can you give a detailed explanation how to fix it , I'm facing the same issue.

[giordyb]

@alexec that did the trick thank you.

Can you give a detailed explanation how to fix it , I'm facing the same issue.

I showed the fix in my previous post, you need to mount your own ssl CA in the container and mount it as /etc/ssl/certs/ca.crt

[asaorgad]

@alexec that did the trick thank you.

Can you give a detailed explanation how to fix it , I'm facing the same issue.

I showed the fix in my previous post, you need to mount your own ssl CA in the container and mount it as /etc/ssl/certs/ca.crt

so my question is , how do i generate or how where from should i export this CA certs ,?
can this tool the the job : https://github.com/smallstep/autocert ?
excuse my ignorance

[giordyb]

@alexec that did the trick thank you.

Can you give a detailed explanation how to fix it , I'm facing the same issue.

I showed the fix in my previous post, you need to mount your own ssl CA in the container and mount it as /etc/ssl/certs/ca.crt

so my question is , how do i generate or how where from should i export this CA certs ,?
can this tool the the job : https://github.com/smallstep/autocert ?
excuse my ignorance

You need to get/export the CA that signed the SSL certificate that is installed on your SSO authenticator and add it to the Argo Server container as explained above. Unfortunately I don't know anything about your specific configuration so I cannot help you further. In my case the Keycloak server's certificate was internal and it was signed by the CA created by Active Directory certificate server.

[asaorgad]

@alexec that did the trick thank you.

Can you give a detailed explanation how to fix it , I'm facing the same issue.

I showed the fix in my previous post, you need to mount your own ssl CA in the container and mount it as /etc/ssl/certs/ca.crt

so my question is , how do i generate or how where from should i export this CA certs ,?
can this tool the the job : https://github.com/smallstep/autocert ?
excuse my ignorance

You need to get/export the CA that signed the SSL certificate that is installed on your SSO authenticator and add it to the Argo Server container as explained above. Unfortunately I don't know anything about your specific configuration so I cannot help you further. In my case the Keycloak server's certificate was internal and it was signed by the CA created by Active Directory certificate server.

That's makes more sense, thanks alot.

[jinnjwu]

I have same issue for 3 month