OpenShift 4.1 Docker login internal registry error: x509: certificate signed by unknown authority

[chrisc66]

I have an OpenShift 4.1 cluster installed recently. I am trying to access OpenShift cluster internal docker registry. It is a 3 master, 5 worker nodes cluster. I am having trouble with finding internal registry certificate.

Version: OpenShift 4.1

# oc version
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.9-201907311355+45479a5-dirty", GitCommit:"45479a5", GitTreeState:"dirty", BuildDate:"2019-07-31T21:33:07Z", GoVersion:"go1.11.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+8560dd6", GitCommit:"8560dd6", GitTreeState:"clean", BuildDate:"2019-07-31T21:30:54Z", GoVersion:"go1.11.6", Compiler:"gc", Platform:"linux/amd64"}

Steps To Reproduce

1. oc login with cluster admin (kubeadmin) credentials

# oc login
Authentication required for https://api.tariff.os.fyre.ibm.com:6443 (openshift)
Username: kubeadmin
Password:
Login successful.

2. docker login with same credentials

docker login default-route-openshift-image-registry.apps.tariff.os.fyre.ibm.com

Current Result

Error response from daemon: Get https://default-route-openshift-image-registry.apps.tariff.os.fyre.ibm.com/v1/users/: x509: certificate signed by unknown authority

Expected Result

Login Succeed

Additional Information

# oc project
Using project "openshift-image-registry" on server "https://api.tariff.os.fyre.ibm.com:6443".

# oc get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
image-registry   ClusterIP   172.30.237.41   <none>        5000/TCP   5d

# oc get imagestream
NAME                           IMAGE REPOSITORY                                                                                                        TAGS   UPDATED
image-stream-name   default-route-openshift-image-registry.apps.tariff.os.fyre.ibm.com/ibm-java-acceleration/ibm-java-acceleration-server

# pwd
/etc/docker/certs.d
# ls
access.redhat.com  
redhat.io                   
registry.redhat.io
cdn.redhat.com     
redhat.com                                                          
registry.access.redhat.com

I found that there is no internal registry certificate under /etc/docker/certs.d. So I added a /etc/docker/certs.d/default-route-openshift-image-registry.apps.tariff.os.fyre.ibm.com directory. Where and how should I get the ca.crt file?

[adambkaplan]

Since you're exposing via the default route, the registry uses the certificates configured by the ingress operator. Out of the box, the ingress operator uses a self-signed certificate.

You have a few options:

1. Configure docker to recognize the OpenShift registry as an insecure registry.
2. Update the ingress operator to use a cert signed by a trusted CA (ex: LetsEncrypt) [1]
3. Update the image registry operator to use a custom route, and provide it a cert signed by a trusted CA [2]

[1] https://docs.openshift.com/container-platform/4.1/networking/ingress-operator.html#nw-ingress-setting-a-custom-default-certificate_configuring-ingress
[2] https://docs.openshift.com/container-platform/4.1/registry/securing-exposing-registry.html

/close

[openshift-ci-robot]

@adambkaplan: Closing this issue.

In response to this:

Since you're exposing via the default route, the registry uses the certificates configured by the ingress operator. Out of the box, the ingress operator uses a self-signed certificate.

You have a few options:

1. Configure docker to recognize the OpenShift registry as an insecure registry.
2. Update the ingress operator to use a cert signed by a trusted CA (ex: LetsEncrypt) [1]
3. Update the image registry operator to use a custom route, and provide it a cert signed by a trusted CA [2]

[1] https://docs.openshift.com/container-platform/4.1/networking/ingress-operator.html#nw-ingress-setting-a-custom-default-certificate_configuring-ingress
[2] https://docs.openshift.com/container-platform/4.1/registry/securing-exposing-registry.html

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[YevheniiPokhvalii]

@chrisc66 was asking where to get the ca.crt file. I'm also wondering. Is it in the image-registry-tls secret in the openshift-image-registry namespace?

Edited: It seems that most service accounts in Kubernetes/Openshift have access to the root certificates via their secrets. So you can create your own secret for a service account and mount it to a pod:

apiVersion: v1
kind: Secret
metadata:
  name: sa1-token
  annotations:
    kubernetes.io/service-account.name: sa1
type: kubernetes.io/service-account-token

You may also need to run the update-ca-certificates command in a pod after that.