CRC Image Registry docker login issue on Windows - 127.0.0.1:80 connection refused #2354

ShrayRastogi · 2021-05-19T03:45:04Z

OS: Windows 10
Hyper-V
Laptop

When trying to docker login to CRC internal image registry default-route-openshift-image-registry.apps-crc.testing throws error.
Its only happening on windows. Works fine when running on Linux.

Version
Client Version: 4.7.5
Kubernetes Version: v1.20.0+bafe72f

Steps To Reproduce
crc daemon
crc start (Windows)
docker login -u kubeadmin -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing
Current Result
Get http://default-route-openshift-image-registry.apps-crc.testing/v2/: dial tcp 127.0.0.1:80: connect: connection refused

Expected Result
Login Succeeded!

Additional Information
I have tried the following -

oc patch configs.imageregistry.operator.openshift.io/cluster --type merge -p '{"spec":{"defaultRoute":true}}'
oc policy add-role-to-user registry-editor kubeadmin
oc policy add-role-to-user registry-viewer kubeadmin
oc get clusteroperator image-registry
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
image-registry 4.7.5 True False False 15d

oc registry info --public
default-route-openshift-image-registry.apps-crc.testing

oc get pods -n openshift-image-registry NAME READY STATUS RESTARTS AGE
cluster-image-registry-operator-66b48f9bd-ln99j 1/1 Running 0 33d
image-registry-77979c4dc7-nszrj 1/1 Running 0 15d
node-ca-q6gc8 1/1 Running 0 34d

oc edit configs.imageregistry.operator.openshift.io
apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
creationTimestamp: "2021-04-08T06:11:05Z"
finalizers:

imageregistry.operator.openshift.io/finalizer
generation: 5
name: cluster
resourceVersion: "28869"
selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
uid: 4644048c-fdd2-4c9d-a86a-d398326ee3e3
spec:
defaultRoute: true
httpSecret: a2361d60ce4c3abf3d39b2810532c3ad6252772f0c1894f68013bbb82904475f8ecd5425ab8df66fad0a6aee616a4c1fc95bb463440b07fced01908543e804d6
logLevel: Normal
managementState: Managed
observedConfig: null
operatorLogLevel: Normal
proxy: {}
replicas: 1
requests:
read:
maxWaitInQueue: 0s
write:
maxWaitInQueue: 0s
rolloutStrategy: RollingUpdate
storage:
managementState: Managed
pvc:
claim: crc-image-registry-storage
unsupportedConfigOverrides: null
status:
conditions:
lastTransitionTime: "2021-04-09T06:37:34Z"
reason: PVC Exists
status: "True"
type: StorageExists
lastTransitionTime: "2021-04-26T15:55:33Z"
message: The registry is ready
reason: Ready
status: "True"
type: Available
lastTransitionTime: "2021-04-26T15:55:56Z"
message: The registry is ready
reason: Ready
status: "False"
type: Progressing
lastTransitionTime: "2021-04-26T15:55:33Z"
status: "False"
type: Degraded
lastTransitionTime: "2021-04-08T06:11:06Z"
status: "False"
type: Removed
lastTransitionTime: "2021-04-08T06:11:06Z"
reason: AsExpected
status: "False"
type: ImageRegistryCertificatesControllerDegraded
lastTransitionTime: "2021-04-08T06:11:07Z"
reason: AsExpected
status: "False"
type: ImageConfigControllerDegraded
lastTransitionTime: "2021-04-08T06:11:07Z"
reason: AsExpected
status: "False"
type: NodeCADaemonControllerDegraded
generations:
group: apps
hash: ""
lastGeneration: 5
name: image-registry
namespace: openshift-image-registry
resource: deployments
group: apps
hash: ""
lastGeneration: 0
name: node-ca
namespace: openshift-image-registry
resource: daemonsets
observedGeneration: 5
readyReplicas: 0
storage:
managementState: Managed
pvc:
claim: crc-image-registry-storage
storageManaged: true

The text was updated successfully, but these errors were encountered:

praveenkumar · 2021-05-19T04:08:02Z

@ShrayRastogi Are you using the vsock as part of config? Can you provide output of crc config view ?

guillaumerose · 2021-05-19T06:31:07Z

In crc 1.25.0, the registry on the port 80 is not exposed. It is fixed in 1.26.0. Can you give it a try?

Thanks.

ShrayRastogi · 2021-05-19T06:44:17Z

In crc 1.25.0, the registry on the port 80 is not exposed. It is fixed in 1.26.0. Can you give it a try?

Thanks.

I upgraded the CRC to v4.7.8 and 1.26.0+31f06c09, still same error

ShrayRastogi · 2021-05-19T06:45:16Z

@ShrayRastogi Are you using the vsock as part of config? Can you provide output of crc config view ?

crc config view

consent-telemetry : yes
network-mode : vsock

ShrayRastogi · 2021-05-19T07:03:04Z

@praveenkumar @guillaumerose -
Here is the gist of CRC Start logs and crc daemon logs - https://gist.github.com/ShrayRastogi/77b08a9738d59f510530981d027fcf8b

praveenkumar · 2021-05-19T07:42:27Z

In crc 1.25.0, the registry on the port 80 is not exposed. It is fixed in 1.26.0. Can you give it a try?
Thanks.

I upgraded the CRC to v4.7.8 and 1.26.0+31f06c09, still same error

Have you deleted the old crc instance and started fresh with new version. I just tested vsock on linux and it does work as expected.

$ dig +short default-route-openshift-image-registry.apps-crc.testing
127.0.0.1
$ podman login -u kubeadmin -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing --tls-verify=false
Login Succeeded!

ShrayRastogi · 2021-05-19T08:06:09Z

In crc 1.25.0, the registry on the port 80 is not exposed. It is fixed in 1.26.0. Can you give it a try?
Thanks.

I upgraded the CRC to v4.7.8 and 1.26.0+31f06c09, still same error

Have you deleted the old crc instance and started fresh with new version. I just tested vsock on linux and it does work as expected.
$ dig +short default-route-openshift-image-registry.apps-crc.testing
127.0.0.1
$ podman login -u kubeadmin -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing --tls-verify=false
Login Succeeded!

@praveenkumar - Yes, I deleted the old instance and started fresh. It is working on linux for me as well, however, not working on Windows Enterprise. Please help! Let me know if you need any other logs or something.

I am also able to login to default image registry via VSCode Docker Extension (using URL like https://default-route-openshift-image-registry.apps-crc.testing). However, via command line - docker login or docker push, I get same error as above. Via command-line docker by default is going to http://default-route-openshift-image-registry.apps-crc.testing instead of https.

guillaumerose · 2021-05-19T08:52:58Z

I am also able to login to default image registry via VSCode Docker Extension (using URL like https://default-route-openshift-image-registry.apps-crc.testing). However, via command line - docker login or docker push, I get same error as above. Via command-line docker by default is going to http://default-route-openshift-image-registry.apps-crc.testing instead of https.

This is because you are using Docker Desktop or running docker/podman in WSL I guess.
If it is the case, please look at this issue #2092 (comment).
Let me know if it helps or not.

ShrayRastogi · 2021-05-19T09:02:26Z

I am also able to login to default image registry via VSCode Docker Extension (using URL like https://default-route-openshift-image-registry.apps-crc.testing). However, via command line - docker login or docker push, I get same error as above. Via command-line docker by default is going to http://default-route-openshift-image-registry.apps-crc.testing instead of https.

This is because you are using Docker Desktop or running docker/podman in WSL I guess.
If it is the case, please look at this issue #2092 (comment).
Let me know if it helps or not.

@guillaumerose @praveenkumar - Yes, I am using Docker Desktop. However, I am getting this error now -
docker login -u kubeadmin -p $(oc whoami -t) host.docker.internal
Error response from daemon: Get https://host.docker.internal/v2/: x509: certificate is valid for *.apps-crc.testing, not host.docker.internal

guillaumerose · 2021-05-19T09:22:11Z

You need to configure certificates then. Follow this: https://docs.docker.com/docker-for-windows/#adding-tls-certificates

ShrayRastogi · 2021-05-19T09:27:06Z

You need to configure certificates then. Follow this: https://docs.docker.com/docker-for-windows/#adding-tls-certificates

@praveenkumar @guillaumerose - So there is no any other alternative than to use host.docker.internal during docker login and docker push commands?

guillaumerose · 2021-05-19T11:09:20Z

I can't find an alternative. But for sure, we can simplify this and crc could configure that for you.

ShrayRastogi · 2021-05-19T14:25:14Z

I can't find an alternative. But for sure, we can simplify this and crc could configure that for you.

@praveenkumar @guillaumerose - That would be super helpful. So, by using host.docker.internal, do I need to select external image registry and creating image pull secret while creating an app or would the image be pushed to default internal openshift registry?

ShrayRastogi · 2021-05-19T14:52:47Z

@praveenkumar @guillaumerose - I have tried the following based on your suggestion with docker -

docker login -u kubeadmin -p $(oc whoami -t) host.docker.internal
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://host.docker.internal/v2/: x509: certificate is valid for *.apps-crc.testing, not host.docker.internal
If I create a registry container as suggested here - https://docs.docker.com/registry/deploying/#get-a-certificate
docker run -d --restart=always --name registry -v C:/Projects/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/crc.crt -e REGISTRY_HTTP_TLS_KEY=/certs/crc.key -p 4430:443 registry:2
I created the certificate for host.docker.internal, but I have to run the container on different host port other than 443 as it complains that something is already running on 443 (probably default openshift registry).
I am able to login and push the images like -
docker login -u kubeadmin -p $(oc whoami -t) host.docker.internal:4430
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

docker tag myimage:v1 host.docker.internal:4430/myproject/myimage:v1
docker push host.docker.internal:4430/myproject/myimage:v1

But I don't see the images in Openshift internal image registry

Could you please help with this, if possible? How could I solve this docker login using host.docker.internal on 443 port?

ShrayRastogi · 2021-05-21T15:21:25Z

@guillaumerose @praveenkumar - could you help with this?

praveenkumar · 2021-05-27T06:48:05Z

@ShrayRastogi what is the output of oc get is after you push the image to openshift internal registry?

ShrayRastogi · 2021-05-27T12:49:07Z

@ShrayRastogi what is the output of oc get is after you push the image to openshift internal registry?

@praveenkumar - I am not able to login to openshift internal registry itself using docker. I tried multiple things but to no avail.
For method 2 using docker container as registry, I don't think that would route internal to openshift so.... thats why I dont see images there.

If you could, then help me with docker login issue on Windows Enterprise... else you can close the bug...

gbraad · 2021-05-31T05:40:15Z

docker login -u kubeadmin -p $(oc whoami -t) host.docker.internal
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://host.docker.internal/v2/: x509: certificate is valid for *.apps-crc.testing, not host.docker.internal

the certificate does not work for this name; SNI will deny this... please only try to use something using .apps-crc.testing. We might be able to add ANOTHER hostname to the cluster, but instead we might allow crc.testing.

guillaumerose · 2021-05-31T08:36:05Z

@ShrayRastogi I found a solution for you:

Add default-route-openshift-image-registry.apps-crc.testing in the ìnsecure-registries` list in Docker Desktop configuration
Run docker run -it --rm --privileged --pid=host dockerpinata/nsenter-dockerd
Inside this shell, run echo "192.168.65.2 default-route-openshift-image-registry.apps-crc.testing" >> /etc/hosts. 192.168.65.2 is the result of nslookup host.docker.internal.

You will be able to login with docker login -u kubeadmin -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing

Each time you restart Docker you will have to run step 2 and 3 again.

I also tried on macOS and it works fine by default. Directly docker login.

ShrayRastogi · 2021-05-31T10:22:54Z

docker login -u kubeadmin -p $(oc whoami -t) host.docker.internal
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://host.docker.internal/v2/: x509: certificate is valid for *.apps-crc.testing, not host.docker.internal
the certificate does not work for this name; SNI will deny this... please only try to use something using .apps-crc.testing. We might be able to add ANOTHER hostname to the cluster, but instead we might allow crc.testing.

@gbraad - Thank you for input, however, the default route for internal registry goes to http instead of https when running crc with vsock. That's why need to use host.docker.internal which goes to https but that causes an issue with certificate. I tried creating multiple certificates that has subject with alt names for both host.docker.internal and default route but nothing worked.

ShrayRastogi · 2021-05-31T10:24:25Z

@ShrayRastogi I found a solution for you:

Add default-route-openshift-image-registry.apps-crc.testing in the ìnsecure-registries` list in Docker Desktop configuration

Run docker run -it --rm --privileged --pid=host dockerpinata/nsenter-dockerd

Inside this shell, run echo "192.168.65.2 default-route-openshift-image-registry.apps-crc.testing" >> /etc/hosts. 192.168.65.2 is the result of nslookup host.docker.internal.

You will be able to login with docker login -u kubeadmin -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing

Each time you restart Docker you will have to run step 2 and 3 again.

I also tried on macOS and it works fine by default. Directly docker login.

@guillaumerose - Thanks for providing the way. I will try this out and let you know if it's works.

ShrayRastogi · 2021-05-31T12:33:34Z

docker login -u kubeadmin -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing

@guillaumerose - IT WORKED!!!!!!!!!!!!!!!!!!!!!!!!!!!! THANK YOU SO MUCH.........................!!!!!!!!!!!!!!!!!!!!!!!!!!! Able to now docker login and push images to internal registry.

Could you please explain a bit more, if that's ok, as to what this docker container is doing and what made it work?

gbraad · 2021-05-31T13:30:31Z

@ShrayRastogi
The nsenter binary allows you to enter the namespace of another process, which i how containers work. It runs as privileged and allows to write to the ~~host's~~ Docker container-runtime image host-lookup file and adds an entry to 192.... This is why you need to re-run this, as these settings aren't permanent... however, we could make that in the SNC image generation; @guillaumerose, right?

Oh wait, that is on the docker's runtime image of course... So, this is something specific to Docker for Windows. BUT, if we add this entry to the host itself, would this get taken?

ShrayRastogi · 2021-05-31T14:06:32Z

@ShrayRastogi
The nsenter binary allows you to enter the namespace of another process, which i how containers work. It runs as privileged and allows to write to the ~~host's~~ Docker container-runtime image host-lookup file and adds an entry to 192.... This is why you need to re-run this, as these settings aren't permanent... however, we could make that in the SNC image generation; @guillaumerose, right?

Oh wait, that is on the docker's runtime image of course... So, this is something specific to Docker for Windows. BUT, if we add this entry to the host itself, would this get taken?

@gbraad @guillaumerose - Thanks for the explanation. If CRC would be able to do it by default, it would help a lot or at least a documentation of this process to help Windows Enterprise users.

I think we can close the bug now.
Again, thank you for all the help, really appreciate it.

guillaumerose · 2021-05-31T14:11:05Z

On macOS, dockerd is configured to use the built-in http proxy of VPNKit. That means that all HTTP requests made by dockerd are in reality, made on the host: DNS resolution, connection etc.

On Windows, there is no http proxy configured for dockerd, so: the DNS resolution happens and VPNKit answers 127.0.0.1 for default-route-openshift-image-registry.apps-crc.testing.
Then, dockerd tries to connect to this. It is localhost, it can't.

gbraad · 2021-05-31T14:37:04Z

On Windows, there is no http proxy configured for dockerd

Which means this issue is due to missing functionality from DfW; there is no feature parity. Perhaps an oversight (forgotten use-case as it is not often needed)

rprakashg · 2021-12-16T01:43:08Z

I'm having the same error with PODMAN on MacOS with latest version of CRC, login worked built image with podman but push to internal registry with CRC failed.

cfergeau · 2021-12-16T07:57:55Z

@rprakashg can you open a new issue, and provide the details listed in the issue template? Thanks!

ShrayRastogi added the kind/bug Something isn't working label May 19, 2021

praveenkumar added the os/windows label May 19, 2021

adrianriobo added this to To do in Quality Engineering May 31, 2021

guillaumerose closed this as completed May 31, 2021

ayates83 mentioned this issue Jul 8, 2022

[BUG] podman copy to internal crc registry fails connection refused #3246

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CRC Image Registry docker login issue on Windows - 127.0.0.1:80 connection refused #2354

CRC Image Registry docker login issue on Windows - 127.0.0.1:80 connection refused #2354

ShrayRastogi commented May 19, 2021 •

edited by praveenkumar

praveenkumar commented May 19, 2021

guillaumerose commented May 19, 2021

ShrayRastogi commented May 19, 2021

ShrayRastogi commented May 19, 2021

ShrayRastogi commented May 19, 2021 •

edited

praveenkumar commented May 19, 2021

ShrayRastogi commented May 19, 2021 •

edited

guillaumerose commented May 19, 2021

ShrayRastogi commented May 19, 2021

guillaumerose commented May 19, 2021

ShrayRastogi commented May 19, 2021

guillaumerose commented May 19, 2021

ShrayRastogi commented May 19, 2021

ShrayRastogi commented May 19, 2021

ShrayRastogi commented May 21, 2021

praveenkumar commented May 27, 2021

ShrayRastogi commented May 27, 2021

gbraad commented May 31, 2021 •

edited

guillaumerose commented May 31, 2021

ShrayRastogi commented May 31, 2021

ShrayRastogi commented May 31, 2021

ShrayRastogi commented May 31, 2021

gbraad commented May 31, 2021 •

edited

ShrayRastogi commented May 31, 2021

guillaumerose commented May 31, 2021

gbraad commented May 31, 2021

rprakashg commented Dec 16, 2021

cfergeau commented Dec 16, 2021

CRC Image Registry docker login issue on Windows - 127.0.0.1:80 connection refused #2354

CRC Image Registry docker login issue on Windows - 127.0.0.1:80 connection refused #2354

Comments

ShrayRastogi commented May 19, 2021 • edited by praveenkumar

praveenkumar commented May 19, 2021

guillaumerose commented May 19, 2021

ShrayRastogi commented May 19, 2021

ShrayRastogi commented May 19, 2021

ShrayRastogi commented May 19, 2021 • edited

praveenkumar commented May 19, 2021

ShrayRastogi commented May 19, 2021 • edited

guillaumerose commented May 19, 2021

ShrayRastogi commented May 19, 2021

guillaumerose commented May 19, 2021

ShrayRastogi commented May 19, 2021

guillaumerose commented May 19, 2021

ShrayRastogi commented May 19, 2021

ShrayRastogi commented May 19, 2021

ShrayRastogi commented May 21, 2021

praveenkumar commented May 27, 2021

ShrayRastogi commented May 27, 2021

gbraad commented May 31, 2021 • edited

guillaumerose commented May 31, 2021

ShrayRastogi commented May 31, 2021

ShrayRastogi commented May 31, 2021

ShrayRastogi commented May 31, 2021

gbraad commented May 31, 2021 • edited

ShrayRastogi commented May 31, 2021

guillaumerose commented May 31, 2021

gbraad commented May 31, 2021

rprakashg commented Dec 16, 2021

cfergeau commented Dec 16, 2021

ShrayRastogi commented May 19, 2021 •

edited by praveenkumar

ShrayRastogi commented May 19, 2021 •

edited

ShrayRastogi commented May 19, 2021 •

edited

gbraad commented May 31, 2021 •

edited

gbraad commented May 31, 2021 •

edited