Cant push image to local registry #196
Comments
|
Hi @lethevimlet , Could you share the report produced by Thanks |
|
Unfortunately, the results produced by microk8s.inspect yield sensitive information that I can't share since it's against our privacy policy but I can tell you more information about the setup if that helps to replicate the issue. I'm using Ubuntu Server 16.04.5 LTS Inspect shows the following error for docker: |
|
Do you think you could share a docker image I could build that would cause this error? Just to give you some context, the docker registry is set to be insecure: https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/default-args/docker-daemon.json and this is how we test the registry before every release https://github.com/ubuntu/microk8s/blob/master/tests/validators.py#L182 |
|
I am basically asking for some instructions on how to reproduce the error. |
|
Just to double check, you do not have a second dockerd installed on your system, right? |
|
On a cluster that looks like this: With a docker file like this: I saw no issues: |
|
Apparently, no registry container is running and this is why push fails regardless of the image used. I've created a separate VM with a private Docker registry added it to the insecure array, and had no issues pushing images to that. I even had luck using them with microk8s. It seems as if although I enable registry with |
|
This is strange. It is as if the API server did not apply the registry manifest (https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/registry.yaml). Can you share the logs of the API server when you Thanks |
|
Logs for the apiserver after running microk8s.enable registry: journal.log systemctl.log |
|
Actually, I just check with all namespaces and the docker-registry created by microk8s.enable registry is running: But the problem persists, I can't push any image to that registry. |
|
I'm having the same issue and seeing similar evidence on my system with log message errors (Address isn't valid, Failed to find a valid address, etc). I also see the container-registry pod running. Also running Ubuntu Server 16.04.5 LTS I tried the above "hello world" docker container and resulted in the same thing: |
same hang also on Ubuntu Server 16.04 |
|
Ok here is the reason for me localhost refers to the IPV6 version - and the command below hangs But if I comment out the line below from my /etc/hosts #::1 localhost ip6-localhost ip6-loopback it works now. |
|
@cneberg great find, I found the same thing on my end. But, I'm not sure how that works because ping, dig, etc all return the IPv4 address. I tried re-building/tag and push with |
|
Someone replied to this thread and I received the message from Gelinger Media through email from github (I'm not sure how it was sent because its not in the issue history) but I'm replying here in hopes it gets back to them. It appears in push example to push example-php-dbconnect you tag as 32 hundred as your port and not 32 thousand. Later in your next examples you push my-busybox to port 32000 and it appears to work. From: Gelinger Media We have the same problem here with custom built images microk8s.docker push localhost:3200/heptio/example-php-dbconnect In comparison to it sample pull-tag-push worked from docks root@kubernetes-2gb-nbg1-1: Any Ideas? |
|
Any update on this issue, when it is expected to be resolved? |
|
I am going to close this issue since I was not able to reproduce the initial case. I am also not sure what can be done in the case of IPv6. If anyone has a suggestion please step forward to discuss how we can move forward. @maherkamal, if you are running into trouble pushing to the registry please open a new issue describing your setup and a way to reproduce. Thank you all. |
|
I disabled IPv6 on my linux machine and everything is working fine, Thank you ktsakalozos |
|
I dont't like workaround with disabling IPv6 loopback and it seems there is better way: Notice that I used 127.0.0.1 instead of localhost. |
|
The 127.0.0.1:32000 work-around didn't work for me, because 127.0.0.1 isn't listed as an insecure registry :( |
|
rebuilt with tag 127.0.0.1 instead of localhost and everything worked fine |
|
For me I was able to push to the private registry on |
|
I see the same problem. Reproduced by snap install microk8s (1176) and snap install docker (423). The initial The local registry hangs when you try to communicate with it over IPv6, but not over IPv4: I can remove However, even after There seem to be two issues here. The first (and most important) is that the microk8s bundled registry accepts but blackholes requests on To workaround this with docker, I can tag and push images to 127.0.0.1:32000 instead of localhost:32000. However then I get problems with microk8s pulling from this registry: It gives errors saying that it gets a HTTP response when trying to talk to a HTTPS endpoint. To fix this, edit (and It is not a good user experience to have to do all this debugging to get to this point :-( Could I request that this ticket be re-opened? It is very simple to reproduce that the local registry is listening on ::1, accepts connections and hangs. If that problem were fixed, none of the workarounds would be required. |
|
Extra data point: So the problem is not with the registry container itself: it's with the k8s infrastructure which is forwarding port 32000 to container port 5000. Relates to:
The latter claims to be fixed by recent commit kubernetes/kubernetes#87699, so should eventually get picked up in a future k8s release. From reading the above: it seems that k8s binds a socket purely as a way of reserving the port number - the actual (IPv4) traffic is redirected through iptables and never touches the socket. Unfortunately, there are no IPv6 rules and so IPv6 traffic does still hit the socket :-( The commit changes it to listen on IPv4 only (for a v4 cluster anyway). So another workaround might be to add a manual ip6tables rule rejecting connections to |
|
Commenting out the IPV6 entries in my /etc/hosts file resolved the problem for me |
|
In my case registry didn't work after Commenting out the IPV6 entries in my /etc/hosts file resolved it. Thanks folks for pointing it out. |
|
This seriously needs some built-in documentation - here we are several years later and people (like me) are only getting the solution from a closed github issue? |
|
@invertigo "several years"??? O.o @ktsakalozos if you're willing to re-open this issue, i have a suggestion on how to move forward. What we did in Tilt is try to detect if someone is using a microk8s registry, and if they are, double-check if localhost resolves to an ipv4 address. If it doesn't resolve, we error out and don't let them push to the registry. Here's what it looks like: https://github.com/tilt-dev/tilt/pull/2370/files I think it would make sense to put a similar check here: https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/enable.registry.sh#L6, happy to send a PR if that makes sense |
|
@nicks sounds like a plan. Do you think we should fail the installation of the registry or throw a warning on what to do? |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
If you are still having issues despite commenting out the IPV6 entry, double check the /etc/docker/daemon.json file. I copy pasted the required lines in using vim and I think there was some character in there messing things up because it worked once I deleted any excess white spaces. |
|
still having this issue in WSL2 on windows in 2023 |
|
I had this issue. Turns out I had minikube registry running as well which was interfering with microk8s registry localhost though they were on different ports, 5000 vs 32000. Once i stopped my minikube cluster i was able to push it to microk8s registry. The following page helped me understand what to look for. |
Did you find a solution on wls? I am in the exact same situation. As a workaround for now I created a registry in docker: and pushed to that instead. and updated my image path in the deployment |
I enabled registry with
mircrok8s.enable registrythen I successfully build my image with a Dockerfile.
microk8s.docker build -t localhost:32000/test:1 .But I can't push a custom image to the local registry
microk8s.docker push localhost:32000/test:1It will stay stuck with the following message
The push refers to a repository [localhost:32000/test]Am I missing something? any ideas of what could be wrong with the registry or my setup?
The text was updated successfully, but these errors were encountered: