Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Images cannot be pulled after starting minishift, connection refused #109

Closed
donovanmuller opened this Issue Oct 3, 2016 · 7 comments

Comments

Projects
None yet
4 participants
@donovanmuller
Copy link

commented Oct 3, 2016

Installed Minishift 0.7.1 with brew cask install minishift.
minishift start starts successfully.

$ minishift status
Running

However, the registry image cannot be pulled:

 $ oc get pod
NAME                       READY     STATUS              RESTARTS   AGE
docker-registry-1-deploy   0/1       ContainerCreating   0          29s
$ oc get events --watch
LASTSEEN                    FIRSTSEEN                   COUNT     NAME                       KIND               SUBOBJECT   TYPE      REASON                    SOURCE                           MESSAGE
2016-10-03T06:41:03+02:00   2016-10-03T06:41:03+02:00   1         minishift                  Node                           Normal    Starting                  {kube-proxy minishift}           Starting kube-proxy.
2016-10-03T06:41:04+02:00   2016-10-03T06:41:04+02:00   1         minishift                  Node                           Normal    Starting                  {kubelet minishift}              Starting kubelet.
2016-10-03T06:41:04+02:00   2016-10-03T06:41:04+02:00   1         minishift                  Node                           Normal    NodeHasSufficientDisk     {kubelet minishift}              Node minishift status is now: NodeHasSufficientDisk
2016-10-03T06:41:04+02:00   2016-10-03T06:41:04+02:00   1         minishift                  Node                           Normal    NodeHasSufficientMemory   {kubelet minishift}              Node minishift status is now: NodeHasSufficientMemory
2016-10-03T06:41:07+02:00   2016-10-03T06:41:07+02:00   1         minishift                  Node                           Normal    RegisteredNode            {controllermanager }             Node minishift event: Registered Node minishift in NodeController
2016-10-03T06:41:09+02:00   2016-10-03T06:41:09+02:00   1         docker-registry            DeploymentConfig               Normal    DeploymentCreated         {deploymentconfig-controller }   Created new deployment "docker-registry-1" for version 1
2016-10-03T06:41:09+02:00   2016-10-03T06:41:09+02:00   1         docker-registry-1-deploy   Pod                            Normal    Scheduled                 {default-scheduler }             Successfully assigned docker-registry-1-deploy to minishift
2016-10-03T06:41:10+02:00   2016-10-03T06:41:10+02:00   1         docker-registry-1-deploy   Pod                            Warning   FailedSync                {kubelet minishift}              Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "Error while pulling image: Get https://index.docker.io/v1/repositories/openshift/origin-pod/images: dial tcp: lookup index.docker.io on 192.168.64.1:53: read udp 192.168.64.5:34706->192.168.64.1:53: read: connection refused"

2016-10-03T06:41:22+02:00   2016-10-03T06:41:22+02:00   1         docker-registry-1-deploy   Pod                 Warning   FailedSync   {kubelet minishift}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "Error while pulling image: Get https://index.docker.io/v1/repositories/openshift/origin-pod/images: dial tcp: lookup index.docker.io on 192.168.64.1:53: read udp 192.168.64.5:46653->192.168.64.1:53: read: connection refused"

2016-10-03T06:41:37+02:00   2016-10-03T06:41:37+02:00   1         docker-registry-1-deploy   Pod                 Warning   FailedSync   {kubelet minishift}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with ImagePullBackOff: "Back-off pulling image \"openshift/origin-pod:v1.3.0\""

LASTSEEN                    FIRSTSEEN                   COUNT     NAME                       KIND      SUBOBJECT   TYPE      REASON       SOURCE                MESSAGE
2016-10-03T06:41:49+02:00   2016-10-03T06:41:49+02:00   1         docker-registry-1-deploy   Pod                   Warning   FailedSync   {kubelet minishift}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "Error while pulling image: Get https://index.docker.io/v1/repositories/openshift/origin-pod/images: dial tcp: lookup index.docker.io on 192.168.64.1:53: read udp 192.168.64.5:34395->192.168.64.1:53: read: connection refused"

This line looks suspicious:

... lookup index.docker.io on 192.168.64.1:53 ...

and I'm assuming it's trying to hit the default DNS server on 192.168.64.1, which doesn't seem valid on my machine, hence the connection refused.

ifconfig gives me this:

...
bridge100: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    options=3<RXCSUM,TXCSUM>
    ether a6:5e:60:1d:57:64
    inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
    Configuration:
        id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
        maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
        root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
        ipfilter disabled flags 0x2
    member: en7 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 22 priority 0 path cost 0
    nd6 options=1<PERFORMNUD>
    media: autoselect
    status: active

and if I try connect to 192.168.64.1:53:

$ ping 192.168.64.1
PING 192.168.64.1 (192.168.64.1): 56 data bytes
64 bytes from 192.168.64.1: icmp_seq=0 ttl=64 time=0.174 ms
64 bytes from 192.168.64.1: icmp_seq=1 ttl=64 time=0.092 ms
64 bytes from 192.168.64.1: icmp_seq=2 ttl=64 time=0.097 ms
^C
--- 192.168.64.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.092/0.121/0.174/0.038 ms

$ 192.168.64.1 53
Trying 192.168.64.1...
telnet: connect to address 192.168.64.1: Connection refused
telnet: Unable to connect to remote host

I get the same.

Not sure where to go from here.
Is this an issue on my setup or how/where should the DNS be pointing too?

Thanks

@dmhumph

This comment has been minimized.

Copy link

commented Oct 6, 2016

I am having the same issue. Attached is a screenshot from my console.

screen shot 2016-10-06 at 4 11 56 pm
Thanks

@jimmidyson

This comment has been minimized.

Copy link
Member

commented Oct 6, 2016

@davidaparicio

This comment has been minimized.

Copy link

commented Oct 12, 2016

I have the same issue

oc get events --watch

2016-10-12T16:03:02+02:00   2016-10-12T15:20:50+02:00   13        aggregation-http-d-vuxbb   Pod       spec.containers{aggregation-http-d}   Normal    Pulling      {kubelet minishift}   pulling image "vertx-microservices-example/aggregation-http-d:1.1-SNAPSHOT"
2016-10-12T16:03:05+02:00   2016-10-12T15:20:53+02:00   13        aggregation-http-d-vuxbb   Pod       spec.containers{aggregation-http-d}   Warning   Failed       {kubelet minishift}   Failed to pull image "vertx-microservices-example/aggregation-http-d:1.1-SNAPSHOT": Error: image vertx-microservices-example/aggregation-http-d not found
2016-10-12T16:03:05+02:00   2016-10-12T15:20:53+02:00   13        aggregation-http-d-vuxbb   Pod                                             Warning   FailedSync   {kubelet minishift}   Error syncing pod, skipping: failed to "StartContainer" for "aggregation-http-d" with ErrImagePull: "Error: image vertx-microservices-example/aggregation-http-d not found"

2016-10-12T16:05:05+02:00   2016-10-12T15:20:03+02:00   191       aggregation-http-b-32tbu   Pod       spec.containers{aggregation-http-b}   Normal    BackOff      {kubelet minishift}   Back-off pulling image "vertx-microservices-example/aggregation-http-b:1.1-SNAPSHOT"
2016-10-12T16:05:05+02:00   2016-10-12T15:20:03+02:00   191       aggregation-http-b-32tbu   Pod                                             Warning   FailedSync   {kubelet minishift}   Error syncing pod, skipping: failed to "StartContainer" for "aggregation-http-b" with ImagePullBackOff: "Back-off pulling image \"vertx-microservices-example/aggregation-http-b:1.1-SNAPSHOT\""

docker version

Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 23:54:00 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 23:54:00 2016
 OS/Arch:      linux/amd64

minishift version
minishift version: 0.7.1

docker info

Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
Images: 42
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 84
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.16-boot2docker
Operating System: Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.858 GiB
Name: default
ID: UONI:BPTI:4W5L:E6Y2:PC4N:KD5P:WHFF:RZMC:7UXY:YA5D:TXBH:5SK4
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 13
 Goroutines: 24
 System Time: 2016-10-12T14:01:21.795464834Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8

docker-machine version
docker-machine version 0.8.0, build b85aac1

@donovanmuller

This comment has been minimized.

Copy link
Author

commented Oct 16, 2016

@davidaparicio I don't believe this is the same issue.
Your issue seems to be that the image you are trying to pull does not exist: Error: image vertx-microservices-example/aggregation-http-d *not found*.

The issue @dmhumph1547 and I are facing is due to this "Error while pulling image: Get https://index.docker.io/v1/repositories/openshift/origin-pod/images: dial tcp: lookup index.docker.io on 192.168.64.1:53: read udp 192.168.64.5:34395->192.168.64.1:53: read: connection refused".

@donovanmuller

This comment has been minimized.

Copy link
Author

commented Oct 16, 2016

@jimmidyson Yes, replacing the existing value with nameserver 8.8.8.8 does the trick. However, annoyingly, you have to change this every time after a minishift start. 😞

Correct me if I'm wrong but this seems to be a Docker for Mac issue? https://forums.docker.com/t/docker-pull-not-using-correct-dns-server-when-private-registry-on-vpn/11117/28 and hence why not everyone is seeing this issue?

@dmhumph1547 are you using Docker for Mac?

@dmhumph

This comment has been minimized.

Copy link

commented Oct 18, 2016

@donovanmuller Yes, I am using Docker on Mac as well.

@donovanmuller

This comment has been minimized.

Copy link
Author

commented Oct 19, 2016

Closing this as there are two workarounds to this issue (which is not minishift's anyway):

  1. After minishift start, log into the VM with minishift ssh and change the nameserver x.x.x.x value in /etc/resolv.conf to a valid DNS server (as per my previous comment. Note you will have to do this everytime you stop/start the VM.
  2. A more permanent fix is have a local dnsmasq listening on all interfaces (listen-address=0.0.0.0), so that the default DNS value for the VM can use your dnsmasq instance as a DNS server.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.