Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument #3860

Closed
hect1c opened this issue Mar 12, 2019 · 11 comments

Comments

@hect1c
Copy link

@hect1c hect1c commented Mar 12, 2019

Hey,

I'm trying to do a quick-start local setup on my MacOS Machine:

Run the following command

minikube start --vm-driver=hyperkit -v=9

And I get the following output / debug

`minikube v0.35.0 on darwin (amd64)
馃敟 Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
...
...
About to run SSH command:
sudo mkdir -p /etc/sysconfig && printf %s "
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '
" | sudo tee /etc/sysconfig/crio.minikube
SSH cmd err, output: :
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '

...
馃摱 "minikube" IP address is 192.168.64.6
馃惓 Configuring Docker as the container runtime ...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:62493
() Calling .GetVersion
Using API Version 1
() Calling .SetConfigRaw
() Calling .GetMachineName
Preparing Kubernetes environment ...
(minikube) Calling .GetURL
馃殰 Pulling images required by Kubernetes v1.13.4 ...
Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.4": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
: Process exited with status 1
馃殌 Launching Kubernetes v1.13.4 using kubeadm ...
馃挘 Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.4
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.4: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

馃樋 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
馃憠 https://github.com/kubernetes/minikube/issues/new`

Again my OS is MacOS High Sierra Version 10.13.6

I am able to pull the images using docker pull so don't think it's a network issue, however it seems unable to download or access the kubeadm, which I think is something I may need to install separately but having found any guides or documentation on this. I may be missing something but greatly appreciate your assistance on this ahead of time.

@huguesalary

This comment has been minimized.

Copy link
Contributor

@huguesalary huguesalary commented Mar 12, 2019

Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443

This is quite a weird IP. What does nslookup k8s.gcr.io return? It seems to me you have an internal DNS issue.

@hect1c

This comment has been minimized.

Copy link
Author

@hect1c hect1c commented Mar 13, 2019

This is my response

Server: 10.128.34.91
Address: 10.128.34.91#53

Non-authoritative answer:
Name: k8s.gcr.io
Address: 0.0.44.248

Please note: I am in China where google is blocked, but I do have my VPN on.

@huguesalary

This comment has been minimized.

Copy link
Contributor

@huguesalary huguesalary commented Mar 13, 2019

That鈥檚 what I thought, the issue is with the DNS returning an invalid IP. I鈥檇 have thought that through a VPN you鈥檇 get the correct ip.

Not sure what your options are since you are in China. For me the k8s.gcr.io resolves to 74.125.20.82.

@tstromberg tstromberg changed the title minikube start --vm-driver=hyperkit fails on pulling kube images China: failed to pull image... Get https://k8s.gcr.io/v2/: dial tcp 0.0.42.22:443: connect: invalid argument Mar 13, 2019
@tstromberg

This comment has been minimized.

Copy link
Contributor

@tstromberg tstromberg commented Mar 13, 2019

Your DNS is responding with an address that apparently is not available. You can try this to possibly work-around the issue by fetching the images outside of the VM using:

minikube start --cache-images

Or by passing information to Docker so that it can use an HTTP proxy: https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md

Let me know what you find out! Once #3714 is merged, you'll be able to specify --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

@daniel-farina

This comment has been minimized.

Copy link

@daniel-farina daniel-farina commented Mar 17, 2019

Deleting the local config fixed this issue for me:

rm -rf ~/.minikube

I run it again and all errors were gone.

@Kingonion

This comment has been minimized.

Copy link

@Kingonion Kingonion commented Apr 2, 2019

In china, because of the GFW, you can't pull images from k8s.gcr.io without a VPN. But you can pull the needed images from registry.cn-hangzhou.aliyuncs.com/google_containers and tag them. If the needed images exist on the local system, the starting process will go on even with pulling error. I wrote a shell script that can help you install minikube on Linux OS easily. And you can also make a offline installation package by using the script. The project home is https://github.com/Kingonion/minikube-on-linux

@gbraad

This comment has been minimized.

Copy link
Contributor

@gbraad gbraad commented Apr 2, 2019

@maxnilz

This comment has been minimized.

Copy link

@maxnilz maxnilz commented Apr 17, 2019

This is my response

Server: 10.128.34.91
Address: 10.128.34.91#53

Non-authoritative answer:
Name: k8s.gcr.io
Address: 0.0.44.248

Please note: I am in China where google is blocked, but I do have my VPN on.

Hi, I have the same problem with you, after some digging, I found the docker pull request will first go to docker daemon via REST endpoint /images/create, and the docker daemon will take over the pulling work. In order to make docker daemon to use the proxy, we need to set http_proxy & https_proxy to the place where dockerd running, i.e, root user.

Here is my scenario:
OS: I'm using ubuntu18.04 desktop
Where the docker daemon is running: the docker daemon run in root user
What supervisor host the docker daemon: systemd
Do I set the https_proxy & http_proxy: yes, I set them for the normal user, but didn't set them for root user.

How I solve this:
Because I just want to set http_proxy & https_proxy for docker daemon(dockerd) only for root, I set the proxy info into systemd service unit, like this(for my computer, the config file is located here: /lib/systemd/system/docker.service:

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
Environment="http_proxy=http://127.0.0.1:7777"
Environment="https_proxy=http://127.0.0.1:7777"
ExecStart=/usr/bin/dockerd -D -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

Hope this will help somebody who got the same error like me.

@singleye

This comment has been minimized.

Copy link

@singleye singleye commented Apr 24, 2019

I had the same issue on 'MacOS 10.14.4 Majave'

Pull and tag images doesn't work for me

$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.1
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.0
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1

$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.0 k8s.gcr.io/kube-apiserver:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.0 k8s.gcr.io/kube-controller-manager:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.0 k8s.gcr.io/kube-scheduler:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.0 k8s.gcr.io/kube-proxy:v1.14.0
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
$ docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

$ docker images
REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                                                         v1.14.0             5cd54e388aba        4 weeks ago         82.1MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.14.0             5cd54e388aba        4 weeks ago         82.1MB
k8s.gcr.io/kube-controller-manager                                            v1.14.0             b95b1efa0436        4 weeks ago         158MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.14.0             b95b1efa0436        4 weeks ago         158MB
k8s.gcr.io/kube-scheduler                                                     v1.14.0             00638a24688b        4 weeks ago         81.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.14.0             00638a24688b        4 weeks ago         81.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.14.0             ecf910f40d6e        4 weeks ago         210MB
k8s.gcr.io/kube-apiserver                                                     v1.14.0             ecf910f40d6e        4 weeks ago         210MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.3.1               eb516548c180        3 months ago        40.3MB
k8s.gcr.io/coredns                                                            1.3.1               eb516548c180        3 months ago        40.3MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.3.10              2c4adeb21b4f        4 months ago        258MB
k8s.gcr.io/etcd                                                               3.3.10              2c4adeb21b4f        4 months ago        258MB
k8s.gcr.io/pause                                                              3.1                 da86e6ba6ca1        16 months ago       742kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        16 months ago       742kB

$ minikube start
馃槃  minikube v1.0.0 on darwin (amd64)
馃す  Downloading Kubernetes v1.14.0 images in the background ...
馃挕  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
馃攧  Restarting existing hyperkit VM for "minikube" ...
鈱  Waiting for SSH access ...r     y    d         r          r   s           7                   1                   4 m      a          2
馃摱  "minikube" IP address is 192.168.64.2
馃惓  Configuring Docker as the container runtime ...
馃惓  Version of container runtime is 18.06.2-ce
鈱  Waiting for image downloads to compe_te ...                               3                   d                   1  m      a         7
鉁  Preparing Kubernetes environment ...                                      9                   1                   1  m      a         2
馃殰  Pulling images required by Kubernetes v1.14.0 ...
鉂  Unable to pull images, which may be OK: runnigz cmd: sudo kubeadm cgnfig images pull --config /var/lib/kubeadm.yaml: command faierdv sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting heade
rs)
, error: exit status 1
: Process exited with status 1
馃攧  Relaunching Kubernetes v1.14.0 using kubeadm ...
鈱  Waiting for pods: apiserver proxy etcd scheduler controller dns           v                   5                   4 w     a           8
馃摨  Updating kube-proxy configuration ...
馃  Verifying component health ......
馃挆  kubectl is now configured to use "minikube"
馃弰  Done! Thank you for using minikube!

But specify '--image-repository' works for me

$ minikube start --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

$ minikube start --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
馃槃  minikube v1.0.0 on darwin (amd64)
馃す  Downloading Kubernetes v1.14.0 images in the background ...
馃挕  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
馃攧  Restarting existing hyperkit VM for "minikube" ...
鈱  Waiting for SSH access ...          "                        6 w     a           E      (     6 w     a                                         f
馃摱  "minikube" IP address is 192.168.64.2
馃惓  Configuring Docker as the container runtime ...
馃惓  Version of container runtime is 18.06.2-ce
鈱  Waiting for image downloads to complete ...                  6 w     a           E      (   6 w     a                                           v
鉁  Preparing Kubernetes 4nvironment ..."                        6 w     a           E      (   6 w     a                                           h
馃殰  Pulling images required by Kubernetes v1.14.0 ...
馃攧  Relaunching Kubernetes v1.14.0 using kubeadm ...
鈱  Waiting for pods: apiserver proxy etcd skhrduler controller dnsw     a           E      (   6 w     a                                           g
馃摨  Updating kube-proxy configuration ...
馃  Verifying component health .....
馃挆  kubectl is now configured to use "minikube"
馃弰  Done! Thank you for using minikube!

BTW, I'm using minikube v1.0.0

$ minikube version
minikube version: v1.0.0
@tstromberg

This comment has been minimized.

Copy link
Contributor

@tstromberg tstromberg commented May 14, 2019

v1.0.1 now includes a user-friendly message to resolve the original requesters issue. Thank you for opening the bug report!

@BingerCoder

This comment has been minimized.

Copy link

@BingerCoder BingerCoder commented Jul 31, 2019

rm -rf ~/.minikube
and with --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
works for me ~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can鈥檛 perform that action at this time.