Ingress插件下载镜像失败

[mengzyou]

重现问题所需的命令：

minikube addons enable ingress

失败的命令的完整输出：

❯ minikube addons enable ingress
🔎 Verifying ingress addon...

❌ Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

😿 If the above advice does not help, please let us know:
👉 https://github.com/kubernetes/minikube/issues/new/choose

minikube logs命令的输出：

Nov 11 06:51:04 minikube dockerd[379]: time="2020-11-11T06:51:04.777543128Z" level=info msg="Attempting next endpoint for pull after error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 06:51:04 minikube dockerd[379]: time="2020-11-11T06:51:04.777588759Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 06:52:47 minikube dockerd[379]: time="2020-11-11T06:52:47.802932242Z" level=warning msg="Error getting v2 registry: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 06:52:47 minikube dockerd[379]: time="2020-11-11T06:52:47.802998904Z" level=info msg="Attempting next endpoint for pull after error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 06:52:47 minikube dockerd[379]: time="2020-11-11T06:52:47.803057218Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 06:55:47 minikube dockerd[379]: time="2020-11-11T06:55:47.754045281Z" level=warning msg="Error getting v2 registry: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 06:55:47 minikube dockerd[379]: time="2020-11-11T06:55:47.754103119Z" level=info msg="Attempting next endpoint for pull after error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 06:55:47 minikube dockerd[379]: time="2020-11-11T06:55:47.754155489Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 07:01:12 minikube dockerd[379]: time="2020-11-11T07:01:12.750508087Z" level=warning msg="Error getting v2 registry: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 07:01:12 minikube dockerd[379]: time="2020-11-11T07:01:12.750555517Z" level=info msg="Attempting next endpoint for pull after error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 07:01:12 minikube dockerd[379]: time="2020-11-11T07:01:12.750610549Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 07:06:31 minikube dockerd[379]: time="2020-11-11T07:06:31.758504040Z" level=warning msg="Error getting v2 registry: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 07:06:31 minikube dockerd[379]: time="2020-11-11T07:06:31.758548406Z" level=info msg="Attempting next endpoint for pull after error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 11 07:06:31 minikube dockerd[379]: time="2020-11-11T07:06:31.758587573Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

使用的操作系统版本：

❯ cat /etc/os-release 
NAME="openSUSE Leap"
VERSION="15.2"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.2"
PRETTY_NAME="openSUSE Leap 15.2"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.2"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"

补充信息：

Pod Events

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  21m                 default-scheduler  Successfully assigned kube-system/ingress-nginx-controller-799c9469f7-qtfq8 to minikube
  Normal   Pulling    19m (x4 over 21m)   kubelet            Pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
  Warning  Failed     19m (x4 over 21m)   kubelet            Failed to pull image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20": rpc error: code = Unknown desc = Error response from daemon: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     19m (x4 over 21m)   kubelet            Error: ErrImagePull
  Warning  Failed     18m (x6 over 21m)   kubelet            Error: ImagePullBackOff
  Normal   BackOff    93s (x77 over 21m)  kubelet            Back-off pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"

我知道是由于我的网络限制问题，导致无法拉取镜像 us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1 ，但是我看了Pod的镜像拉取策略为 imagePullPolicy: IfNotPresent ，因此我是先将该镜像放入了 minikube 里，在启动 ingress-nginx-controller Pod的时候，应该优先使用本地镜像，可是为什么还是会去拉取镜像，导致失败？

❯ docker image ls
REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.19.2             d373dd5a8593        7 weeks ago         118MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.19.2             607331163122        7 weeks ago         119MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.19.2             8603821e1a7a        7 weeks ago         111MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.19.2             2f32d66b884f        7 weeks ago         45.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner       v3                  bad58561c4be        2 months ago        29.7MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.4.13-0            0369cf4303ff        2 months ago        253MB
us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller                         v0.34.1             6fb0739a741f        3 months ago        329MB
jettech/kube-webhook-certgen                                                  v1.2.2              5693ebf5622a        4 months ago        49MB
registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard                 v2.0.3              503bc4b7440b        4 months ago        225MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.7.0               bfe3a36ebd25        4 months ago        45.2MB
alpine                                                                        3.12                a24bb4013296        5 months ago        5.57MB
registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper           v1.0.4              86262685d9ab        7 months ago        36.9MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.2                 80d28bedfe5d        9 months ago        683kB

[jjsty1e]

@mengzyou 请问您解决此问题了吗？

[mengzyou]

@Jaggle 还没有

[jjsty1e]

@mengzyou 我已经解决了此问题，是由于墙的原因，你可以找到kube-system命名空间下的有一个失败的pod:

kubectl -n kube-system get pods

# 查看这个pod
kubectl -n kube-system describe pods <podname>

这里会提示某一个image pull失败，你可以去docker找到这个image,pull下来，然后给这个image重新打一个他们的tag，大概是us.gcr.io/xx/xxx啥的，

删掉pod后让pod重新生成即可。

提示：minikube貌似在macos下的docker运行时不支持ingress，我现在已经用virtualbox搭虚拟机整了(这样还是会遇到墙的问题)。。。。

（写的比较潦草，如果你需要详细帮助，可以找到我的联系方式一起交流交流）

[mengzyou]

@Jaggle 你的这个方法，我已经试过了，你看看我上面的描述，我已经手动下载了 us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller 镜像到minikube的环境里，但是在重新启动pod的时候，还是会去尝试pull

  Normal   Pulling           28s   kubelet            Pulling image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20"
  Warning  Failed            13s   kubelet            Failed to pull image "us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20": rpc error: code = Unknown desc = Error response from daemon: Get https://us.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed            13s   kubelet            Error: ErrImagePull

很奇怪，似乎 imagePullPolicy: IfNotPresent 不起作用

[mengzyou]

@Jaggle 我知道了，原来是deployment的image值，带了后面的hash值，我 kubectl edit 把hash值去掉就可以了，谢谢！

[Bytesu]

@Jaggle 我知道了，原来是deployment的image值，带了后面的hash值，我 kubectl edit 把hash值去掉就可以了，谢谢！

怎么去掉这个hash值，minikube ssh 编辑"/etc/kubernetes/addons/ingress-db.yaml"?

[jjsty1e]

@Bytesu kubectl edit deployment <deployment-name>

[mike-just]

我解决了这个问题，原因应该是需要重新启动minikube，我用的mac,使用的docker虚拟机是hyperkit,网络用的代理

1. 关闭minikube的所有状态

2. 重新下载 brew reinstall hyperkit
3. 启动minikube minikube start --vm-driver=hyperkit
4. 查看minkube的状态

5. minikube addons enable ingress

[autoool]

@make-just 你需要配置vpn 代理吗？

[autoool]

@Jaggle 我知道了，原来是deployment的image值，带了后面的hash值，我 kubectl edit 把hash值去掉就可以了，谢谢！

怎么去掉这个hash值，minikube ssh 编辑"/etc/kubernetes/addons/ingress-db.yaml"?

我也在考虑编辑yaml 修改镜像，但是我linux 下找不到这个文件