Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot start ingress addon: waiting for pod #10544

Closed
newtoniumx3 opened this issue Feb 20, 2021 · 53 comments
Closed

Cannot start ingress addon: waiting for pod #10544

newtoniumx3 opened this issue Feb 20, 2021 · 53 comments
Labels
addon/ingress area/addons lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. top-10-issues Top 10 support issues

Comments

@newtoniumx3
Copy link

newtoniumx3 commented Feb 20, 2021

Steps to reproduce the issue:

  1. minikube start --vm-driver hyperv --cpus 2 --memory=8192
  2. minikube addons enable ingress --alsologtostderr

Full output of failed command:

❯ minikube addons enable ingress --alsologtostderr
I0220 16:57:51.899458   28192 out.go:229] Setting OutFile to fd 88 ...
I0220 16:57:51.900398   28192 out.go:276] TERM=,COLORTERM=, which probably does not support color
I0220 16:57:51.900398   28192 out.go:242] Setting ErrFile to fd 92...
I0220 16:57:51.901397   28192 out.go:276] TERM=,COLORTERM=, which probably does not support color
I0220 16:57:51.921175   28192 addons.go:55] Setting ingress=true in profile "minikube"
I0220 16:57:51.921826   28192 addons.go:131] Setting addon ingress=true in "minikube"
I0220 16:57:51.922369   28192 host.go:66] Checking if "minikube" exists ...
I0220 16:57:51.922904   28192 main.go:119] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
I0220 16:57:52.389395   28192 main.go:119] libmachine: [stdout =====>] : Running

I0220 16:57:52.390040   28192 main.go:119] libmachine: [stderr =====>] :
I0220 16:57:52.390040   28192 addons.go:247] installing /etc/kubernetes/addons/ingress-configmap.yaml
I0220 16:57:52.390734   28192 ssh_runner.go:310] scp memory --> /etc/kubernetes/addons/ingress-configmap.yaml (1251 bytes)
I0220 16:57:52.390734   28192 main.go:119] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
I0220 16:57:52.848442   28192 main.go:119] libmachine: [stdout =====>] : Running

I0220 16:57:52.850487   28192 main.go:119] libmachine: [stderr =====>] :
I0220 16:57:52.851015   28192 main.go:119] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Hyper-V\Get-VM minikube ).networkadapters[0]).ipaddresses[0]
I0220 16:57:53.576395   28192 main.go:119] libmachine: [stdout =====>] : 172.23.0.69

I0220 16:57:53.576923   28192 main.go:119] libmachine: [stderr =====>] :
I0220 16:57:53.576923   28192 sshutil.go:48] new ssh client: &{IP:172.23.0.69 Port:22 SSHKeyPath:C:\Users\XX\.minikube\machines\minikube\id_rsa Username:docker}
I0220 16:57:53.675885   28192 addons.go:247] installing /etc/kubernetes/addons/ingress-rbac.yaml
I0220 16:57:53.675885   28192 ssh_runner.go:310] scp memory --> /etc/kubernetes/addons/ingress-rbac.yaml (4828 bytes)
I0220 16:57:53.682452   28192 addons.go:247] installing /etc/kubernetes/addons/ingress-dp.yaml
I0220 16:57:53.682452   28192 ssh_runner.go:310] scp memory --> /etc/kubernetes/addons/ingress-dp.yaml (8533 bytes)
I0220 16:57:53.717473   28192 ssh_runner.go:149] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.2/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml
I0220 16:57:53.859260   28192 addons.go:346] Verifying addon ingress=true in "minikube"
I0220 16:57:53.861262   28192 out.go:119] * Verifying ingress addon...
* Verifying ingress addon...
I0220 16:57:53.869261   28192 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "kube-system" ...
I0220 16:57:53.884052   28192 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0220 16:57:53.884052   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:54.391870   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:54.889169   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:55.400154   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:55.900645   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:56.403320   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:56.904065   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:57.400503   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:57.902159   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:58.391056   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0220 16:57:58.891532   28192 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]

Windows 10
Docker version 20.10.2, build 2291f61
minikube version: v1.17.1
commit: 043bdca

@newtoniumx3
Copy link
Author

newtoniumx3 commented Feb 20, 2021

I checked the logs kubectl get pods --namespace kube-system

and see this

ingress-nginx-admission-create-lp98w        0/1     ImagePullBackOff    0          88s
ingress-nginx-admission-patch-pbff9         0/1     ErrImagePull        0          88s
ingress-nginx-controller-558664778f-s5c9t   0/1     ContainerCreating   0          88s

I've also set NO_PROXY

❯ echo $env:NO_PROXY
localhost,127.0.0.1,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24

@ilya-zuyev ilya-zuyev added kind/support Categorizes issue or PR as a support question. co/hyperv HyperV related issues labels Feb 24, 2021
@priyawadhwa
Copy link

Hey @newtoniumx3 thanks for opening this issue. Could you provide more details around the pods so we can figure out why the image isn't being pulled?

kubectl describe po ingress-nginx-admission-create-lp98w --namespace kube-system

If you have docker running, I'm also curious if this happens with the docker driver as well? If you could try:

minikube start --driver docker -p docker
minikube addons enable ingress
minikube delete -p docker

that should let us know if it works with ingress.

@priyawadhwa priyawadhwa added the triage/needs-information Indicates an issue needs more information in order to work on it. label Feb 26, 2021
@newtoniumx3
Copy link
Author

Hey @newtoniumx3 thanks for opening this issue. Could you provide more details around the pods so we can figure out why the image isn't being pulled?

kubectl describe po ingress-nginx-admission-create-lp98w --namespace kube-system

If you have docker running, I'm also curious if this happens with the docker driver as well? If you could try:

minikube start --driver docker -p docker
minikube addons enable ingress
minikube delete -p docker

that should let us know if it works with ingress.

Hey I’m currently afraid to touch it as I finally got everything working using Kubernetes that ships with docker desktop.
I’m not sure I’m willing to risk messing with my config again as I was down over a week when this stopped working for some strange reason (because I always had it working for months prior).

I will attempt on different machine to see if I can reproduce it.

@priyawadhwa priyawadhwa removed the triage/needs-information Indicates an issue needs more information in order to work on it. label Feb 26, 2021
@priyawadhwa
Copy link

@newtoniumx3 thanks, no worries if it's too difficult to try to reproduce!

@spowelljr
Copy link
Member

@newtoniumx3 I'm going to close this issue, but if you end up being able to reproduce the issue feel free to reopen.

@natesales
Copy link

I'm seeing this same issue when enabling the ingress addon. Let me know what other information would be helpful here.

$ uname -a
Linux awx 4.19.0-14-arm64 #1 SMP Debian 4.19.171-2 (2021-01-30) aarch64 GNU/Linux
$ minikube version
minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5
$ minikube addons enable ingress -v=5 --alsologtostderr
I0411 08:10:33.701931   21466 out.go:278] Setting OutFile to fd 1 ...
I0411 08:10:33.702134   21466 out.go:330] isatty.IsTerminal(1) = true
I0411 08:10:33.702167   21466 out.go:291] Setting ErrFile to fd 2...
I0411 08:10:33.702191   21466 out.go:330] isatty.IsTerminal(2) = true
I0411 08:10:33.702383   21466 root.go:317] Updating PATH: /home/nate/.minikube/bin
W0411 08:10:33.702615   21466 root.go:292] Error reading config file at /home/nate/.minikube/config/config.json: open /home/nate/.minikube/config/config.json: no such file or directory
I0411 08:10:33.703183   21466 addons.go:55] Setting ingress=true in profile "minikube"
I0411 08:10:33.703226   21466 addons.go:131] Setting addon ingress=true in "minikube"
I0411 08:10:33.703292   21466 host.go:66] Checking if "minikube" exists ...
I0411 08:10:33.704276   21466 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0411 08:10:33.775282   21466 out.go:157]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
I0411 08:10:33.776992   21466 out.go:157]     ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
I0411 08:10:33.778497   21466 out.go:157]     ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
I0411 08:10:33.778595   21466 addons.go:261] installing /etc/kubernetes/addons/ingress-configmap.yaml
I0411 08:10:33.778638   21466 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-configmap.yaml (1865 bytes)
I0411 08:10:33.778744   21466 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0411 08:10:33.837673   21466 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49157 SSHKeyPath:/home/nate/.minikube/machines/minikube/id_rsa Username:docker}
I0411 08:10:33.957576   21466 addons.go:261] installing /etc/kubernetes/addons/ingress-rbac.yaml
I0411 08:10:33.957619   21466 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-rbac.yaml (6005 bytes)
I0411 08:10:33.977878   21466 addons.go:261] installing /etc/kubernetes/addons/ingress-dp.yaml
I0411 08:10:33.977918   21466 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-dp.yaml (9125 bytes)
I0411 08:10:33.998092   21466 ssh_runner.go:149] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.2/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml
I0411 08:10:34.351846   21466 addons.go:299] Verifying addon ingress=true in "minikube"
I0411 08:10:34.354113   21466 out.go:157] 🔎  Verifying ingress addon...
🔎  Verifying ingress addon...
I0411 08:10:34.357070   21466 kapi.go:59] client config for minikube: &rest.Config{Host:"https://192.168.49.2:8443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/home/nate/.minikube/profiles/minikube/client.crt", KeyFile:"/home/nate/.minikube/profiles/minikube/client.key", CAFile:"/home/nate/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0x132efd0), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}
I0411 08:10:34.357853   21466 cert_rotation.go:137] Starting client certificate rotation controller
I0411 08:10:34.361097   21466 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0411 08:10:34.387895   21466 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0411 08:10:34.387929   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:34.893599   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:35.392547   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:35.892783   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:36.392584   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0411 08:10:36.893074   21466 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]

@spowelljr spowelljr reopened this Apr 12, 2021
@medyagh
Copy link
Member

medyagh commented May 19, 2021

@newtoniumx3 do u mind trying with latest version of minikube again, we fixed an inagress bug in v1.20.0 that could help with this ?

@x3a
Copy link

x3a commented May 20, 2021

@medyagh I gave v1.20.0 a try, but unfortunately, the error still occurs for me.

$ minikube version                                     

minikube version: v1.20.0
commit: c61663e942ec43b20e8e70839dcca52e44cd85ae-dirty
$ minikube addons enable ingress -v=5 --alsologtostderr

I0520 22:43:30.604806  640679 out.go:291] Setting OutFile to fd 1 ...
I0520 22:43:30.604943  640679 out.go:343] isatty.IsTerminal(1) = true
I0520 22:43:30.604952  640679 out.go:304] Setting ErrFile to fd 2...
I0520 22:43:30.604958  640679 out.go:343] isatty.IsTerminal(2) = true
I0520 22:43:30.605082  640679 root.go:316] Updating PATH: /home/x3a/.minikube/bin
W0520 22:43:30.605191  640679 root.go:291] Error reading config file at /home/x3a/.minikube/config/config.json: open /home/x3a/.minikube/config/config.json: no such file or directory
I0520 22:43:30.605551  640679 addons.go:55] Setting ingress=true in profile "minikube"
I0520 22:43:30.605568  640679 addons.go:131] Setting addon ingress=true in "minikube"
I0520 22:43:30.605587  640679 host.go:66] Checking if "minikube" exists ...
I0520 22:43:30.606144  640679 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0520 22:43:30.643166  640679 out.go:170]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
I0520 22:43:30.646769  640679 out.go:170]     ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
I0520 22:43:30.648047  640679 out.go:170]     ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
I0520 22:43:30.648082  640679 addons.go:261] installing /etc/kubernetes/addons/ingress-configmap.yaml
I0520 22:43:30.648104  640679 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-configmap.yaml (1865 bytes)
I0520 22:43:30.648157  640679 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0520 22:43:30.672385  640679 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49172 SSHKeyPath:/home/x3a/.minikube/machines/minikube/id_rsa Username:docker}
I0520 22:43:30.762305  640679 addons.go:261] installing /etc/kubernetes/addons/ingress-rbac.yaml
I0520 22:43:30.762330  640679 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-rbac.yaml (6005 bytes)
I0520 22:43:30.769416  640679 addons.go:261] installing /etc/kubernetes/addons/ingress-dp.yaml
I0520 22:43:30.769438  640679 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-dp.yaml (9394 bytes)
I0520 22:43:30.778162  640679 ssh_runner.go:149] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.2/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml
I0520 22:43:31.100192  640679 addons.go:299] Verifying addon ingress=true in "minikube"
I0520 22:43:31.102061  640679 out.go:170] 🔎  Verifying ingress addon...
🔎  Verifying ingress addon...
I0520 22:43:31.103062  640679 kapi.go:59] client config for minikube: &rest.Config{Host:"https://192.168.49.2:8443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/home/x3a/.minikube/profiles/minikube/client.crt", KeyFile:"/home/x3a/.minikube/profiles/minikube/client.key", CAFile:"/home/x3a/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0x55cf714627a0), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}
I0520 22:43:31.103381  640679 cert_rotation.go:137] Starting client certificate rotation controller
I0520 22:43:31.104004  640679 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0520 22:43:31.116662  640679 kapi.go:86] Found 1 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0520 22:43:31.116681  640679 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0520 22:43:31.620476  640679 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0520 22:43:32.119752  640679 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]

...

@mjmichael73
Copy link

Same issue happened for me , any suggestion to fix it ?

@zbeedatm
Copy link

zbeedatm commented May 27, 2021

In my case, apache o ubunto was using port 80
sudo /etc/init.d/apache2 stop solved it

@adamf
Copy link

adamf commented Jun 9, 2021

+1, i'm also seeing this issue. Will do more debugging.

@adamf
Copy link

adamf commented Jun 9, 2021

Output of -v=5:

$ minikube addons enable ingress -v=5 --alsologtostderr
I0609 13:44:53.407702   50376 out.go:291] Setting OutFile to fd 1 ...
I0609 13:44:53.407987   50376 out.go:343] isatty.IsTerminal(1) = true
I0609 13:44:53.408001   50376 out.go:304] Setting ErrFile to fd 2...
I0609 13:44:53.408006   50376 out.go:343] isatty.IsTerminal(2) = true
I0609 13:44:53.408104   50376 root.go:316] Updating PATH: /Users/adam/.minikube/bin
I0609 13:44:53.408614   50376 addons.go:55] Setting ingress=true in profile "minikube-dev-adam-Adams-Mac-mini.local"
I0609 13:44:53.408631   50376 addons.go:131] Setting addon ingress=true in "minikube-dev-adam-Adams-Mac-mini.local"
W0609 13:44:53.408638   50376 addons.go:140] addon ingress should already be in state true
I0609 13:44:53.408658   50376 host.go:66] Checking if "minikube-dev-adam-Adams-Mac-mini.local" exists ...
I0609 13:44:53.408998   50376 main.go:128] libmachine: Found binary path at /Users/adam/.minikube/bin/docker-machine-driver-hyperkit
I0609 13:44:53.409056   50376 main.go:128] libmachine: Launching plugin server for driver hyperkit
I0609 13:44:53.423011   50376 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:53568
I0609 13:44:53.424225   50376 main.go:128] libmachine: () Calling .GetVersion
I0609 13:44:53.425188   50376 main.go:128] libmachine: Using API Version  1
I0609 13:44:53.425210   50376 main.go:128] libmachine: () Calling .SetConfigRaw
I0609 13:44:53.425635   50376 main.go:128] libmachine: () Calling .GetMachineName
I0609 13:44:53.426082   50376 main.go:128] libmachine: Found binary path at /Users/adam/.minikube/bin/docker-machine-driver-hyperkit
I0609 13:44:53.426120   50376 main.go:128] libmachine: Launching plugin server for driver hyperkit
I0609 13:44:53.441083   50376 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:53574
I0609 13:44:53.442200   50376 main.go:128] libmachine: () Calling .GetVersion
I0609 13:44:53.443164   50376 main.go:128] libmachine: Using API Version  1
I0609 13:44:53.443189   50376 main.go:128] libmachine: () Calling .SetConfigRaw
I0609 13:44:53.443626   50376 main.go:128] libmachine: () Calling .GetMachineName
I0609 13:44:53.443834   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) Calling .GetState
I0609 13:44:53.444029   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) DBG | exe=/Users/adam/.minikube/bin/docker-machine-driver-hyperkit uid=0
I0609 13:44:53.444331   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) DBG | hyperkit pid from json: 7241
I0609 13:44:53.447095   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) Calling .DriverName
I0609 13:44:53.473628   50376 out.go:170]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
I0609 13:44:53.499238   50376 out.go:170]     ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
I0609 13:44:53.524354   50376 out.go:170]     ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
I0609 13:44:53.524794   50376 addons.go:261] installing /etc/kubernetes/addons/ingress-configmap.yaml
I0609 13:44:53.524876   50376 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-configmap.yaml (1865 bytes)
I0609 13:44:53.524959   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) Calling .GetSSHHostname
I0609 13:44:53.525330   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) Calling .GetSSHPort
I0609 13:44:53.525524   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) Calling .GetSSHKeyPath
I0609 13:44:53.525699   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) Calling .GetSSHUsername
I0609 13:44:53.525852   50376 sshutil.go:53] new ssh client: &{IP:192.168.64.140 Port:22 SSHKeyPath:/Users/adam/.minikube/machines/minikube-dev-adam-Adams-Mac-mini.local/id_rsa Username:docker}
I0609 13:44:53.580031   50376 addons.go:261] installing /etc/kubernetes/addons/ingress-rbac.yaml
I0609 13:44:53.580075   50376 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-rbac.yaml (6005 bytes)
I0609 13:44:53.595222   50376 addons.go:261] installing /etc/kubernetes/addons/ingress-dp.yaml
I0609 13:44:53.595251   50376 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-dp.yaml (9394 bytes)
I0609 13:44:53.618277   50376 ssh_runner.go:149] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.2/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml
I0609 13:44:53.876058   50376 main.go:128] libmachine: Making call to close driver server
I0609 13:44:53.876083   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) Calling .Close
I0609 13:44:53.876304   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) DBG | Closing plugin on server side
I0609 13:44:53.876336   50376 main.go:128] libmachine: Successfully made call to close driver server
I0609 13:44:53.876356   50376 main.go:128] libmachine: Making call to close connection to plugin binary
I0609 13:44:53.876368   50376 main.go:128] libmachine: Making call to close driver server
I0609 13:44:53.876385   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) Calling .Close
I0609 13:44:53.876558   50376 main.go:128] libmachine: (minikube-dev-adam-Adams-Mac-mini.local) DBG | Closing plugin on server side
I0609 13:44:53.876634   50376 main.go:128] libmachine: Successfully made call to close driver server
I0609 13:44:53.876648   50376 main.go:128] libmachine: Making call to close connection to plugin binary
I0609 13:44:53.876666   50376 addons.go:299] Verifying addon ingress=true in "minikube-dev-adam-Adams-Mac-mini.local"
I0609 13:44:53.902697   50376 out.go:170] 🔎  Verifying ingress addon...
🔎  Verifying ingress addon...
I0609 13:44:53.909612   50376 kapi.go:59] client config for minikube-dev-adam-Adams-Mac-mini.local: &rest.Config{Host:"https://192.168.64.140:6443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/Users/adam/.minikube/profiles/minikube-dev-adam-Adams-Mac-mini.local/client.crt", KeyFile:"/Users/adam/.minikube/profiles/minikube-dev-adam-Adams-Mac-mini.local/client.key", CAFile:"/Users/adam/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0x229a3c0), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}
I0609 13:44:53.910808   50376 cert_rotation.go:137] Starting client certificate rotation controller
I0609 13:44:53.913638   50376 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0609 13:44:53.932166   50376 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0609 13:44:53.932219   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:54.437704   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:54.941440   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:55.439702   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:55.940979   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:56.439771   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:56.945570   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:57.438554   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:57.940722   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:58.436692   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:58.938086   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0609 13:44:59.439583   50376 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]

@andriyDev
Copy link
Contributor

andriyDev commented Jun 9, 2021

@adamf Can you do a fresh setup and send the logs?

$ minikube delete --purge --all
$ minikube start --addons=ingress --alsologtostderr

I'm having trouble recreating the issue. Also add any driver settings and make sure to paste them here.

@andriyDev
Copy link
Contributor

@Bytesu There's an even easier way to test this actually, just use minikube addons enable ingress --images="IngressController=ingress-nginx/controller:v0.44.0"

Are either of you using a corp registry of some kind?

@Bytesu
Copy link

Bytesu commented Aug 24, 2021

minikube addons enable ingress --images="IngressController=ingress-nginx/controller:v0.44.0

no , I used another way to solve my problem . Thanks very much . and I suppose my issue is leaded by network that I couldn't access the k8s.gcr.io

@Himkwok
Copy link

Himkwok commented Aug 24, 2021

minikube addons enable ingress --images="IngressController=ingress-nginx/controller:v0.44.0

no , I used another way to solve my problem . Thanks very much . and I suppose my issue is leaded by network that I couldn't access the k8s.gcr.io

you can pull image from dockerhub

docker pull pollyduan/ingress-nginx-controller:v0.44.0

docker tag pollyduan/ingress-nginx-controller:v0.44.0 k8s.gcr.io/ingress-nginx/controller:v0.44.0

@Himkwok
Copy link

Himkwok commented Aug 24, 2021

....
containers:

  • name: controller
    image: 'k8s.gcr.io/ingress-nginx/controller:v0.44.0'
    ......

i edited yaml ,changed 'k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:........ ' to 'k8s.gcr.io/ingress-nginx/controller:v0.44.0', then kubectl apply -f spec.yaml and minikube addons enable ingress , IT WORKS!!!!!

where is the spec.yaml?

kubectl edit pods ingress-nginx-controller-7b8cbd57c9-jnzmd -n ingress-nginx and change to 'image: k8s.gcr.io/ingress-nginx/controller:v0.44.0', I hope these can help

@Bytesu
Copy link

Bytesu commented Aug 26, 2021

minikube addons enable ingress --images="IngressController=ingress-nginx/controller:v0.44.0

no , I used another way to solve my problem . Thanks very much . and I suppose my issue is leaded by network that I couldn't access the k8s.gcr.io

you can pull image from dockerhub

docker pull pollyduan/ingress-nginx-controller:v0.44.0

docker tag pollyduan/ingress-nginx-controller:v0.44.0 k8s.gcr.io/ingress-nginx/controller:v0.44.0

I had tried like this before I pasted here, it's still something wrong with it

@yangyaofei
Copy link

[minikube@localhost root]$ minikube kubectl -- get pods --selector="app.kubernetes.io/name=ingress-nginx" -n ingress-nginx
NAME                                       READY   STATUS              RESTARTS   AGE
ingress-nginx-admission-create-k9dr6       0/1     ImagePullBackOff    0          17m
ingress-nginx-admission-patch-cjwvj        0/1     ImagePullBackOff    0          17m
ingress-nginx-controller-8668846c7-8q7fp   0/1     ContainerCreating   0          17m

[minikube@localhost root]$ minikube kubectl -- logs ingress-nginx-admission-create-k9dr6  -n ingress-nginx
Error from server (BadRequest): container "create" in pod "ingress-nginx-admission-create-k9dr6" is waiting to start: trying and failing to pull image

[minikube@localhost root]$ minikube kubectl -- logs ingress-nginx-admission-patch-cjwvj  -n ingress-nginx
Error from server (BadRequest): container "patch" in pod "ingress-nginx-admission-patch-cjwvj" is waiting to start: trying and failing to pull image

[minikube@localhost root]$ minikube kubectl -- logs ingress-nginx-controller-8668846c7-8q7fp  -n ingress-nginx
Error from server (BadRequest): container "controller" in pod "ingress-nginx-controller-8668846c7-8q7fp" is waiting to start: ContainerCreating

It's like image not pulled , but the log below looks like pulled success.

[minikube@localhost root]$ minikube addons enable ingress
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/docker.io/jettech/kube-webhook-certgen:v1.5.1@sha256:950833e19ade18cd389d647efb88992a7cc077abedef343fa59e012d376d79b7 (global image repository)
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a (global image repository)
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/docker.io/jettech/kube-webhook-certgen:v1.5.1@sha256:950833e19ade18cd389d647efb88992a7cc077abedef343fa59e012d376d79b7 (global image repository)
🔎  Verifying ingress addon...

kube connect this

Running on machine: localhost
Binary: Built with gc go1.16.4 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0827 15:15:16.997734   60608 out.go:286] Setting OutFile to fd 1 ...
I0827 15:15:16.998152   60608 out.go:338] isatty.IsTerminal(1) = true
I0827 15:15:16.998158   60608 out.go:299] Setting ErrFile to fd 2...
I0827 15:15:16.998161   60608 out.go:338] isatty.IsTerminal(2) = true
I0827 15:15:16.998366   60608 root.go:312] Updating PATH: /home/minikube/.minikube/bin
I0827 15:15:16.999761   60608 addons.go:59] Setting ingress=true in profile "minikube"
I0827 15:15:16.999776   60608 addons.go:135] Setting addon ingress=true in "minikube"
I0827 15:15:17.000074   60608 host.go:66] Checking if "minikube" exists ...
I0827 15:15:17.001043   60608 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0827 15:15:17.169484   60608 out.go:165]     ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a (global image repository)
I0827 15:15:17.170033   60608 out.go:165]     ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/docker.io/jettech/kube-webhook-certgen:v1.5.1@sha256:950833e19ade18cd389d647efb88992a7cc077abedef343fa59e012d376d79b7 (global image repository)
I0827 15:15:17.170506   60608 out.go:165]     ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/docker.io/jettech/kube-webhook-certgen:v1.5.1@sha256:950833e19ade18cd389d647efb88992a7cc077abedef343fa59e012d376d79b7 (global image repository)
I0827 15:15:17.170579   60608 addons.go:275] installing /etc/kubernetes/addons/ingress-configmap.yaml
I0827 15:15:17.170598   60608 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-configmap.yaml (1865 bytes)
I0827 15:15:17.170680   60608 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0827 15:15:17.216884   60608 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49172 SSHKeyPath:/home/minikube/.minikube/machines/minikube/id_rsa Username:docker}
I0827 15:15:17.339541   60608 addons.go:275] installing /etc/kubernetes/addons/ingress-rbac.yaml
I0827 15:15:17.339557   60608 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-rbac.yaml (6005 bytes)
I0827 15:15:17.353338   60608 addons.go:275] installing /etc/kubernetes/addons/ingress-dp.yaml
I0827 15:15:17.353345   60608 ssh_runner.go:316] scp memory --> /etc/kubernetes/addons/ingress-dp.yaml (9539 bytes)
I0827 15:15:17.367147   60608 ssh_runner.go:149] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.21.2/kubectl apply -f /etc/kubernetes/addons/ingress-configmap.yaml -f /etc/kubernetes/addons/ingress-rbac.yaml -f /etc/kubernetes/addons/ingress-dp.yaml
I0827 15:15:18.310897   60608 addons.go:313] Verifying addon ingress=true in "minikube"
I0827 15:15:18.322408   60608 out.go:165] 🔎  Verifying ingress addon...
I0827 15:15:18.339342   60608 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0827 15:15:18.419365   60608 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0827 15:21:18.423923   60608 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0827 15:21:18.426565   60608 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0827 15:21:18.426575   60608 kapi.go:108] duration metric: took 6m0.089261197s to wait for app.kubernetes.io/name=ingress-nginx ...
I0827 15:21:18.435685   60608 out.go:165]
W0827 15:21:18.435908   60608 out.go:230] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
W0827 15:21:18.435937   60608 out.go:230]
W0827 15:21:18.446326   60608 out.go:230] ╭─────────────────────────────────────────────────────────────────────────────╮
│                                                                             │
│    😿  If the above advice does not help, please let us know:               │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose             │
│                                                                             │
│    Please attach the following file to the GitHub issue:                    │
│    - /tmp/minikube_addons_657d376187cd72746604141ceddc839ee4e6f05e_0.log    │
│                                                                             │
╰─────────────────────────────────────────────────────────────────────────────╯

@yangyaofei
Copy link

yangyaofei commented Aug 27, 2021

I think I know why, it because the China's great GFW build by you know who.

In the minikube , here is a --image-mirror-country , and unfortunately I should set it to cn ,to use aliyun's registry.

But in some reason, it not available. so I get that error.


If other dude in China, not use --image-mirror-country, it will use gcr and will get same thing for good reason by GFW.


solution

In minikube ssh pull image from docker hub :

  1. jettech/kube-webhook-certgen:v1.5.1
  2. willdockerhub/ingress-nginx-controller:v0.44.0

The second is a personal image just mirror ingress-nginx/controller for gcr

And then rename tag to what you need, and then minikube addons enable ingress


I think the registry of aliyun can use for his own build maybe

@pshendre1
Copy link

I had the same issue with 'minikube addons enable ingress' command.
Hence tried the following solution:
Tried to pull the ingress image directly using 'docker pull' from command line
Got the error - Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp 192.168.10.1:443: connect: connection refused
Changed the network to another ISP and reissued the 'docker pull' command.
This time the 2 images (k8s.gcr.io/ingress-nginx/controller:v0.44.0 , docker.io/jettech/kube-webhook-certgen:v1.5.1 ) downloaded properly
Post that, issued the minikube addons enable ingress command and it worked successfully

I faced this image downloading issue during other scenarios also and had to use the same - switch ISP solution.
So bottomline - one possibility to check is

  1. Change ISP
  2. Download images using normal 'docker pull '
  3. Try 'minikube addons enable ingress' command again

@medyagh
Copy link
Member

medyagh commented Sep 22, 2021

I think I know why, it because the China's great GFW build by you know who.

In the minikube , here is a --image-mirror-country , and unfortunately I should set it to cn ,to use aliyun's registry.

But in some reason, it not available. so I get that error.

If other dude in China, not use --image-mirror-country, it will use gcr and will get same thing for good reason by GFW.

solution

In minikube ssh pull image from docker hub :

  1. jettech/kube-webhook-certgen:v1.5.1
  2. willdockerhub/ingress-nginx-controller:v0.44.0

The second is a personal image just mirror ingress-nginx/controller for gcr

And then rename tag to what you need, and then minikube addons enable ingress

I think the registry of aliyun can use for his own build maybe

@yangyaofei
I would be happy to accept a PR or review a PR that fixes it

do you think this PR helps ?

#12411

@yangyaofei
Copy link

I think I know why, it because the China's great GFW build by you know who.
In the minikube , here is a --image-mirror-country , and unfortunately I should set it to cn ,to use aliyun's registry.
But in some reason, it not available. so I get that error.
If other dude in China, not use --image-mirror-country, it will use gcr and will get same thing for good reason by GFW.

solution

In minikube ssh pull image from docker hub :

  1. jettech/kube-webhook-certgen:v1.5.1
  2. willdockerhub/ingress-nginx-controller:v0.44.0

The second is a personal image just mirror ingress-nginx/controller for gcr
And then rename tag to what you need, and then minikube addons enable ingress
I think the registry of aliyun can use for his own build maybe

@yangyaofei
I would be happy to accept a PR or review a PR that fixes it

do you think this PR helps ?

#12411

Hi @medyagh thank you to let me know this.

Actually, I'm a starter with this. so I don't know it will help or not . But I saw #11785 , I think this will be helped.

And I have a suggest that is add more error message when cannot pull the image, and add little more document about it and a general solution for this. There are two types dev in china, one is read official doc and use google, another is read chinese blog and use baidu . The devs only use baidu will follow the chinese blog and probably use the version build by alibaba( aliyun ), The devs can use google, only thing they need is to know why this error occur, and then they can solve it by this own or with a simple guide with few sentences.

And for #11785 , it's good. I'm not sure that googlecontainer repo is build by aliyun or some user of aliyun. I think that's not reliable, unless that repo is build by aliyun officially and can offer a map for the image's name. If not, sooner or later , someone will open a same issue. And that's why I think a few extra document is needed.

For the record, same block for the ghcr and other repo, so technically is not the problem of minikube or gcr 😂😂😂

@spowelljr spowelljr added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed kind/support Categorizes issue or PR as a support question. labels Oct 6, 2021
@SteamUpdate
Copy link

I had same problem. There is how I solved it:
OS: Ubuntu 20.04.2 LTS
minikube: v1.16.0
driver: docker

An error:

ERR command "minikube addons enable ingress" output:
* Verifying ingress addon...
 
X Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
*
* If the above advice does not help, please let us know:

The solution:

  1. Looking for pod "kube-proxy" with status "CrashLoopBackOff":
    kubectl get pod -n kube-system
  2. Need to get logs of this service:
    kubectl logs kube-proxy-randomsymbols -n kube-system
  3. Logs has to contain an error linked with parameter net/netfilter/nf_conntrack_max by reason Permission denied.
    Need to set this parameter manually. Value of this parameter contains in logs. In my case value was 196608.
    sudo sysctl net/netfilter/nf_conntrack_max=196608
  4. Restart or recreate minikube

Note: Value of net/netfilter/nf_conntrack_max resets after every reboot

@kevinw66
Copy link

kevinw66 commented Oct 24, 2021

I'm in china and here is my way to solve this problem:
minikube version: v1.23.2

I found no matter how I specify the --image="IngressController=***", It still goes to k8s.gcr.io
So I looked into source code and found this

"ingress": NewAddon([]*BinAsset{
		MustBinAsset(addons.IngressAssets,
			"ingress/ingress-deploy.yaml.tmpl",
			vmpath.GuestAddonsDir,
			"ingress-deploy.yaml",
			"0640"),
	}, false, "ingress", "", map[string]string{
		// https://github.com/kubernetes/ingress-nginx/blob/557604f4ef526f7755d36089b617bc7686c389f9/deploy/static/provider/kind/deploy.yaml#L323
		"IngressController": "ingress-nginx/controller:v1.0.0-beta.3@sha256:44a7a06b71187a4529b0a9edee5cc22bdf71b414470eff696c3869ea8d90a695",
		// https://github.com/kubernetes/ingress-nginx/blob/557604f4ef526f7755d36089b617bc7686c389f9/deploy/static/provider/kind/deploy.yaml#L612
		"KubeWebhookCertgenCreate": "k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068",
		// https://github.com/kubernetes/ingress-nginx/blob/557604f4ef526f7755d36089b617bc7686c389f9/deploy/static/provider/kind/deploy.yaml#L660
		"KubeWebhookCertgenPatch": "k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068",
	}, map[string]string{
		"IngressController": "k8s.gcr.io",
	}),
func NewAddon(assets []*BinAsset, enabled bool, addonName string, maintainer string, images map[string]string, registries map[string]string) *Addon {
	a := &Addon{
		Assets:     assets,
		enabled:    enabled,
		addonName:  addonName,
		Maintainer: maintainer,
		Images:     images,
		Registries: registries,
	}
	return a
}

So I use minikube addons enable ingress --registries="IngressController=docker.io" --images="IngressController=wangshun1024/ingress-nginx-controller:v1.0.0-beta.3,KubeWebhookCertgenCreate=liangjw/kube-webhook-certgen:v1.0,KubeWebhookCertgenPatch=liangjw/kube-webhook-certgen:v1.0" and it works.

For others:
You can found a replacement ingress and kube-webhook-certgen images for your version of minikube and use the same command as I but with different --images and --registries parameters.

Some suggestions:
Maybe remove the Registries parameter for IngressController and move into Images directly is a better idea? just like
KubeWebhookCertgenCreate and KubeWebhookCertgenPatch.

@Erfan-Motallebi
Copy link

Erfan-Motallebi commented Nov 9, 2021

I got the same issue when running minikube addons enable ingress

  • minikube version: v1.24.0
  • Kubernetes version : It's been said in the detail mode.
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
- Docker Version: It's been said in the detail mode .
Client: Docker Engine - Community Version: 20.10.10 API version: 1.41 Go version: go1.16.9 Git commit: b485636 Built: Mon Oct 25 07:42:59 2021 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.10
API version: 1.41 (minimum version 1.12)
Go version: go1.16.9
Git commit: e2f740d
Built: Mon Oct 25 07:41:08 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.11
GitCommit: 5b46e404f6b9f661a205e28d59c982d3634148f8
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0

I even tested out this approach:

  • minikube stop
  • minikube start
  • minikube addons enable ingress

Hint: I first enabled the ingress add-on and then went for the Mandatory ingress-nginx services. The result was confusing. It went all good for all of them with the exception of the last 2 items on the list.

  • job.batch/ingress-nginx-admission-create unchanged
  • job.batch/ingress-nginx-admission-patch unchanged

What's wrong with them?
I even deleted the minikube and started it over. It hasn't worked though.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 7, 2022
@prezha
Copy link
Collaborator

prezha commented Feb 7, 2022

there seem to be potentially multiple different/similar problems reported and also several solutions/workarounds in this issue (specifically around firewalls) and we've implemented a number of improvements in the meantime (last being #13439 that's merger but not yet released), so I'll close this issue now and please open a new one if you still face issues with the ingress addon

thank you

/close

@prezha prezha closed this as completed Feb 7, 2022
@adn1107
Copy link

adn1107 commented May 1, 2022

I have tried all the suggestions in this thread but still having this issue? What is the final solution?

@yzcyayaya
Copy link

我已经尝试了这个线程中的所有建议,但仍然有这个问题?最终的解决方案是什么?
I also have this situation and it is not resolved at present

@nicholasyin
Copy link

Even if I managed to manually pull the 2 images related to enabling ingress, the minikube addons enable ingress still failed with the same error. The logic is kind of dumb when it can't detect whether the required images are already in minikube VM. Simply put, if you can't pull, I pull it for you, and you still don't work.

@ongiant
Copy link

ongiant commented Jul 10, 2023

I think I know why, it because the China's great GFW build by you know who.

In the minikube , here is a --image-mirror-country , and unfortunately I should set it to cn ,to use aliyun's registry.

But in some reason, it not available. so I get that error.

If other dude in China, not use --image-mirror-country, it will use gcr and will get same thing for good reason by GFW.

solution

In minikube ssh pull image from docker hub :

  1. jettech/kube-webhook-certgen:v1.5.1
  2. willdockerhub/ingress-nginx-controller:v0.44.0

The second is a personal image just mirror ingress-nginx/controller for gcr

And then rename tag to what you need, and then minikube addons enable ingress

I think the registry of aliyun can use for his own build maybe

I failed after using your method(network proxy were running on my local machine):

docker@polar:~$ docker pull registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f 
Error response from daemon: Get "https://registry.k8s.io/v2/": proxyconnect tcp: dial tcp 0.0.0.0:7891: connect: connection refused

@KaiJiaBrother
Copy link

KaiJiaBrother commented Nov 23, 2023

In my situation, the main issue is that the images were not successfully pulled.

I tried minikube delete --all --purge, unfortunately it didn't help.

So I edited the pods and removed the @sha256... bit from the image:

kubectl -n ingress-nginx edit pod ingress-nginx-admission-create-xxxx
kubectl -n ingress-nginx edit pod ingress-nginx-admission-patch-xxxx
kubectl -n ingress-nginx edit pod ingress-nginx-controller-xxxx

e.g.: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9 to registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen

After modification, there is no need to manually pull the images, the images will be pulled automatically.

Finally, try

minikube addons enable ingress

And it works!


If you are in China, try to add --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers in minikube start, similar to this:

minikube start \
    --image-mirror-country=cn \
    --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
    --force \
    --addons ingress

I hope these could help somebody!

@itwjl
Copy link

itwjl commented May 16, 2024

In my situation, the main issue is that the images were not successfully pulled.

I tried minikube delete --all --purge, unfortunately it didn't help.

So I edited the pods and removed the @sha256... bit from the image:

kubectl -n ingress-nginx edit pod ingress-nginx-admission-create-xxxx
kubectl -n ingress-nginx edit pod ingress-nginx-admission-patch-xxxx
kubectl -n ingress-nginx edit pod ingress-nginx-controller-xxxx

e.g.: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9 to registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen

After modification, there is no need to manually pull the images, the images will be pulled automatically.

Finally, try

minikube addons enable ingress

And it works!

If you are in China, try to add --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers in minikube start, similar to this:

minikube start \
    --image-mirror-country=cn \
    --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
    --force \
    --addons ingress

I hope these could help somebody!

it works, very good

In my situation, the main issue is that the images were not successfully pulled.

I tried minikube delete --all --purge, unfortunately it didn't help.

So I edited the pods and removed the @sha256... bit from the image:

kubectl -n ingress-nginx edit pod ingress-nginx-admission-create-xxxx
kubectl -n ingress-nginx edit pod ingress-nginx-admission-patch-xxxx
kubectl -n ingress-nginx edit pod ingress-nginx-controller-xxxx

e.g.: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9 to registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen

After modification, there is no need to manually pull the images, the images will be pulled automatically.

Finally, try

minikube addons enable ingress

And it works!

If you are in China, try to add --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers in minikube start, similar to this:

minikube start \
    --image-mirror-country=cn \
    --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
    --force \
    --addons ingress

I hope these could help somebody!

minikube_addons_1ba5e275a043117389ed64b6bf4c3b62e1556582_0.log
报错信息(log日志):
启用 'ingress' 返回了错误: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline exceeded]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
addon/ingress area/addons lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. top-10-issues Top 10 support issues
Projects
None yet
Development

No branches or pull requests