Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

istio-ingressgateway: tunnel doesn't start #10085

Open
alexstaroselsky opened this issue Jan 2, 2021 · 26 comments
Open

istio-ingressgateway: tunnel doesn't start #10085

alexstaroselsky opened this issue Jan 2, 2021 · 26 comments
Assignees
Labels
area/tunnel Support for the tunnel command help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@alexstaroselsky
Copy link

alexstaroselsky commented Jan 2, 2021

While minikube seems to be starting and running effectively with minikube start, I am unable to successfully execute the command minikube tunnel. After entering the password and waiting a significant amount of time, no output similar to what is show on Accessing apps displays nor does the minikube ip seem to be responsive. The last message to display with logging verbosity level 2 is Patched istio-ingressgateway with IP 127.0.0.1. I've tried running with sudo as well, but same hanging.

This is using the default kube config generated from minikube start. Minikube was installed via homebrew.

System:
MacOS - 11.1
Docker - 20.10.0
Kubernetes - 1.19.3
Minikube - 1.16.0

Steps to reproduce the issue:

  1. minikube start
  2. minikube tunnel --alsologtostderr --v=2

Full output of failed command:

minikube tunnel --alsologtostderr --v=2
I0102 13:56:53.026587    5794 out.go:221] Setting OutFile to fd 1 ...
I0102 13:56:53.027277    5794 out.go:273] isatty.IsTerminal(1) = true
I0102 13:56:53.027295    5794 out.go:234] Setting ErrFile to fd 2...
I0102 13:56:53.027303    5794 out.go:273] isatty.IsTerminal(2) = true
I0102 13:56:53.027417    5794 root.go:280] Updating PATH: /Users/someuser/.minikube/bin
W0102 13:56:53.027576    5794 root.go:255] Error reading config file at /Users/someuser/.minikube/config/config.json: open /Users/someuser/.minikube/config/config.json: no such file or directory
I0102 13:56:53.028066    5794 mustload.go:66] Loading cluster: minikube
I0102 13:56:53.029093    5794 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0102 13:56:53.184465    5794 host.go:66] Checking if "minikube" exists ...
I0102 13:56:53.184936    5794 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" minikube
I0102 13:56:53.329575    5794 api_server.go:146] Checking apiserver status ...
I0102 13:56:53.329732    5794 ssh_runner.go:149] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0102 13:56:53.329817    5794 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0102 13:56:53.480793    5794 sshutil.go:48] new ssh client: &{IP:127.0.0.1 Port:55007 SSHKeyPath:/Users/someuser/.minikube/machines/minikube/id_rsa Username:docker}
I0102 13:56:53.621072    5794 ssh_runner.go:149] Run: sudo egrep ^[0-9]+:freezer: /proc/1884/cgroup
I0102 13:56:53.632939    5794 api_server.go:162] apiserver freezer: "7:freezer:/docker/f65f71a326b1bc0138a18b4f832afb887fd58b3e919089379f915cb88d2f67ae/kubepods/burstable/pod524cecac593a7ad14f29307cb61f56b8/7f39232f1fc0ca71da44a5579f60e7d6b0839e7717a4bafd3470a7ef23ba5eee"
I0102 13:56:53.633091    5794 ssh_runner.go:149] Run: sudo cat /sys/fs/cgroup/freezer/docker/f65f71a326b1bc0138a18b4f832afb887fd58b3e919089379f915cb88d2f67ae/kubepods/burstable/pod524cecac593a7ad14f29307cb61f56b8/7f39232f1fc0ca71da44a5579f60e7d6b0839e7717a4bafd3470a7ef23ba5eee/freezer.state
I0102 13:56:53.650094    5794 api_server.go:184] freezer state: "THAWED"
I0102 13:56:53.650147    5794 api_server.go:221] Checking apiserver healthz at https://127.0.0.1:55004/healthz ...
I0102 13:56:53.663613    5794 api_server.go:241] https://127.0.0.1:55004/healthz returned 200:
ok
I0102 13:56:53.663646    5794 tunnel.go:57] Checking for tunnels to cleanup...
I0102 13:56:53.665014    5794 kapi.go:59] client config for minikube: &rest.Config{Host:"https://127.0.0.1:55004", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/Users/someuser/.minikube/profiles/minikube/client.crt", KeyFile:"/Users/someuser/.minikube/profiles/minikube/client.key", CAFile:"/Users/someuser/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0x541a300), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil)}
I0102 13:56:53.669189    5794 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0102 13:56:53.838208    5794 out.go:119] ❗  The service istio-ingressgateway requires privileged ports to be exposed: [80 443]
❗  The service istio-ingressgateway requires privileged ports to be exposed: [80 443]
I0102 13:56:53.843739    5794 out.go:119] 🔑  sudo permission will be asked for it.
🔑  sudo permission will be asked for it.
I0102 13:56:53.851142    5794 out.go:119] 🏃  Starting tunnel for service istio-ingressgateway.
🏃  Starting tunnel for service istio-ingressgateway.
I0102 13:56:53.854697    5794 loadbalancer_patcher.go:121] Patched istio-ingressgateway with IP 127.0.0.1

Full output of minikube start command used, if not already included:

😄  minikube v1.16.0 on Darwin 11.1
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Optional: Full output of minikube logs command:

@lucashimizu
Copy link

Exact same problem, trying to open traffic through an Istio Ingress Gateway.

@priyawadhwa priyawadhwa added the kind/support Categorizes issue or PR as a support question. label Jan 25, 2021
@medyagh medyagh changed the title tunnel doesn't start istio-ingressgateway: tunnel doesn't start Jan 27, 2021
@medyagh medyagh added kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Jan 27, 2021
@medyagh
Copy link
Member

medyagh commented Jan 27, 2021

@lucashimizu @alexstaroselsky
Thank you for reporting this. This does seem like a bug, I would accept a PR from any isto experts to fix this !

@berk2s
Copy link

berk2s commented Feb 23, 2021

same issue

@slonka
Copy link

slonka commented Mar 23, 2021

Any updates on this?

@jonassteinberg1
Copy link

jonassteinberg1 commented Mar 29, 2021

any update on this?

@sharifelgamal sharifelgamal added area/tunnel Support for the tunnel command help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. and removed kind/support Categorizes issue or PR as a support question. labels Apr 28, 2021
@sharifelgamal
Copy link
Collaborator

This does indeed seem to be a bug with minikube tunnel. We'd love some help pinning down what the exact issue is. Help wanted!

@martinknechtel
Copy link

same for me with
MacOS - 11.3.1
Docker - 20.10.5
Kubernetes - 1.20.2
Minikube - 1.19.0

@martinknechtel
Copy link

With Hyperkit (v0.20210107-2-g2f061e) instead of Docker, it is running fine.

@vaibhavmagon
Copy link

Same here. Two problems:

  1. The minikube tunnel assigns some different IP and not 127.0.0.1

Screenshot 2021-05-10 at 5 57 56 PM

  1. Not able to access even though individual services are working fine.

Screenshot 2021-05-10 at 5 57 39 PM

@AlbertMarashi
Copy link

AlbertMarashi commented May 13, 2021

image

Also getting this issue

Windows
Minikube - v1.20.0
Docker - v20.10.5

Related issues:

#10762
#10152
#10265

@AlbertMarashi
Copy link

AlbertMarashi commented May 13, 2021

With Hyperkit (v0.20210107-2-g2f061e) instead of Docker, it is running fine.

How did you fix it? @martinknechtel

@AlbertMarashi
Copy link

This is coming up as the 4th result on google for "minikube tunnel not working" and 3rd for "minikube tunnel not starting"

@medyagh @sharifelgamal how can we escalate this? Seems like a quite impactful bug with tunnel. Tried a fresh install on my MacOS and it doesn't work on that either.

@AlbertMarashi
Copy link

AlbertMarashi commented May 13, 2021

I found a solution

I had to expose a "LoadBalancer" in order for me to reach the app. This was mentioned nowhere on the docs.

Here's what I had to do.

my-ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: hello-nodejs-service
            port:
              number: 80

my-service.yml

apiVersion: v1
kind: Service
metadata:
  name: hello-nodejs-service
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: hello-nodejs

my-deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-nodejs-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello-nodejs
  template:
    metadata:
      labels:
        app: hello-nodejs
    spec:
      containers:
      - image: hello-nodejs:latest #you need to switch this with your own container image / or use a public docker image
        imagePullPolicy: IfNotPresent
        name: hello-nodejs
        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"
        ports:
        - containerPort: 80

Apply the configs with `kubectl apply -f filename.yml

Then I had to run the following:
kubectl expose deployment my-deployment --type=LoadBalancer --port=80
after that was done
minikube tunnel would start and output a message. It wasn't "hanging" It just had no deployment running

edit
still having issues with this now

@martinknechtel
Copy link

With Hyperkit (v0.20210107-2-g2f061e) instead of Docker, it is running fine.

How did you fix it? @martinknechtel

@AlbertMarashi The only pitfall I had on starting up minikube is broken DNS connection, but thats another problem ;-) Observation:

❯ minikube start
[...]
❗  This VM is having trouble accessing https://k8s.gcr.io

Solution:

minikube ssh
rm -f /etc/resolv.conf && echo nameserver 192.168.178.1 > /etc/resolv.conf #replace with your nameserver IP

@AlbertMarashi
Copy link

I don't know why this wasn't mentioned in the docs anywhere, but you need to run the following before your ingress works
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/cloud/deploy.yaml

After I ran this command, my endpoints were available on 127.0.0.1

If you are using hosts, don't forget to put them in your /etc/hosts

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 14, 2021
@netfishx
Copy link

Any updates on this?

@sharifelgamal sharifelgamal removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 1, 2021
@sharifelgamal
Copy link
Collaborator

I suspect this remains an issue, but we haven't had the bandwidth to look at this more closely. Help is of course wanted and we'd be happy to review any PRs that fix this.

@alekseinovikov
Copy link

I have the same issue

@aleti-pavan
Copy link

I have same issue with 'minikube tunnel' I ran this part of istio installation and sample application deployment

@pgoldste
Copy link

Having the same issue. MacOS 11.6, Docker. minikube installed via homebrew. Trying to follow istio's tutorial and the minikube tunnel doesn't get past asking me for sudo password.

@kty1965
Copy link

kty1965 commented Oct 22, 2021

Follow this guide

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}')
export TCP_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="tcp")].port}')

and then minikube tunnel

http://127.0.0.1/productpage

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 20, 2022
@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 19, 2022
@sharifelgamal sharifelgamal added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Mar 2, 2022
@kubernetes kubernetes deleted a comment from k8s-triage-robot Mar 2, 2022
@kubernetes kubernetes deleted a comment from k8s-triage-robot Mar 2, 2022
@sharifelgamal
Copy link
Collaborator

If there is a way to integrate the steps specified above into a PR in minikube's code directly, I would to review it.

@chungjin
Copy link
Contributor

chungjin commented Mar 7, 2022

/assign

@Jonnymcc
Copy link

After trying the code to export the host in the comment by @kty1965 I got the istio book info demo to work. I'm not sure if there is a problem with minikube or it is istio's docs. I was thrown off here when it says if the external-ip is pending that you should use the nodeport. It is pending until you start the tunnel. After, in my case on a Mac, the external-ip is no longer "pending" but 127.0.0.1.

It seems to me that minikube tunnel is doing what it is expected to do albeit not printing anything after entering the root password. What I wonder is, as per istio's docs, should I be able to use the node port and the ip provided by minikube ip?

@spowelljr spowelljr added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Jun 29, 2022
@surajkrishan
Copy link

any update on this ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/tunnel Support for the tunnel command help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Development

No branches or pull requests