Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minikube Ingress TCP Services Not Forwarding Traffic #2840

Closed
justinrcs opened this issue May 24, 2018 · 30 comments
Closed

Minikube Ingress TCP Services Not Forwarding Traffic #2840

justinrcs opened this issue May 24, 2018 · 30 comments

Comments

@justinrcs
Copy link

@justinrcs justinrcs commented May 24, 2018

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version):
minikube version: v0.27.0

  • OS (e.g. from /etc/os-release):
    MacOS 10.12.6 Sierra

  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
    virtualbox

  • ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):

minikube-v0.26.0.iso

  • Install tools:
    brew

What happened:
Enabled ingress on minikube via addons manager
Created tcp-services.yaml in the kube-system namespace to forward traffic to a RabbitMQ service in the default namespace.
Verified RabbitMQ is accessible inside the minikube cluster via ClusterIP service
Verified that the nginx ingress has loaded the tcp-services.yaml ConfigMap and has a stanza in the nginx.conf file.
When attempting to telnet from host to minikube on the ingress port, get Connection Refused

What you expected to happen:
To be able to connect to a TCP-based service via the Ingress

How to reproduce it (as minimally and precisely as possible):
minikube addons enable ingress
create any tcp-based pod and appropriate service (RabbitMQ, Postgres, MySQL, etc)
add the following configmap to kube-system, substituting port, namespace and service-name

kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: kube-system data: port: "namespace/service-name:port"

Verify the TCP Services are configured with the Ingress Controller:
kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

Attempt to connect to the TCP service: telnet $(minikube ip) <port>

@seborama

This comment has been minimized.

Copy link
Contributor

@seborama seborama commented May 29, 2018

Hi,

Can you supply the following?

  • your tcp-services.yaml file
  • output from minikube status
  • output from minikube service list
  • output from minikube addons list
  • output from kubectl get ingress,services -n=kube-system
  • output from kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

Regards

@justinrcs

This comment has been minimized.

Copy link
Author

@justinrcs justinrcs commented May 30, 2018

Requested info below, thank you.

tcp-services.yaml file

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: kube-system
data:
  5672: "default/rabbitmq:5672"

output from minikube status

minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

output from minikube service list

|-------------|----------------------|--------------------------------|
|  NAMESPACE  |         NAME         |              URL               |
|-------------|----------------------|--------------------------------|
| default     | kubernetes           | No node port                   |
| default     | rabbitmq             | No node port                   |
| kube-system | default-http-backend | http://192.168.99.100:30001    |
| kube-system | kube-dns             | No node port                   |
| kube-system | kubernetes-dashboard | http://192.168.99.100:30000    |
| kube-system | tiller-deploy        | No node port                   |
|-------------|----------------------|--------------------------------|

output from minikube addons list

- addon-manager: enabled
- coredns: disabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- heapster: disabled
- ingress: enabled
- kube-dns: enabled
- metrics-server: disabled
- registry: disabled
- registry-creds: enabled
- storage-provisioner: enabled

output from kubectl get ingress,services -n=kube-system

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
svc/default-http-backend   NodePort    10.102.229.59    <none>        80:30001/TCP    6d
svc/kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   6d
svc/kubernetes-dashboard   NodePort    10.103.255.166   <none>        80:30000/TCP    6d
svc/tiller-deploy          ClusterIP   10.96.254.176    <none>        44134/TCP       6d

output from kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

printing the relevant part for the rabbit ingress:

stream {
	log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
	
	access_log /var/log/nginx/access.log log_stream;
	
	error_log  /var/log/nginx/error.log;
	
	# TCP services
	
	upstream tcp-5672-default-rabbitmq-5672 {
		
		server                  172.17.0.10:5672;
		
	}
	server {
		
		listen                  5672;
		
		listen                  [::]:5672;
		
		proxy_timeout           600s;
		proxy_pass              tcp-5672-default-rabbitmq-5672;
		
	}
	
	# UDP services
	
}
@seborama

This comment has been minimized.

Copy link
Contributor

@seborama seborama commented May 30, 2018

Hi,

You don't have an ingress defined :) The ingress controller is effectively your nginx reverse proxy. You need to define an ingress now to make use of it.

Example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress-path
  namespace: kube-system
  annotations:
    nginx.org/server-snippet: "proxy_ssl_verify off;"
    nginx.ingress.kubernetes.io/rewrite-target: "/"
spec:
  rules:
  - host: your.domain.name
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 80
        path: /dashboard

Then, add an entry in your DNS (or local hosts file, etc) for your.domain.name to point to your minikube ip. You should now be able to browse to http://your.domain.name/dashboard/ from your Mac. You could also define a host ingress if you prefer.

Regards.

@justinrcs

This comment has been minimized.

Copy link
Author

@justinrcs justinrcs commented May 30, 2018

Appreciate the response, but that's not how the TCP/UDP services work with the Ingress Controller. The Ingress sample you have above is for http (or https)-based services.

Assuming Minikube is using the same Ingress Controller via the AddOn as Kubernetes does, here is the documentation for exposing TCP/UDP:
Exposing TCP and UDP services

We have a similar setup in our cloud environments.

Thank you

@seborama

This comment has been minimized.

Copy link
Contributor

@seborama seborama commented May 30, 2018

Hi,

Struggling with it too.

On the minikube VM, after setting up the ingress controller, I can:

$ telnet 172.17.0.5 5672
GET / HTTP/1.1
AMQP	Connection closed by foreign host

Where 172.17.0.5 is the ingress controller and 172.17.0.7 is rabbitmq - so I can also:

$ telnet 172.17.0.7 5672
GET / HTTP/1.1
AMQP	Connection closed by foreign host

So the ingress controller works but I can't access rabbitmq outside the minikube VM. I think this is a limitation of the ingress in K8s. As the doc states, "Ingress does not support TCP or UDP services". The Nginx Ingress controller offers the work around based on ConfigMap but this has its limitations, I suspect. It can expose the port onto the Node but as this isn't a K8s Ingress, you cannot access the port outside the minikube VM - it isn't proxied.

The Helm package suggests these instructions (it didn't seem to work for me though):
Create a proxy server between localhost and the Kubernetes API Server:
kubectl proxy --port 8002

To Access the RabbitMQ AMQP port:

URL : amqp://127.0.0.1:8002/api/v1/proxy/namespaces/default/services/rabbitmq:5672/

To Access the RabbitMQ Management interface:

URL : http://127.0.0.1:8002/api/v1/proxy/namespaces/default/services/rabbitmq:15672/

Some other ideas (long shots):

  • You might be able to install nginx to proxy amqp between the minikube vm and the nginx ingress controller
  • Perhaps nc can do that too
  • Find another lightweight utility that can or write your own.
  • kubectl port-forward $POD_NAME 15672:15672

Please share here if you find a workaround / solution.

Best regards.

@lrvick

This comment has been minimized.

Copy link

@lrvick lrvick commented Jul 15, 2018

Same issue here. If I connect to the nginx-ingress container I can confirm I can reach my container's tcp port through nginx, but it does not get exposed through minikube.

https://github.com/lrvick/weechat-helm/blob/master/templates/ingress.yaml

@edwardhey

This comment has been minimized.

Copy link

@edwardhey edwardhey commented Jul 27, 2018

@juan131 minikube only expose port 80,443,18080 in ingress pod, so you should create another ingress yourself, then forward 5672 to your rabbitmq.

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Dec 18, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@sseide

This comment has been minimized.

Copy link

@sseide sseide commented Dec 18, 2018

Are there any plans to change the current behaviour and let user easily expose additional ports via builtin nginx ingress controller?

And it would be really helpful to have a link to ome ressources describing neccessary steps to deploy a new ingress controller to minikube to archive this feature requested here.

@MerlinPong

This comment has been minimized.

Copy link

@MerlinPong MerlinPong commented Jan 2, 2019

I compare the ingress controller between minikube and the k8s on server.

Find out minikube's ingress without the configuration as below:

spec.template.spec.hostNetwork = true

After I add it, port can be able to forwarded to the vm host.

Hope this can help.

@tstromberg

This comment has been minimized.

Copy link
Contributor

@tstromberg tstromberg commented Jan 24, 2019

I don't know anything about Ingress yet, but would be happy to assist someone with getting this experience improved for all users. Help wanted!

@unixunion

This comment has been minimized.

Copy link

@unixunion unixunion commented Jan 25, 2019

I am running into this same problem, where TCP services are not exposed on the IP the Minikube is running on. Ingress for HTTP works as expected, but not tcp-services defined as ConfigMap. My nginx config contains the relevant tcp mapping. I have also tried applying the 'hostNetwork = true' setting to the pod spec, but no success.

@sahlex

This comment has been minimized.

Copy link

@sahlex sahlex commented Feb 27, 2019

Still not solved it seems. However, I have the same issue. Minikube's nginx ingress-controller does not do the job properly.

Please help.

@skryvets

This comment has been minimized.

Copy link

@skryvets skryvets commented Apr 9, 2019

Update: The link to manifest file is fixed (was broken due to changes made in nginx repo)

Thanks to @MerlinPong I was able to make minikube forward TCP/UDP traffic. Here are the steps:

  1. Disable default ingress addon

    minikube addons disable ingress
    
  2. Deploy standalone nginx ingress controller:

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
    

    Here's a link to the official documentation.

  3. Tweak the deployment of nginx-ingress-controller.
    Make sure it's running:

    $ kubectl get deploy nginx-ingress-controller -n ingress-nginx
    
    $ NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
      nginx-ingress-controller   1/1     1            1           8m39s
    

    Save the output to a file:

    kubectl get deploy nginx-ingress-controller -n ingress-nginx -o yaml>> nginx-ingress-controller.yaml
    

    Edit the file by applying spec.template.spec.hostNetwork = true and apply the configuration.

  4. Use the example provided in the docs -> https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

I also wrote more detailed explanation in the article (but steps are exactly the same as here) -> Exposing TCP and UDP Services via ingress on Minikube

@Igor-lkm

This comment has been minimized.

Copy link

@Igor-lkm Igor-lkm commented Apr 10, 2019

@skryvets @MerlinPong thank you, working as expected :)

@skryvets At the step Save the edited file and apply it by executing I have to use flag --force:

 kubectl apply -f nginx-ingress-controller.yaml  --force
@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented May 15, 2019

This worked for me:

kubectl patch deployment nginx-ingress-controller -n kube-system --patch '{"spec":{"template":{"spec":{"hostNetwork":true}}}}'

kubectl patch configmap tcp-services -n kube-system --patch '{"data":{"3306":"default/mysql-service:3306"}}'

The main issue is that Ingress controller doesn't support tcp services. I really hope to see this supported soon. I have no idea why they ports are limited for Ingress.

This is based on the nginx hackaround https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

Good luck!

@tstromberg tstromberg added the r/2019q2 label May 22, 2019
@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Sep 27, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented Oct 1, 2019

/assign

@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented Oct 1, 2019

@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented Oct 1, 2019

@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented Oct 1, 2019

@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented Oct 1, 2019

Resolved by #5517

@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented Oct 1, 2019

Relates to #5518

@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented Oct 1, 2019

The only other thing I think we could do here other than a tutorial is add a custom patching container that watches changes on the tcp and udp configmaps and updates the ingress deployment so that the pods listen on the appropriate ports. I will leave it up to the project maintainers to determine whether or not this issue can be closed without that capability.

@bygui86

This comment has been minimized.

Copy link

@bygui86 bygui86 commented Oct 2, 2019

I don't know if could help or not.
After NGINX I tried Traefik and it does not work as well. But I don't know how to fix it...

@woodcockjosh

This comment has been minimized.

@bygui86

This comment has been minimized.

Copy link

@bygui86 bygui86 commented Oct 2, 2019

@woodcockjosh thanks a lot! NGINX ingress controller works perfectly right now.
I haven't found a fix for Traefik. Do you have an advice?

@woodcockjosh

This comment has been minimized.

Copy link
Contributor

@woodcockjosh woodcockjosh commented Oct 2, 2019

@bygui86

This comment has been minimized.

Copy link

@bygui86 bygui86 commented Oct 3, 2019

Thanks @woodcockjosh I will try asap and let you know!!

@tstromberg

This comment has been minimized.

Copy link
Contributor

@tstromberg tstromberg commented Oct 14, 2019

@woodcockjosh mentioned that this should be closed now. Please /reopen if that is not the case. Thanks!

@tstromberg tstromberg closed this Oct 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.