Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to access service running on host OS from inside minikube VM? #2735

Closed
rahulwa opened this issue Apr 17, 2018 · 26 comments
Closed

How to access service running on host OS from inside minikube VM? #2735

rahulwa opened this issue Apr 17, 2018 · 26 comments

Comments

@rahulwa
Copy link

@rahulwa rahulwa commented Apr 17, 2018

Is it possible to access services running on the host from pod created by minikube with hyperkit/xhyve driver? I am especially interested in minikube with hyperkit/xhyve driver on macOS.
Something like

@fkrauthan

This comment has been minimized.

Copy link

@fkrauthan fkrauthan commented Jun 19, 2018

Are there any news on this? I need something similar to be able to create a small development proxy server that we currently use with docker-compose to start some microservices locally while most of the infrastructure runs within Kubernetes.

@rahulwa

This comment has been minimized.

Copy link
Author

@rahulwa rahulwa commented Jun 19, 2018

Docker's Kubernetes implementation has support for this.

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Sep 17, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@tstromberg tstromberg changed the title Access host port(services) from pod How to access service running on host OS from inside minikube VM? Sep 19, 2018
@tstromberg

This comment has been minimized.

Copy link
Contributor

@tstromberg tstromberg commented Sep 19, 2018

Agreed, we should make this easy to do by default across platforms. I'm a little iffy on making it the default due to the possibility of lateral movement from a compromised container, but we should at least make it an easy option.

@tstromberg tstromberg added the top5 label Sep 20, 2018
@tstromberg

This comment has been minimized.

Copy link
Contributor

@tstromberg tstromberg commented Sep 21, 2018

https://medium.com/tarkalabs/proxying-services-into-minikube-8355db0065fd does a pretty good job of explaining one method of forwarding ports from the host OS into minikube. We should do our own, as well.

@gbraad

This comment has been minimized.

Copy link
Contributor

@gbraad gbraad commented Sep 21, 2018

@tstromberg we are actually working on a solution for Minishift that places a small proxy inside the VM, which is also used to overcome the issue with self-signed or custom CA signed certificates. while ssh or sshuttle works, this is not really easy for all platforms...

minishift/minishift#2788 has some of the findings and thoughts on this.

@TBeijen

This comment has been minimized.

Copy link

@TBeijen TBeijen commented Oct 10, 2018

Would a minikube addon be an idea?
Which, when enabled, would introduce a service along the lines of:

apiVersion: v1
kind: Service
metadata:
  name: minikube-host
spec:
    type: ExternalName
    externalName: 192.168.99.1

This is pretty trivial ofc. but having it as addon would have the added value of being an easy to document step, instead of describing how to look up the host ip, setup the service etc.

@tstromberg tstromberg added this to Q4 in Kanban Oct 30, 2018
@tstromberg tstromberg removed this from Q4 in Kanban Oct 30, 2018
@ctoomey

This comment has been minimized.

Copy link

@ctoomey ctoomey commented Nov 30, 2018

https://medium.com/tarkalabs/proxying-services-into-minikube-8355db0065fd does a pretty good job of explaining one method of forwarding ports from the host OS into minikube. We should do our own, as well.

I tried to get this working on my mac and followed the above instructions. I was able to connect out from the minikube VM itself, but not from containers running in it. So I tried the kubernetes provided by Docker for Mac and the docker.for.mac.localhost DNS entry worked perfectly in containers. I don't know if that's seen as a "competitor" for minikube, but making it that easy is at least a higher bar to shoot for.

@gaziqbal

This comment has been minimized.

Copy link

@gaziqbal gaziqbal commented Jan 3, 2019

Thoughts about exposing the host machine's IP via a minikube environment var?

@tstromberg

This comment has been minimized.

Copy link
Contributor

@tstromberg tstromberg commented Jan 24, 2019

@gaziqbal - That sounds like a reasonable place to start, assuming if the IP is routable (it at least is in kvm2). I'd be happy to approve any PR's which implement this. Help wanted!

@ardalanrazavi

This comment has been minimized.

Copy link

@ardalanrazavi ardalanrazavi commented Mar 9, 2019

Any news on this issue?

@ZenSoftware

This comment has been minimized.

Copy link

@ZenSoftware ZenSoftware commented Mar 14, 2019

Would a minikube addon be an idea?
Which, when enabled, would introduce a service along the lines of:

apiVersion: v1
kind: Service
metadata:
  name: minikube-host
spec:
    type: ExternalName
    externalName: 192.168.99.1

This is pretty trivial ofc. but having it as addon would have the added value of being an easy to document step, instead of describing how to look up the host ip, setup the service etc.

This is a wonderful suggestion. There is often a need to connect to a local database running on the host during development. It would make creating tutorials with minikube straightforward if we had some sort of addon like this.

@yzhong52

This comment has been minimized.

Copy link

@yzhong52 yzhong52 commented May 26, 2019

Without needing any add-ons, this is a solution that works for me:

kind: Service
apiVersion: v1
metadata:
  name: minikube-host
spec:
  type: ExternalName
  externalName: minikube.host 

The idea is that instead of specifying the IP, you'll need to use a DNS name, which is what ExternalName service type supports. And all we need to do is adding this line to the etc/hosts file so that it can resolve it:

10.0.2.2        minikube.host

That 10.0.2.2 is for VirtualBox. As mentioned by others in the thread, for hyperkit/xhyve, you may need to use 192.168.99.1, though I haven't tested these drivers.


Edit:

Actually, you can use ip addr or ifconfig to determine the host IP. The IP address under vboxnet1 is the IP that you need to access the host from within a pod. See https://github.com/kubernetes/minikube/blob/master/docs/accessing_etcd.md.

@staticdev

This comment has been minimized.

Copy link

@staticdev staticdev commented May 29, 2019

I am struggling with kvm2 as it has a dynamic host IP. Enabling GatewayPorts makes me able to curl a service from host during my minikube ssh session:

ssh -i $(minikube ssh-key) docker@$(minikube ip) -R 9200:localhost:9200
curl localhost:9200

{
"name" : "ZOym_lv",
"cluster_name" : "log",
"cluster_uuid" : "BGCPx-nXRL-MUVW1pMLD4w",
"version" : {
"number" : "6.2.4",
"build_hash" : "ccec39f",
"build_date" : "2018-04-12T20:37:28.497551Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

But.. when a POD tries to connect to localhost:9200 it can't connect.

I also tried adding this ExternalName Service and curl minikube-host:9200, minikube.host:9200 or even localhost:9200. Didn't make any difference.

$ curl minikube-host:9200
curl: (6) Could not resolve host: minikube-host
$ curl minikube.host:9200
curl: (6) Could not resolve host: minikube.host

Another try was let ssh running and manually editing the curl with minikube ip:

ssh -Ni $(minikube ssh-key) docker@$(minikube ip) -R 9200:localhost:9200` running.
curl 192.168.39.11:9200

It works, the problem is that there is no environment variable or a way TTBOMK to make the POD know its host ip. Maybe adding it to the /etc/hosts on the VM creation could be a way around.

@staticdev

This comment has been minimized.

Copy link

@staticdev staticdev commented Jun 1, 2019

I found another way of accessing the service from the VM. Inside /etc/resolv.conf of the VM there is this nameserver line with the host ip. If you add this to /etc/hosts it can resolve the name:

minikube ssh
sudo su -c 'echo -e "$(cat /etc/resolv.conf | grep nameserver | cut -d\  -f2)\tminikube.host" >> /etc/hosts'

Now you can curl a service with: curl minikube.host:9200

I also enabled GatewayPorts on /etc/ssh/sshd_config, reloaded sshd, added the ExternalName Service and my PODs still can't curl minikube-host. Help needed.

@mati865

This comment has been minimized.

Copy link

@mati865 mati865 commented Jun 5, 2019

FYI kvm2 driver seems to use 192.168.122.1 by default, you can check it via virt-manager.

@staticdev

This comment has been minimized.

Copy link

@staticdev staticdev commented Jun 5, 2019

@mati865 I tried that.. from minikube ssh I can access the host, but not from the PODs.

@mati865

This comment has been minimized.

Copy link

@mati865 mati865 commented Jun 5, 2019

@staticdev it works for me, what driver are you using?

@staticdev

This comment has been minimized.

Copy link

@staticdev staticdev commented Jun 6, 2019

@mati865 I am using kvm2.. sorry actually it works calling the IP directly. But I don't know why it doesn't work using name (/etc/hosts or ExternalName Service).

@stafot

This comment has been minimized.

Copy link

@stafot stafot commented Jun 13, 2019

Having a similar problem with virtualbox driver. It looks like resolving is not working as expected. My service that exposes host inside cluster is the following.

apiVersion: v1
kind: Service
metadata:
  name: minikube-host
spec:
  type: ExternalName
  externalName: 192.168.99.1

Getting the following error:

curl -vvv http://minikube-host:6005
* Rebuilt URL to: http://minikube-host:6005/
* Could not resolve host: minikube-host
* Closing connection 0
curl: (6) Could not resolve host: minikube-host

If I added it manually on pod's /etc/hosts works as expected.

192.168.99.1 minikube-host
 curl -vvv http://minikube-host:6005
* Rebuilt URL to: http://minikube-host:6005/
*   Trying 192.168.99.1...
* TCP_NODELAY set
* Connected to minikube-host (192.168.99.1) port 6005 (#0)
> GET / HTTP/1.1
> Host: minikube-host:6005
> User-Agent: curl/7.52.1
> Accept: */*

So I wonder what am I missing here.
Also cusrls work from minikube to host and from pod to host when I am using 192.168.99.1 as ip and when I am using 10.0.2.2
From pod:

curl -vvv 192.168.99.1:6005
* Rebuilt URL to: 192.168.99.1:6005/
*   Trying 192.168.99.1...
* TCP_NODELAY set
* Connected to 192.168.99.1 (192.168.99.1) port 6005 (#0)
> GET / HTTP/1.1
> Host: 192.168.99.1:6005
> User-Agent: curl/7.52.1
> Accept: */*
 curl -vvv 10.0.2.2:6005
* Rebuilt URL to: 10.0.2.2:6005/
*   Trying 10.0.2.2...
* TCP_NODELAY set
* Connected to 10.0.2.2 (10.0.2.2) port 6005 (#0)
> GET / HTTP/1.1
> Host: 10.0.2.2:6005
> User-Agent: curl/7.52.1
> Accept: */*

From minikube:

 curl -vvv 10.0.2.2:6005
> GET / HTTP/1.1
> Host: 10.0.2.2:6005
> User-Agent: curl/7.60.0
> Accept: */*
> 
curl -vvv 192.168.1.1:6005
^C
$ curl -vvv 192.168.99.1:6005
> GET / HTTP/1.1
> Host: 192.168.99.1:6005
> User-Agent: curl/7.60.0
> Accept: */*

Any ideas would be appreciated.

@teejae

This comment has been minimized.

Copy link
Contributor

@teejae teejae commented Jun 13, 2019

The problem is that you shouldn't be using ExternalName, as that does a lookup in DNS. The solution is to use this:

apiVersion: v1
kind: Service
metadata:
  name: minikube-host
  namespace: default
spec:
  ports:
  - protocol: TCP
    port: 80
---
apiVersion: v1
kind: Endpoints
metadata:
  name: minikube-host
subsets:
  - addresses:
      - ip: ${MINIKUBE_IP}
    ports:
      - port: <port>
@yzhong52

This comment has been minimized.

Copy link

@yzhong52 yzhong52 commented Jun 14, 2019

@teejae what's the ${MINIKUBE_IP} and <port> above? Is that the IP for the VM? And how about port?

@stafot

This comment has been minimized.

Copy link

@stafot stafot commented Jun 14, 2019

@teejae Thanks, initially I tried this but I declared on service also targetPort and was not working. Thanks again for the clarification.
@yzhong52 on minikube ip i believe it will work either with 192.168.99.1 or 10.0.2.2 if you are using virtualbox driver. And on the port use the port on host that your service runs. In my example above 6005

@tstromberg

This comment has been minimized.

Copy link
Contributor

@tstromberg tstromberg commented Sep 20, 2019

Now documented:

https://minikube.sigs.k8s.io/docs/tasks/accessing-host-resources/

Please feel free to improve it.

@tstromberg tstromberg closed this Sep 20, 2019
@dannyharding10

This comment has been minimized.

Copy link

@dannyharding10 dannyharding10 commented Nov 5, 2019

@tstromberg in the linked documentation, it says

Prerequisites
The service running on your host must either be bound to all IP’s (0.0.0.0) and interfaces, or to the IP and interface your VM is bridged against. If the service is bound only to localhost (127.0.0.1), this will not work.

Is this the default setup, and if not how do we "bind the service to all IPs and interfaces"?

@alika

This comment has been minimized.

Copy link

@alika alika commented Nov 27, 2019

@dannyharding10, the "default" setup will be dependent on the service on your host OS you are trying to connect to from inside minikube. For instance postgres will by default only bind to localhost. To make the postgres service "bound to all IP's" you have to:

  • add the line listen_addresses = '*' to postgres.conf
  • add the line host all all 0.0.0.0/0 md5 to pg_hba.conf
  • restart the postgres service

Then you can use the technique (minikube ssh "route -n | grep ^0.0.0.0 | awk '{ print \$2 }'" in the linked documentation to get the IP of the bridge IP. The bridge IP will be the IP you connect to inside minikube.

Note: You should be careful about opening up postgres to all remote IP. A safer way would be to only allow IPs from the minikube environment. For instance inside my minikube environment my bridge ip is 192.168.64.1. I added the line host all all 192.168.64.1/24 md5 to pg_hba.conf.

hope this helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.