Can't use Minikube on VPN #1099
Comments
|
When I switched to the xhyve driver I can start the instance just fine, but I still cannot interact using kubectl, minikube status, minikube dashboard, etc although I can |
|
You should make sure that you are on the latest minikube version This error usually means that docker took too long to start up. I think we've made this faster in the latest version. If the latest version doesn't work, can you run a |
|
I updated minikube, sorry. I forgot to update the OP version. There doesn't seem to be anything wrong with the running vm. I think the problem is related to networking from host to vm. When I disable my VPN I can interact with minikube just fine, but when the VPN is connected I get the following: minikube dashboard just sits there waiting and never terminates.
|
|
Weird. I believe this is an issue with this Cisco VPN Client and has nothing to do with minikube. I'll close the issue and look into it. Thanks for entertaining me. |
|
I've looked more into this and I still think there might be an issue with the networking in minikube. When my VPN client has Tunnel Mode set to Tunnel All I cannot access anything on the minikube instance, but I can Here is the output of |
|
Looks like this is related boot2docker/boot2docker#628 |
|
After hours of searching and troubleshooting I've determined that It has something to do with Cisco AnyConnect VPN client routing traffic poorly. |
|
@blockloop I see the exact same issue. I cannot use minikube when I'm connected to the Cisco AnyConnect VPN 😞. Here are my minukube configs: Here are the logs I see when I start minikube in debug mode. Looks like it just hangs indefinitely when SSH'ing. I see above the exit status 255 when minikube does SSH. Interestingly, I see in the Cisco AnyConnect VPN logs that it flaps every time I start minikube. Is this issue due to boot2docker/boot2docker#628? Is there a work-around? Should we ask Cisco's AnyConnect VPN team to fix this? |
|
Browsing around I've discovered that the Cisco AnyConnect App supposedly wreaks havoc on the iptables. For some reason I can ssh to the machine but I cannot interact with it in the browser or even use |
|
Thanks for the info @blockloop. I'll see if I can setup a split tunnel. |
|
I generated the DART (Diagnostic And Reporting Tool) report from Cisco AnyConnect VPN, and see these errors in |
|
I found a solution for our VPN situation: All the traffic is routed through the VPN tunnel, but there is a proxy for "browsing the internet". The static IP address of the proxy host is e.g. 12.34.56.78 Then I can set these environment to get Hope this helps, at least after adapting for your specific proxy. |
|
is there no solution for this that works at the tcp/ip layer? |
|
for anyone else that come across this, there is a discussion on xhyve (machyve/xhyve#84), and the solution worked for me: https://gist.github.com/mowings/633a16372fb30ee652336c8417091222 |
|
@duncanphillips thanks. I'll give that a try. It's important to note that – according to machyve/xhyve#84 – you have to run this script every time the VPN starts |
|
Just posting here to help anyone else who has this problem.
|
|
@nrichardson-nm I am getting "File Exists" error with that one. |
|
Yep. You have to delete it first if you've ever ran it. (Using the top command first) |
|
@blockloop I ran into this issue as well on OSX Sierra today with Cisco AnyConnect Client. I was able to work around this issue, by using native OSX VPN client, additionally i have to configure a HostOnly network in VMware Fusion, since there currently no way to pass network information to minikube i had to manually switch the network while the VM boots. |
|
@blockloop - I use Cisco AnyConnect as well, and at least my setup AnyConnect routes 192.168.96/19 through it's tunnel - this is clobbering the default minikube network of 192.168.99/24. I tried using the --host-only-cidr flag to use a different network, but either that doesn't work - or I did it wrong in some way (I suspect the later). However - I was able to get this working by manually editing .minikube/machines/minikube/config.json and setting the following: I left all other values in there alone - just modifying the above two entries. With that set, I can start and stop minikube, and start and stop the VPN client, and everything always continues to work. (For this setup I'm using VirtualBox on a mac) Note: my choice of 10.254.254/24 network was completely arbitrary, feel free to use whatever you prefer, as long as it isn't something else (like anyconnect) isn't already laying claim to. |
|
@neilneely I got a new job and don't have AnyConnect anymore so I cannot test, but that looks legit. |
|
I know this is closed but I'm adding this comment to describe how we made this work:
When on the VPN you can set kubectl to use the NAT'ed port:
When off the VPN you can use the normal minikube context:
Apparently you can use the same port forwarding for the MiniKube dashboard too
|
|
@thegridman Awesome solution! However, I spent most of today frustrated that Apparently that's fine. You just have to kill it, then run |
|
For the ones, suffering from AnyConnect and hyperkit, I made an instruction that helped me. I tested basic functions like starting up existing Minikube VM, restarting, SSHing into it, running |
|
For me the comment of @thegridman didn't solve the problem of accessing the This can be worked around by just starting the proxy directly using I'm not sure why minikube fails to verify the dashboard help. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report
Minikube version (use
minikube version): 0.16Environment:
cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualbox AND vmwarefusioncat ~/.minikube/machines/minikube/config.json | grep ISO): file:///Users/bjones/.minikube/cache/iso/minikube-v1.0.6.isoWhat happened:
Creating and using minikube works fine. Open Cisco AnyConnect Secure Mobility Client (VPN client for work) and minikube no longer works.
minikube starthangs,kubectlcommands cannot reach the host.What you expected to happen:
Minikube should work while connected to a VPN
How to reproduce it (as minimally and precisely as possible):
Anything else do we need to know:
I'm also not able to create a minikube with
minikube startwhile I'm on the VPN. When I try to create a minikube instance while connected to a VPN network I get the following errorI've tried using the
--host-only-cidr 10.10.10.1/24to avoid conflicts in IP ranges, but that didn't help.Here is the output of
minikube start -v 7when I have the VPN enabled and trying to connect to an existing minikube instance.The text was updated successfully, but these errors were encountered: