Cannot access docker when running VPN (Cisco AnyConnect) #628
Comments
|
I don't know if this helps but running |
|
Sorry to hear my stuff blows up things. I learned today also that my scripts don't work with docker 1.3.x as security has been added. There may also be a change that my crude way of adding a host-only adapter may to VirtualBox may break things for you. For the VPN stuff, I run this script (pre-docker 1.3.x but I presume it still works): |
|
@frosenberg I don't blame you, I had a feeling it wouldn't work with 1.3.0 but gave it a try anyways. I figured I'd be able to undo it all anyway, but I didn't count on weird issues with destroying my boot2docker image and recreating it. The only cleanup I've done on my Mac was to remove the route table entry you add. I ran |
|
Ok, I ran the uninstall script then re-ran the mac installer. Now, I'm getting this: Any thoughts on why I'm getting the timeouts while running docker commands from my mac terminal? |
|
Issue #392 solved my problem. Looks like the host only network was messed up. This comment's instructions fixed things. I'll have to give @frosenberg 's |
|
#392 (comment) had the steps that finally worked for me: adding a port forward and then pointing to |
|
This fix worked for me running the cisco vpn. I'm also experimenting with kitematic which is awesome. But still needs some work if you use a lot of parameters to run docker containers. For simpler uses, it's fantastic. |
|
Cisco AnyConnect messes with your machine's routes. I've done this as a workaround: |
|
Does anybody have a final resolution to resolve this issue? I'm using the Yosemite + Anyconnect. As ipfw is removed from Yosemiti. It cause the script that @frosenberg provided doesn't work. Thanks a lot :-) |
|
Using openconnect instead of Anyconnect, I could connect to containers in Kitematic. |
|
Setting up port forwarding in Virtual Box worked for me, as described here: #628 (comment) (thanks @johnnyt) export DOCKER_HOST=tcp://127.0.0.1:2376 |
|
@chulkilee, openconnect fixed my problem. Thank you. I hate AnyConnect. |
|
Thanks @johnnyt - I am using Yosemite with docker 1.6 with Cisco AnyConnect. Changing the docker host to 127.0.0.1 and adding the port forwarding worked for me. |
|
So last week I installed the new set of docker tools including docker-machine. I also installed a new version of Cisco AnyConnect (4.1.00028). Things are working without any problems for me at the moment. Before, after and while on the VPN connection. |
|
Hi, |
|
Anyone yet found a fix for this with |
|
I got it working yesterday with this flow (Cisco VPN Client and Win VirtualBox):
4 Turn on VPN But today after Win boot it could not connect to that VM. So I had to remove it and recreated it... |
|
@atomantic i was able to get docker-machine to work using @johnnyt solution with a few changes. |
|
@joshskinner's solution appears to have been the key in a Windows 7 environment with docker-machine and Cisco AnyConnect. Setting the port-forward IP to the docker-machine IP rather than 127.0.0.1 worked around certificate issues that were present with @johnnyt's solution. As stated by @Kalle80, be sure to create the VM before connecting to VPN. Also, once having run AnyConnect it was necessary to reboot before AnyConnect was truly shutdown, even after killing the visible Cisco services. |
|
I hacked my way through a similar solution some time back, but never had time to automate it. It was working fine until I got bitten by the certificate issue that @rickpeters mentioned. I just put together a small helper script to re-apply the fixes https://github.com/onejli/docker-vpn-helper. The script will:
In its current state (with more than a few TODOs), the helper script assumes that you're creating (or using) a VM named After running the helper script, you'll be able to:
Cisco AnyConnect removes/redirects routes upon connection, but doesn't restore them after disconnecting. This seems to make the VirtualBox network kernel modules very unhappy. After dropping off of VPN, VirtualBox is able to add host-only network adapters, but it is NOT able to add the routes needed to connect them. I stumbled across this thread and found a solution in the last post that I integrated it into my helper script. @daagar There's no need to reboot after disconnecting from AnyConnect. You just need to:
|
|
Thanks @onejli !! This has been holding me back for months. Finally something worked. |
|
@onejli , we have moved on beyond this issue :-) Second part of the solution is that sometimes we create a really transparant tunnel from the osx host (using sshuttle) to the apache container and the vpn container and just tunnel the complete 10.0.0.0/8 range of addresses through the ssh tunnel. Also works great, but is sometimes a bit slower. The big advantage is that my local mac is not touched by the vpn at all and everything works (and also all docker tools) work like a charm. Even a local Docker swarm is not a problem anymore :-) grtz, |
|
@rickpeters I wish I could run my VPN client from within a VM or container 😢. Unfortunately, there are some corporate security rules that prohibit us from going down this road. @bfarrell Happy to help! Just keep in mind that this solution is more of a band-aid. Unfortunately, it doesn't help when you want to expose a port from within a container to the physical host. A host-only adapter would normally allow you to access any port mapped from a docker container. Due to VPN crippling communication between the physical host and the VM over the host-only adapter, you'll need to manually insert a port forwarding rule over the NAT interface for each container port that you want to expose. |
|
@onejli I think we have the same corporate security rules ;-) However my take on this is that since BYOD is possible and even corporate laptops are able to function on a normal internet connection (and allowed to do so), there is really no big risk in using Docker as a vpn tunnel in this way :-) |
|
I installed Cisco Anyconnect 4.2 and it fixed my Docker issues: I could use Docker (1.9.1) while connected to the VPN, but as soon as I was disconnected I couldn't use docker anymore, I had to add the route again manually. |
|
Count me in the list that havent received a response. |
|
docker/machine#2632 ssh port forwarding fits the bill for me for now |
|
@mchiang0610 I am sending you a mail with this... Thanks! |
|
has anyone been able to make this work with Docker for Mac beta? |
|
Yes, works as documented. Put it in VPN compatibility mode start your VPN and use the address of the docker daemon VM that's in |
|
It will just bind to local host I imagine instead of using host-only On Wed, Apr 13, 2016, 1:41 AM Rick Peters notifications@github.com wrote:
|
|
Don't know the real magic. However to me it looks mostly like it uses a special IP address that your VPN leaves alone. Furthermore if I understand correctly you are not (yet) able to expose ports to your host machine itself. So the main part is that the docker daemon will stay available while you are on your VPN connection. Also, the docker.local alias for addressing the docker VM does not (yet) work when in VPN compatibility mode. |
|
Can the containers communicate together still? Will host be able to act as On Wed, Apr 13, 2016, 2:37 PM Rick Peters notifications@github.com wrote:
|
|
My situation was that I was that as soon as I enabled my VPN (using openconnect), I was no longer able to reach my containers. I took a look at the routes on my host and noticed that there was a duplicate route So I deleted the extraneous route. I didnt want docker traffic going over the VPN, so that is the route I removed Now I'm able to ping my containers, and my containers have network access again It seems that the important thing is that there is a route on your host going to the docker interface and that there is not another route ahead of it. If you are missing the route, simply add it and you should be back in business Of course, this needs to be run each time you reconnect to your VPN, but its easily scripted. |
|
Thoughts if i wanted to hit an internal private docker repo, over VPN? |
|
Can you still hit the repo from your docker host? If you can, then I think the container should be able to hit the repo as well, since its traffic should go from the container over the docker0 interface, then the docker host should route the data to the repo over vpn0 |
|
There is another workaround for this frustrating VPN problem: you can talk to the Docker machine (boot2docker VM) through an (emulated) serial console. I didn't see it documented anywhere, but I noticed in boot2docker's Dockerfile that boot2docker does indeed start up serial consoles on /dev/ttyS0 (COM1) and /dev/ttyS1 (COM2) if those serial ports are available. I tried it with one emulated serial port and it worked. Instructions for Windows are described below. The instructions will be somewhat different for Linux and Mac environments, but setting up emulated serial connections between a host computer (regardless of the OS) and a VM on VirtualBox is well-documented on many sites. To enable emulated serial ports in your Docker machine, shut it down first, then open the VirtualBox GUI, right-click on your Docker machine VM, select Settings... -> select list item "Serial Ports" -> select tab "Port 1" (if not already selected). Configure the settings under the tab as follows (assuming Windows, it will be somewhat different on Linux or Mac hosts):
Now start the Docker machine (boot2docker) VM up again. Do this preferably through the "docker-machine" command, unless of course you're already running a VPN connection, in which case you'll have little choice but to start the VM through VirtualBox, through either the GUI or the VBoxManage command. To connect to the emulated serial port on the now running VM, use a console application that can initiate serial connections to host pipes. I recommend PuTTY for this. In the case of PuTTY, you can connect to the emulated serial port by starting PuTTY and creating a PuTTY session with the following settings/parameters:
You should now have regained control over your Docker machine. And since you are now controlling it over a serial connection (albeit an emulated one), the connection will not be disrupted by any changes in your host computer's TCP/IP configuration. You can continue to enter Docker commands and manage your container instances, regardless of whether you are connected through a VPN or not. Access to the outside Internet from within the Docker machine should also still be possible, since the NAT interface managed by VirtualBox should also remain unaffected by any changes on the host machine, except for possible proxy server issues on the VPN of course. But you can work around those as well by configuring the http_proxy and https_proxy environment variables on the Docker machine and inside your container instances. WARNING: when you disconnect the serial console by closing it (without entering the command "exit"), you will still be logged in the next time you connect to the same serial console. Make sure no one else has access to the user session on your computer. You should still be able to access any local files in your home folder(s) through the path /c/Users if you created the docker VM using the standard docker-machine settings. If you need access to any other local folders, you will have to stop the docker machine VM and add it as a shared folder using the VirtualBox VM and then restart the VM and mount the newly added shared volume. By the way, it would be a very practical and convenient enhancement if the docker-machine client tool could be improved to automatically fall back to an emulated serial connection to the boot2docker VM, whenever SSH connections to it failed. Lastly, it might be a good idea to simply create a separate "client VM" in VirtualBox (alongside your Docker machine) and run a light-weight OS with a GUI and a web browser in it. I'd recommend Lubuntu in Live CD mode for this, but you could for instance also use one of the Windows/IE VMs that Microsoft has officially made available for testing purposes, in case you need to test anything with Internet Explorer. The client VM can then serve to both control and test the Docker machine. To virtually "connect" it to the Docker machine, add a host-only network interface to this new client VM and connect it to the same host-only adapter that is already connected to the Docker machine. This way, the new client VM will be able to access the Docker machine directly. And since the connection between these two VMs will be taking place over an internal virtual "network switch" managed by VirtualBox, this connection will remain in tact, even once the host machine no longer has access to it due to changed routes by a VPN client. You will then be able to continue accessing and testing your docker container instances on the Docker machine by accessing it from the new client VM. In addition, you could also add a second network interface in NAT mode to the client VM, so you can access both the Docker machine and the outside internet (again, taking account any possible proxy issues that pop up while connecting to a corporate VPN). I hope these suggestions help! 😃 |
not on virtual box windows must be "//./pipe/docker_engine" (without the quotes) (the default for kinematic) |
|
Due to this issue in Cisco AnyConnect VPN client version Interestingly, I see in the Cisco AnyConnect VPN logs that it flaps every time I start minikube. |
|
I generated the DART (Diagnostic And Reporting Tool) report from Cisco AnyConnect VPN, and see these errors in |
|
This should do all the machinery required for setting up docker-machine with local port forwarding: https://github.com/onejli/docker-vpn-helper. Plus explains very well where problems are in using docker-machine with a VPN that intercepts all the traffic. |
|
Thank you!! @volkertb |
|
@sarusso - Do you have a solution for windows users? |
|
The same code with minor adjustments worked for me even on Windows (with Bash). But I could not find the time to cleanly publish it here on GitHub yet.. |
|
@sarusso Would be interested in those 'minor adjustments' for windows user! |
|
I haven't read completely through this thread yet, but has anyone here tried to set "Allow Local (LAN) access when using VPN (if configured)" within the Cisco AnyConnect VPN client? I know that in Windows 10, this was the smoking bullet. My symptoms were simply that I could not mount a vol when AnyConnect was connected to my company VPN - even with the local firewall disabled. Enabling this setting within the AnyConnect client fixed the issue immediately. |
|
That option doesn't seem to be available on the Mac OS X version of the client. |
wglambert
added
the
question
|
that option is also not available on the Cisco AnyConnect Secure Mobility Client. |
For me the only changes to make https://github.com/onejli/docker-vpn-helper work on Windows (in MinGW shell that is installed with Docker Toolbox) are to set full path to VBoxManage ("C:\Program Files\Oracle\VirtualBox\VBoxManage.exe"), put additional slash before /CN=localhost, i.e. openssl req -subj "//CN=localhost"... and change path to certs as below |
and others
I could not get boot2docker to work while running the Cisco AnyConnect VPN client. I did not record the console output when I encountered the error, when I see it again then I will post it.
In my efforts to fix it I found a solution by @frosenberg in his blog post: http://www.devopslife.com/2014/08/08/docker-boot2docker-and-dns-resolution-of-containers.html
I ran his
enable-docker-dns.shscript which failed to work with docker 1.3.0. I modified the port 2375 to 2376 which also failed to work. This rendered my boot2docker vm unreachable.To fix the problem I tried running:
I tried the password tcuser as documented at https://docs.docker.com/installation/mac/ but that does not work. Any thoughts on what I need to do to get boot2docker working correctly again?
Also, I would have expected the images I had downloaded to be blown away with
boot2docker destroy. I checked and the~/VirtualBox\ VMs/boot2docker-vm/directory is removed along with theboot2docker-vm.vmdkdisk image. Why are those still hanging around?The text was updated successfully, but these errors were encountered: