Proposal : Use NAT with the Virtualbox Driver to reach containers published port

[dduportal]

Hi all, maintainers and users !

My use case seems quite specific at first. But after discussing with a lot of people at the DockerCon EU 2015, i'm not alone here.
After discussing with @ehazlett and @dgageot, here is the resume idea and a design proposal to address global virtualbox stability and usage.

Do not hesitate to comment, discuss, etc, it's a first "idea shot" !

WHY ? (aka. the "specific" problem)

Running Docker on my company laptops is quite a challenge :

• Windows 7 (cannot upgrade to any other Windows until Q4 2016 maybe) : No HyperV, no VMWare, VirtualBox only.
• McAfee firewall : Creating Windows network interfaces must be done manually outside any tool (docker-machine, vagrant, etc.).
• Cisco VPN : host only interface can be unstable when connecting/disconnecting VPNs.

WHAT ?

Some thought around Virtualbox

Virtualbox is a quite good default solution :

• It free
• It's quite stable (not often a VM is crashing nor kernel-panicing)
• It's easily portable (Mac OS, Win, a lot of Linux distributions), even for advances functions like networking or sharing folders.
• Use local laptop power instead of having a remote system in a cloud/platform you have to manage : it address a lot of "quickstart" use cases

But it has some downsides (life happens !) :

• Additions are strongly needed for host-only network and folder sharing. This lead to spend time "plubing" to keep VM vbox addition drivers in sync with the VirtualBox application on the host.
• At each "major release", you can be sure that host-networking will be broke. It is like a known release behavior we have since 3.x versions (used with vagrant) since 4 years.

What is the problem ?

I want to address here the newtork part.
Host-only is used as default mode and help to ease the docker usage :

• Just one level of port forwarding inside Docker with the "publish" function
• Direct IP access from the host that will help to reach the instance without configuring /etc/hosts or DNS in the host.

But it's somehow very host-intrusive (rights to create a network interface) and quite unstable since monthes at each Virtualbox releases.

On the other side, NAT mode of virtualbox is quite robust and portable.But:

• It is needed to "plumb" a lot to maintain a static list of forwarded ports, and lifecyle of those (like "Crap my local port is not available i need to find a new one which is free to use.. bla bla...).
• It does NOT help at all newcomer to understand network since we have to deal with 2 levels of networks forwarding :
  

=> So : How to enable NAT to enable more stability using Virtualbox without sacrificing docker usability ?

How ?

The proposal

Idea is to use a SOCKS proxy on the host level to contact containers and not having trouble understanding basic ports system.

Socks is :

• Quite standardized : [RFC 1928](https://tools.ietf.org/html/rfc1928]
• Located at the Session level (OSI 5) so below application level
• Supports both TCP and UDP at level 4
• Can be run over an SSH tunnel

Today, running Docker machine require :

• Having a local OpenSSH client
• Start machines in virtualbox with a NAT forwarding used to reach remote instance and dial with it

Proposed lifecyle would be :

• Start the virtualbox machine with no host only interface, only the default eth0 in DHCP mode with a NAT forwarding for SSH
• Once the machine is up, reachable with SSH and provisioned, open a background SSH socks tunnel on a random (and available :) ) port. For example : 54321.
• At the "docker-machine env" action, and points out the value "socks5h://127.0.0.1:54321/" to all "*_proxy" environment variable (during the export DOCKER_HOST" phase).=> All CLI will use this proxy settings to reach the docker containers
• With the help of documentation, we'll provide information to end users to help them configure their working environment.

Some think i'm not sure of (or should be known) :

• I strongly suggest to not let this behavior the default one since it require generally a specific action from the end user. I propose to add a switch to the virtualbox driver like --use-socks-overnat or something like that
• We have to keep in mind that the DNS resolving will be done on the host, while the IP resolving will be delegated to the VM. Example with localhost : localhost resolves on the host to 127.0.0.1, and then the reached 127.0.0.1 will be the "loopback interface" of the VM.
• I suppose that it may break some existing network settings at host level or at least change them. For example, all the Internet traffic of the webbrowser will run inside the VM, goes through the eth0 and go the the VBox virtual routing.

Proof of concept

This PoC is run with the freshly installed docker-toolbox 1.9.1 on a Mac OS El Capitan 10.11 host.

I need to try on my Win7 laptop (but it was with legacy Vagrant boxes so... shoud be OK, i'll comment later)

• Started the "default" machine with the "Docker Quickstart Terminal".
• Ran an nginx container with dynamically published ports :

dadou$ docker run -P -d nginx
38c45a9d00369a51046a20eae277d078294f4bbeb063013c5fcf82d14de42306
dadou$ docker port 38c
443/tcp -> 0.0.0.0:32768
80/tcp -> 0.0.0.0:32769

• Obviously, this curl will fail :

dadou$ curl -I http://localhost:32769
curl: (7) Failed to connect to localhost port 32769: Connection refused

• Trying to fetch the SSH port (with a bit of dynos.) :

dadou$ export MACHINE_NAME=default
dadou$ export SSHPORT=$(grep SSHPort ~/.docker/machine/machines/${MACHINE_NAME}/config.json| awk '{print $2}' | cut -d, -f1)
dadou$ echo $SSHPORT
50859

• Opening the Socks tunnel :

dadou$ ssh -f -N -D 5000 -i ~/.docker/machine/machines/${MACHINE_NAME}/id_rsa -p ${SSHPORT} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null docker@localhost
Warning: Permanently added '[localhost]:50859' (ECDSA) to the list of known hosts.
dadou$ curl 

• Curling the nginx with explicit Socks \o/ :

dadou$ curl -x socks5h://127.0.0.1:5000 -I http://localhost:32769
HTTP/1.1 200 OK
Server: nginx/1.9.7
Date: Sun, 22 Nov 2015 14:18:21 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 17 Nov 2015 15:43:45 GMT
Connection: keep-alive
ETag: "564b4b31-264"
Accept-Ranges: bytes

• Same with environment variables \o/ :

dadou$ ALL_PROXY=socks5h://127.0.0.1:5000 curl -I http://localhost:32769
HTTP/1.1 200 OK
Server: nginx/1.9.7
Date: Sun, 22 Nov 2015 14:22:20 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 17 Nov 2015 15:43:45 GMT
Connection: keep-alive
ETag: "564b4b31-264"
Accept-Ranges: bytes

• Configured Mac OS with the SOCKS proxy, I can reach within Safari, using the VM's eth0 IP (but not localhost, i got strange channel 4: open failed: administratively prohibited: open failedmessages :

So here is the basic idea, I want to hear more advises before trying to implement this.

What do you think ?

[nathanleclaire]

Very interesting proposal (and thank you very much for the level of detail, this use case is something we're adamant about supporting)... I've thought a lot about what to implement in these types of situations and I will follow up in more detail, but FWIW I wanted to point out that your SSH command is probably doable in one step with:

$ docker-machine ssh default -f -N -D 5000

[nathanleclaire]

I had no idea about ssh -D. Very cool option.

[dduportal]

Hello, @nathanleclaire ! Thank you for taking time for a feedback.
BTW, i knew the docker-machine ssh (...) command for running non interactive commande (or connect the machine), but i never read completely the usage page, that's cool to enable the passing of ssh options !

I'm on my way to implement this design proposal, but i'm a slow Go coder :)
I'll open a PR linked here as soon as i have something usable.

[nathanleclaire]

Just out of curiosity, why not start with adding a guide to the docs to explain how to set this up on Windows and on OSX and maybe modify env to have some sort of --use-socks-proxy option that will set DOCKER_HOST to e.g. localhost:2376? (which if I understand correctly should make this workflow "Just Work")

FWIW, I don't really think we want to get Machine into the scope of actually managing the created SSH process though (forking and/or cleaning it up). That needs to be visible to the user somehow.

[nathanleclaire]

Follow up: I tried to connect to Docker on localhost:2376 using the SOCKS proxy (via ALL_PROXY) but I couldn't get it to work. Not sure where the issue resides.

[dduportal]

Hello @nathanleclaire !
Thanks for taking time to point me a path, i had some implementations idea but it will helps a lot !

On the Docker part, I also tried and failed like you and fall on this issue : moby/moby#5989 (Tl;DR : docker client does not handle the socks proxy to reach the Engine, it seems).

I'm likely to test on a modified docker client and let you know, or add a forwarding port for 2375 and 2376 for Docker it does not works :)

[nathanleclaire]

If moby/moby#18373 or something like it were to get merged, I think we could build-in to docker-machine env an option to use SOCKS proxy if available, and we could use this for everything!

[dduportal]

@nathanleclaire WoW ! That is a cool work ! Thanks !

[dduportal]

Just for info : started working with 2 flags :

• --use-socks-proxy as boolean switch
• --socks-proxy-url as string to tune the address

My work is visible here : https://github.com/dduportal/machine (Diff master...dduportal:master )

I'll pull request when it will be functional, with tests and all lint/build passing :)

[dduportal]

Hi !

Just to let you know that i'm waiting for the integration of @nathanleclaire before going more inside :)
I'll try to rebase to latest docker-machine code also asap, to not fall in a rebase-pocalypse :)

[brettdh]

@dduportal This looks super useful. Docker 1.11.0 is out now with SOCKS proxy support; any idea when this might become a PR?

[brettdh]

Also, it seems worth noting that this has drifted a bit from the original scope of the issue. For instance, this would be useful for other drivers than virtualbox - in my case, generic.

It might also be worth considering adding the --use-socks-proxy and --socks-proxy-url options to docker-machine create, which would save the proxy config as part of the machine's settings and ensure that future invocations of env (and others) with that machine would use the previously-specified proxy, without having to specify it again.

Come to think of it... maybe that's what I should be doing over in #3319. Hmm.

@nathanleclaire - thoughts on any of these things? While I was testing the ssh config work at #3319, I was just starting to run into the problem that the remote host doesn't actually seem to have the default docker port (or any I could find, other than ssh) open, so I was going to have to tunnel or use a SOCKS proxy anyway. With docker 1.11.0, I can accomplish that by setting HTTP_PROXY for each command - but this seems like something that docker-machine could make simpler.

Maybe that's what you meant by "we could use this for everything!" above. 😄

[brettdh]

@dduportal, @nathanleclaire
I needed this functionality so as to not be blocked at work right now, so I've implemented a rough attempt at it, which is currently working for my purposes at least. I'll try to clean it up and post a PR soon. It depends on modified versions of #3319 and #3209, so I want to try to untangle the SOCKS stuff from those first.

[dduportal]

Hello ! Happy to see someone interested !

@brettdh , I'm happy to let you implement the idea ! (Since I'm not efficient at writing Go and don't have much time).

Since Docker was going native, I was letting this down, but yeah, in fact this can be pretty useful for remote machines, even outside virtualbox !

Anyway, before closing my issue, I just want to see what @nathanleclaire and @dgageot want to do about splitting, to have atomics implementations (even if they both depend on another):

Another think to know (and understand) to provide a good end-user experience:
I ran in trouble on a project where we packaged a boot2docker OS, and using SOCKS proxy : the boot2docker and the tunneling where stuck during ~10s, then works well for a minute, and froze ~10s, etc.
=> This is due to the fact that SOCKS, when used with v5 which is strongly recommended ( https://curl.haxx.se/docs/knownbugs.html#SOCKS_proxy_connections_are_done ), can also solve domain names remotely. In that case, if the remote SSH Server listens to ipv6 interface (in addition of the ipv4), DNS AAAA requests are done, but not responded from the DNS resolver inside boot2docker.
Sources: http://lists.mindrot.org/pipermail/openssh-bugs/2007-September/005864.html and https://discussions.apple.com/thread/1240242?tstart=0

• Solution I used was to restrict the internal SSHD inside boot2docker to ipv4 (AddressFamily inet instead of the default AddressFamily any). it works well but does not let SSHD aware of ipv6 which seems bad for the future, even in ipv4 NAT mode.
• I do not know if we can make the internal dns resolver to respond to AAAA records ? I tried by adding my custom names within the /etc/hosts of boot2docker but did not solve anything, so i'm thinking that I need to read more documentation, and do more wiresharking :)

[aitorpazos]

Hi, I see this issue has not received much love lately. IMHO the work needed to make docker-machine manage the ssh client process which need to keep the proxy up is not trivial and hard to do within virtualbox driver boundaries.
However, providing a reliable alternative to connect to the docker daemon in environments that prevent connections using the 'Host Only' interface is a low hanging fruit. In master...aitorpazos:master I've added --virtualbox-daemon-forwardparameter which creates a new NAT redirection (besides the one already created for ssh) for the docker daemon so it can be accessed through localhoston the host. A nice side-effect to this is that the url to the daemon won't change no matter what IP VirtualBox assigns to the VM (provided that the port is not in use already).
I'd like to get some feedback before creating the pull request :)

[dduportal]

FWIW I am closing this proposal, since I no longer have this use case.
If anyone want to take ideas and/or the work done: feel free to do it!

[sebinsua]

@aitorpazos Did you ever create a PR for this? I would like to get docker-machine to connect to VirtualBox on a locked-down windows 7 machine with a broken 'Host Only' interface. It seems like your commits give some kind of option which might allow this however I don't fully understand it. Maybe it doesn't allow this -- perhaps you still need a network interface which is 'Host Only' for this to work?

[aitorpazos]

@sebinsua docker-machine SSH into the VirtualBox VM through a VirtualBox NAT that requires no 'Host Only' interface. aitorpazos@048b71a allows the docker daemon on the VM to use the same mechanism, so the 'Host Only' interface is not used (it is still required to access exposed ports though). Didn't create a PR, as I'll need to put extra work to make it PR'able but I will soon as it seems that it will be useful not just for myself.