[WIP] Introduce unix socket proxying for Docker API clients

[n1hility]

Consider this PR a proposal to generate discussion. Should everyone decide this is the direction to go I can clean it up and add tests.

Overview

The main goal of this proposal is to add support for clients (such as testcontainers) that do not support ssh (or if they do (e.g. docker compose), they do not support podman's style of ssh which rightly appends the unix socket path to the URI: docker doesn't support unix domain sockets over ssh, only only supports the usage of the host defined default sock or tcp/tls connections nested under ssh)

To achieve this, this PR introduces a new command podman system unix-proxy, which operates similarly to podman system service, but instead creates a unix domain socket that points to the specified connection entry (e.g. via -c or the default), provided that connection entry is an ssh destination.

Additionally, it modifies docker machine start to automatically launch two child processes (one for rootless, one for rootful), unless --nocompat is specified. It then prints the DOCKER_HOST settings to the screen, so that the user can cut and paste (assumes bash atm).

Additionally, init and start will create a symlink to the root socket. This acquires root privileges via sudo on the first usage (reusing thereafter until moby or some other provider reclaims it). This can be disabled by passing --nolink.

Since the proxy is long-running, there is some logic in this patch to auto-reconnect and retry.

Challenges / Thoughts

• gvproxy is currently noisy, so in the process of waiting for connection availability, which requires polling the connection, it can generate scary looking logs. The machine start code in this patch adds a 1 second sleep to try to hide some of these, but it's marginally effective at suppressing the noise.
• I am interested in discussions around the security impact of this. My thinking is that it is no less secure than what is achievable via podman connection entries, since the files are created with a umask of 1777, as is the case with the podman system service handler
• There seems to be a short delay on the first SSH connections (~1 second), this might be related to /dev/random entropy and key negotiation, but I did not dive into that.
• Windows Support - A future PR will add windows support specifics, and possibly named pipe support since unfortunately moby has not yet added support for Windows AF_UNIX

Example Output (Using compatibility mode)

./bin/darwin/podman machine start
INFO[0000] waiting for clients...
INFO[0000] listening tcp://0.0.0.0:7777
INFO[0000] new connection from to /var/folders/5p/1gyddrwn4cj6w3ccfy8bj4kr0000gs/T/podman/qemu_podman-machine-default.sock
Waiting for VM ...
Waiting on SSH to come up..
Waiting on initial proxy connections..
Machine "podman-machine-default" started successfully!

Podman Clients

Podman clients can now access this machine using standard podman commands.
For example, to run a date command on a rootless container:

podman run ubi8/ubi-micro date

To bind port 80 using a root container:

podman -c podman-machine-default-root run -dt -p 80:80/tcp docker.io/library/httpd

Docker API Clients

Compatibility socket link is present. Docker API clients require no special environment for root containers.

Docker API clients can also access rootless podman with the following environment:

export DOCKER_HOST=unix:///var/folders/5p/1gyddrwn4cj6w3ccfy8bj4kr0000gs/T/podman/podman-machine-default-api.sock

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: n1hility
To complete the pull request process, please assign luap99 after the PR has been reviewed.
You can assign the PR to them by writing /assign @luap99 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[afbjorklund]

Surely this needs to be opt-in, for modern clients. And then enabled on-demand, for the legacy clients ?

Only clients before Docker 18.09 (such as Java docker) need this workaround, the rest can use SSH...

https://docs.docker.com/engine/release-notes/18.09/#18090

Added support for remote connections using SSH

If using the Podman clients instead of the compatibility socket, then these should never be needed.

I had the same issues with gvproxy and waiting on ssh, when trying to add the virtfs mounting feature.

[afbjorklund]

I know that right now CoreOS has a feature to let Docker handle the docker socket, instead of Podman...

So that might be another reason for "opting in", but it still doesn't need tunneling (just other workarounds*)

• uninstall moby-engine

• install podman-docker

[afbjorklund]

I am interested in discussions around the security impact of this.

Note that eventually the VM will have access to all files in your home directory, as per design (inherited from Docker)...

That is, the plan is to mount the home directory from the host under a similar home mountpoint inside the virtual machine.

So the created sockets should probably be restricted to the user. Similar to the podman SSH keys, being stored in ~/.ssh.

[n1hility]

Surely this needs to be opt-in, for modern clients. And then enabled on-demand, for the legacy clients ?

Only clients before Docker 18.09 (such as Java docker) need this workaround, the rest can use SSH...

Last I checked docker could not use podman ssh URLs as it only supported TCP over ssh and not Unix socket tunneling. I will double check this tomorrow.

Although IMHO, as long as the negatives are small defaulting to a mode which maximizes compatibility is a good move when you have a potential mass move of users.

Another approach if it’s decided to disabled by default would be to print the command to cut and paste fo enable it and set DOCKER_HOST.

https://docs.docker.com/engine/release-notes/18.09/#18090

Added support for remote connections using SSH

If using the Podman clients instead of the compatibility socket, then these should never be needed.

Absolutely agree. It’s redundant in that scenario. Although it does open the door to a wider array of clients. For example you can curl on a host with docker running locally but you can’t in the docker machine / remote case, without first doing podman machine ssh

I had the same issues with gvproxy and waiting on ssh, when trying to add the virtfs mounting feature.

Interesting I wonder if (in addition to sending this stuff to the log) there should be an up event that is tied to systemd status.

[afbjorklund]

You can use Docker, as long as you remove the socket path and add the ssh key (to the ssh-agent).

However, if you try this stunt with the new Podman Machine it will be Docker (Moby) Engine answering!

$ podman system connection ls
Name                         Identity                                    URI
podman-machine-default*      /home/anders/.ssh/podman-machine-default  ssh://core@localhost:33475/run/user/1000/podman/podman.sock
podman-machine-default-root  /home/anders/.ssh/podman-machine-default  ssh://root@localhost:33475/run/podman/podman.sock
$ export DOCKER_HOST=ssh://root@localhost:33475
$ docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Sun Aug 15 00:00:00 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.5
  GitCommit:        
 runc:
  Version:          1.0.1
  GitCommit:        06fb178
 docker-init:
  Version:          0.19.0
  GitCommit:        

You might have expected a Podman service, rather than the Docker daemon ?

But for that, you need to hack CoreOS (as per above) to replace docker.sock

[n1hility]

You can use Docker, as long as you remove the socket path and add the ssh key (to the ssh-agent).

However, if you try this stunt with the new Podman Machine it will be Docker Engine answering
You can use Docker, as long as you remove the socket path and add the ssh key (to the ssh-agent).

However, if you try this stunt with the new Podman Machine it will be Docker (Moby) Engine answering!

$ podman system connection ls
Name                         Identity                                    URI
podman-machine-default*      /home/anders/.ssh/podman-machine-default  ssh://core@localhost:33475/run/user/1000/podman/podman.sock
podman-machine-default-root  /home/anders/.ssh/podman-machine-default  ssh://root@localhost:33475/run/podman/podman.sock
$ export DOCKER_HOST=ssh://root@localhost:33475
$ docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Sun Aug 15 00:00:00 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.5
  GitCommit:        
 runc:
  Version:          1.0.1
  GitCommit:        06fb178
 docker-init:
  Version:          0.19.0
  GitCommit:        

You might have expected a Podman service, rather than the Docker daemon ?

But for that, you need to hack CoreOS (as per above) to replace docker.sock

Ah yes they support their sock location only just not custom ones. Then the next question for the user doing that is does that point to rootless or rootful.

[afbjorklund]

But for that, you need to hack CoreOS (as per above) to replace docker.sock

Ah yes they support their sock location only just not custom ones. Then the next question for the user doing that is does that point to rootless or rootful.

Adding rootless to the mix as well just makes it worse, but it's basically the same circus in /run/user/1000.

The quick hack is sudo ln -sf /run/podman/podman.sock /var/run/docker.sock, the long way is rpm-ostree

$ podman machine ssh sudo ln -sf /run/podman/podman.sock /var/run/docker.sock
Could not create directory '/root/.ssh'.
Warning: Permanently added '[localhost]:33475' (ECDSA) to the list of known hosts.
$ docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.40
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: linux/amd64/fedora-34
 Podman Engine:
  Version:          3.3.1
  APIVersion:       3.3.1
  Arch:             amd64
  BuildTime:        2021-08-30T20:46:36Z
  Experimental:     true
  GitCommit:        
  GoVersion:        go1.16.6
  KernelVersion:    5.13.13-200.fc34.x86_64
  MinAPIVersion:    3.1.0
  Os:               linux
 Engine:
  Version:          3.3.1
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.16.6
  Git commit:       
  Built:            Mon Aug 30 20:46:36 2021
  OS/Arch:          linux/amd64
  Experimental:     true

This is because it only allows modifications in /var, but not any changes in /usr (and there is no yum/dnf tool).

https://developers.redhat.com/blog/2020/03/12/how-to-customize-fedora-coreos-for-dedicated-workloads-with-ostree

[rhatdan]

We should change podman machine init to disable moby sockets by default and install docker.sock by default within the VM.
Podman machine should not be running docker at all within the VM,(In a perfect world we convince Fedora CoreOS to replalce Moby with podman-docker.

[rhatdan]

I would want a --compat flag on podman machine start, which would tell it to listen in the standard locations where Docker listens. So the Docker clients would just work. I would guess this would be with a rootfull podman.

[rhatdan]

@ashley-cui @baude @jwhonce @flouthoc WDYT?

[afbjorklund]

Doing explicit tunneling of the unix sockets with ssh should be a usable workaround, in the meantime ?
(while waiting for Docker to support paths in their host urls, or CoreOS to stop supporting docker engine)

Some runtimes even prefer tunneling, for instance using containerd / buildkitd are quite horrible to use:
https://minikube.sigs.k8s.io/docs/handbook/pushing/#6-pushing-directly-to-in-cluster-containerd-buildkitd

Lima works around this by running ssh nerdctl, instead of remoting.

Minikube has a similar workaround, instead of docker-env/podman-env

[afbjorklund]

And if one really really wants to, it is also possible to install rootless docker using good ole curlpipesh.

$ podman machine ssh
[core@localhost ~]$ curl -fsSL https://get.docker.com/rootless | sh
...
[INFO] Creating /var/home/core/.config/systemd/user/docker.service
[INFO] starting systemd service docker.service
...
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger core`

[INFO] Creating CLI context "rootless"
Successfully created context "rootless"

[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):

export PATH=/var/home/core/bin:$PATH
export DOCKER_HOST=unix:///run/user/1000/docker.sock

https://docs.docker.com/engine/security/rootless/

Not sure what would be the point of having 4 different daemons running, but that's another story...

[n1hility]

I would want a --compat flag on podman machine start, which would tell it to listen in the standard locations where Docker listens. So the Docker clients would just work. I would guess this would be with a rootfull podman.

Good idea I’ll add that to the draft

[afbjorklund]

Totally unrelated, but trying to run rootless docker just results in some epic fail.

$ export DOCKER_HOST=ssh://core@localhost:42759
$ docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Cannot connect to the Docker daemon at http://docker.example.com. Is the docker daemon running?

Hmm, nice url there. 😀 Wonder if the debugging logs has any more input...

DEBU[0000] commandconn: starting ssh with [-l core -p 42759 -- localhost docker system dial-stdio] 
DEBU[0000] commandconn (ssh):failed to open the raw stream connection: dial unix /var/run/docker.sock: connect: permission denied 
DEBU[0000] commandconn: starting ssh with [-l core -p 42759 -- localhost docker system dial-stdio] 
DEBU[0000] commandconn (ssh):failed to open the raw stream connection: dial unix /var/run/docker.sock: connect: permission denied 
DEBU[0000] commandconn: starting ssh with [-l core -p 42759 -- localhost docker system dial-stdio] 
DEBU[0000] commandconn (ssh):failed to open the raw stream connection: dial unix /var/run/docker.sock: connect: permission denied 
DEBU[0000] FIXME: Got an status-code for which error does not match any expected type!!!: -1  module=api status_code=-1

---

Apparently one is supposed to tweak the login shell, so that it sets the env var.

https://docs.docker.com/engine/security/rootless/#expose-docker-api-socket-through-ssh

To expose the Docker API socket through SSH, you need to make sure $DOCKER_HOST is set on the remote host.

So that is how you are supposed to make rootless connections work, you change the ~/.bashrc

[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):

export PATH=/var/home/core/bin:$PATH
export DOCKER_HOST=unix:///run/user/1000/docker.sock

[core@localhost ~]$ cat .bashrc.d/docker 
export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock

And now docker can use the rootless podman socket, by choosing the other connection.

export CONTAINER_HOST=ssh://core@localhost:42759/run/user/1000/podman/podman.sock
export DOCKER_HOST=ssh://core@localhost:42759

Similar can be done for root, to make it run podman instead of the default docker socket:

[root@localhost ~]# cat .bashrc.d/docker 
export DOCKER_HOST=unix:///run/podman/podman.sock

export CONTAINER_HOST=ssh://root@localhost:42759/run/podman/podman.sock
export DOCKER_HOST=ssh://root@localhost:42759

Which means the ssh tunneling workaround is not needed anymore, for Docker 18.09+

Only these two files need adding:

/var/home/core/.bashrc.d/docker
/var/roothome/.bashrc.d/docker

Legacy clients that don't support ssh: but only unix: and tcp: still needs the workaround.

[bsideup]

Only clients before Docker 18.09 (such as Java docker) need this workaround, the rest can use SSH...

Note that Docker 18.09 has nothing to do with SSH protocol support in Docker clients in languages other than Golang. Each client library (and docker-java is one of them) needs to support SSH transport, and this is not safe to assume that every other client can use SSH.
AFAIK https://github.com/dotnet/Docker.DotNet is another very popular Docker client library that does not support SSH transport.
Also softprops/shiplift#291 (Rust)
Also upserve/docker-api#447 (Ruby)
No SSH here as well: https://github.com/proger/erldocker (Erlang)

So, it looks like, "the rest" is so far Golang/Python/NodeJS, while other clients are lagging behind :)

Just my 2c, and thank you @n1hility for addressing this! 👍

[afbjorklund]

Note that Docker 18.09 has nothing to do with SSH protocol support in Docker clients in languages other than Golang. Each client library (and docker-java is one of them) needs to support SSH transport, and this is not safe to assume that every other client can use SSH.

Seems more safe to assume that they do not support SSH, and that most of them are using the legacy HTTPS (2376) still...

(Note that legacy varlink and legacy http are not supported in podman, but this ssh tunneling of unix should work in most places)

But yeah, guess I meant "CLI client" there. 🙂

Good to know that Go and Python are on par.

[matejvasek]

Speaking of Go client: I belive that standard Go CLI client does support ssh however the Go client library doesn't.

[afbjorklund]

So only Python SDK then. 😀

Both (CLI and Python) also require that docker (client) is installed on the server, even if you don't have to install dockerd (server) on the server when running Podman

[rhatdan]

Theoretically we would need to implement this in the podman cli for docker emulation?

[afbjorklund]

Seems that way, there was a reference in the Moby code

https://github.com/moby/moby/blob/master/client/client.go#L286

[afbjorklund]

For some reason, they lost the path support in the urls on the way - original PR did use it, but it doesn't work in what got merged...

[matejvasek]

Having something like podman system unix-proxy unix:///tmp/podman.sock seems like viable solution, just user needs to know about such a sub-command.

[matejvasek]

@afbjorklund I am little bit lost in discussion here (I started looking at in just now):

For some reason, they lost the path support in the urls on the way - original PR did use it, but it doesn't work in what got merged...

what do you mean by that

[afbjorklund]

Currently DOCKER_HOST fails, if you include a socket path

[matejvasek]

even if moby supported path part it sill requires additional binary to be present on server (basically socat), also as you mentioned we cannot expect that every docker API client support ssh.

[n1hility]

Question for the pile:

Should I add graphical dialog auth support on MacOS when --compat needs to create the link and there is no tty (e.g. launched from some sort of launcher)? This would support biometric and watch auth. I don't prefer to do dialogs in terminals generally since the user can lose their terminal window when they have a bunch open, so I didn't do it this way originally.

[n1hility]

Seems that way, there was a reference in the Moby code

https://github.com/moby/moby/blob/master/client/client.go#L286

@afbjorklund Just to clarify this point (and not to argue we should not add dial-stdio support). This is only required for podman to support docker clients that use DOCKER_HOST to point to SSH of a podman system. If they point it to a unix socket (e.g. with the proxy, or the default with the compatibility unix socket) this is not required. Is that accurate?

[mheon]

@ashley-cui @baude PTAL

[afbjorklund]

Is that accurate?

That seems accurate, so in that sense it is somewhat off-topic. The unix socket is still needed for other clients.

But the ssh: syntax was added in 2018, in order to not have to set up manual SSH tunnels, that is the relation.

[Luap99]

Why do we need the proxy logic inside podman, this could use ssh to proxy a socket, e.g. ssh -nNT -L/local/path/podman.sock:/run/user/1000/podman/podman.sock ssh://user@host

[matejvasek]

@Luap99 I believe that having it automatically set up by podman machine is much more convenient to user.

[matejvasek]

Otherwise there would be each week new issue from some new user asking how to use docker client with podman VM, we will have to explain ssh tunneling all over again.

[Luap99]

Podman machine can still do this automatically by calling the ssh command. This commit add a lot of logic just to create proxy.

[matejvasek]

Ah I see, well there is already ssh logic for libpod so why not to use it?

[matejvasek]

Or you mean logic for unix socket serving?

[n1hility]

Why do we need the proxy logic inside podman, this could use ssh to proxy a socket, e.g. ssh -nNT -L/local/path/podman.sock:/run/user/1000/podman/podman.sock ssh://user@host

Good question. The benefits of this approach are:

1. Improved Compatibility - we don't have to care what version of ssh is on the box, what bugs it has, what user configuration it has, where it's located, etc
2. Consistency - As @matejvasek mentions this reuses the connection binding ssh client, so 100% equivalent behavior. We don't have to keep redundant sections of code in sync (making sure the ssh command line policy + whatever defaults it currently has semantically match the connection code)
3. Improved Robustness - If the ssh connection drops for whatever reason (vm pause, service timeout etc) we can hold the line with the unix socket client while reestablishing the connection on the backend.
4. Improved Error Reporting - Since we control all interconnection points, we get a richer interface with error conditions and can translate them appropriately (vs say parsing an ssh log)

From a cost perspective, since the ssh client code is reused it amounts to just the unix socket handling, which is just a basic accept loop and two trivial io copy goroutines to form a pipe.

[n1hility]

FYI I split the last commit off into a separate PR since it's orthogonal. (#11703)

[Luap99]

@n1hility Thanks for the detailed answer, this SGTM.

[n1hility]

PR rebased against the commit moved to the other PR.

I will include a test and doc for unix-proxy as the next step and reduce some of the verbosity.

[n1hility]

So thinking about this a little bit this morning and I think I should move the docker.sock creation and auth to init and warn on start. It could then be on by default prompt the user if the file exists and points elsewhere to ask if sure and overwrite. Then start could warn to re-run init to fix the link if docker takes it back.

[n1hility]

PR is updated to create the symlinks on init by default. Options were added to disable the proxies/sockets and the link if so desired.

[n1hility]

Based on some recent discussions we are looking at putting this logic in gvproxy to avoid having extra daemon processes. An initial draft proposal of this is here: containers/gvisor-tap-vsock#58

Closing this PR since it will be superseded.