docker exec -i as a transport for rsync -e

[peter-leonov]

Description of problem:

Trying to use docker CLI as an ssh replacement for rsync; and something goes wrong with a low level error not related to rsync neither docker exec but a tunnel between them (looks like so).

docker version:

Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): darwin/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

docker info:

Containers: 9
Images: 69
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 87
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 4.0.3-boot2docker
Operating System: Boot2Docker 1.6.2 (TCL 5.4); master : 4534e65 - Wed May 13 21:24:28 UTC 2015
CPUs: 4
Total Memory: 1.957 GiB
Name: inshaker-dev
ID: EQXF:E5EP:KYI6:7XYA:MVGO:WDWJ:WV6S:KYVV:3XRO:6LQE:C66D:E5WM
Debug mode (server): true
Debug mode (client): true
Fds: 25
Goroutines: 51
System Time: Tue Jun  2 10:40:17 UTC 2015
EventsListeners: 0
Init SHA1: 7f9c6798b022e64f04d2aff8c75cbf38a2779493
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Username: peterleonov
Registry: [https://index.docker.io/v1/]
Labels:
 provider=virtualbox

uname -a:

Host:

Darwin pl.local 13.4.0 Darwin Kernel Version 13.4.0: Wed Mar 18 16:20:14 PDT 2015; root:xnu-2422.115.14~1/RELEASE_X86_64 x86_64

Docker VM:

Linux inshaker-dev 4.0.3-boot2docker #1 SMP Wed May 13 20:54:49 UTC 2015 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

The docker machine is an up to date boot2docker image run in VirtualBox.

How reproducible:

Try to run rsync -e 'docker exec -i' /

Steps to Reproduce:

1. docker run --name test1 -it debian:jessie bash -c 'apt-get update && apt-get install -y rsync && bash'
2. rsync -r -e 'docker exec -i' -vv test1:/bin .
3. find ./bin

Actual Results:

opening connection using: docker exec -i test1 rsync --server --sender -vvre.iLs . /bin  (10 args)
receiving incremental file list
delta-transmission enabled
skipping non-regular file "bin/dnsdomainname"
skipping non-regular file "bin/domainname"
skipping non-regular file "bin/nisdomainname"
skipping non-regular file "bin/pidof"
skipping non-regular file "bin/rbash"
skipping non-regular file "bin/sh"
skipping non-regular file "bin/sh.distrib"
skipping non-regular file "bin/systemd"
skipping non-regular file "bin/ypdomainname"
time="2015-06-02T14:01:34+03:00" level=fatal msg="write /dev/stdout: resource temporarily unavailable" 
bin/bash
rsync: connection unexpectedly closed (9778 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.0]
rsync: connection unexpectedly closed (1594 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.0]

and an empty local ./bin directory.

Expected Results:

No error messages :) and a recursive copy of the test1 container /bin directory.

Additional info:

Note that time="2015-06-02T14:01:34+03:00" level=fatal msg="write /dev/stdout: resource temporarily unavailable" error. And for really small directory (for ex. an empty `/home/) rsync will do well but it hangs at the end (maybe related to #13052).

Container rsync --version

rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

CLI rsync --version:

rsync  version 3.1.0  protocol version 31
Copyright (C) 1996-2013 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags

[peter-leonov]

rsync -e 'docker run --rm -a stdin -a stdout -a stderr' -rv ubuntu:/bin bin with or w/o -i does not do the trick neither.

[icy]

@peter-leonov How is it going? I can solve the problem with Docker-1.8.3 and nsenter (https://github.com/jpetazzo/nsenter/issues/67)

[samatwork]

I am still experiencing problems with this in Docker 1.9.1. My environment is very simple, just two hosts running Docker Engine on the same network. They're both CentOS 7, identical patch levels, same image on both hosts. Not using Compose, Swarm, Network or anything else, just manually-managed containers.

Both hosts use the same internal CA to listen on TLS. Docker Engine and rsync are installed inside the image, as is the CA cert, so the docker CLI works fine from inside the container. From server A, I start bash in container A and try to use rsync to copy files from container B on server B. For a very small amount of data, I see this:

[root@serverA:/home/samatwork]$ docker exec -it containerA /bin/bash
[root@containerA srv]# mkdir home/tmp2
[root@containerA srv]# cd home/tmp2
[root@containerA tmp2]# rsync -e "docker -H serverB:2376 --tls exec -i" -av containerB:/srv/home/logos .
receiving incremental file list
logos/
logos/myapp-logo-original.png
logos/myapp-logo-scaled.png

sent 53 bytes  received 2424 bytes  1651.33 bytes/sec
total size is 2241  speedup is 0.90
[root@containerA tmp2]#

Works fine. Total file size is about 2.5 KB.

But when I try to transfer more, it breaks every time:

[root@containerA tmp2]# rsync -e "docker -H serverB:2376 --tls exec -i" -av containerB:/srv/home .
receiving incremental file list
home/
home/.myapp-home.lock
home/dbconfig.xml
home/myapp-config.properties
home/analytics-logs/a60f9a3901276ffa4e5e71d9f996caee.foocorp-analytics.log
write /dev/stdout: resource temporarily unavailable
rsync: connection unexpectedly closed (1037682 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [receiver=3.0.9]
rsync: connection unexpectedly closed (362 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9]
[root@containerA tmp2]#

I've repeated this test many times, it never works. The total amount of data I would like to transfer is about 125 GB, but it always stops after a few tens of KB.

[glyph]

Have you tried using the --blocking-io option to rsync? That seems to do the trick for me.

[peter-leonov]

Yes! --blocking-io fixes the bug. Thanks a lot @glyph :)

[samatwork]

Confirmed! Excellent! Thank you!

[icy]

Why docker exec can't work with blocking IO? Should this be a Docker-specific problem? Is there anyway to fix that in Docker instead of fixing a client option?

[glyph]

It's not docker exec, really, it's rsync. In the case of ssh, non-blocking stdin on the client doesn't result in non-blocking stdin on the server; docker exec is much more faithfully reproducing the execution environment on the client on the backend. Nobody's wrong here, exactly, and this is why rsync has the option in the first place; docker exec works more like sh -c than ssh in this regard, and I think it's probably a reasonable design choice (although an unfortunate one from rsync's point of view)

[icy]

If it's not docker exec, then why nsenter can resolve this problem? Can docker exec have the ability of nsenter?

[glyph]

nsenter actually just runs the process directly; docker exec proxies the command through the docker client↔engine connection, so a lot of low-level UNIX details change. The point is, just pass --blocking-io and it tells rsync to do the right thing for this situation. The --blocking-io option specifically mentions that some programs used with -e require it; docker exec is one.

[icy]

Thanks for your explanation. I think I will use nsenter instead of docker exec which doesn't have any benefit except that it brings more layers of complexity . Because ssh prefers non-blocking-io, it's very hard to convince myself to use blocking-io within docker exec :)

Update: fix typoe.

[glyph]

The benefit of docker exec is that it is a docker command. For example, if you are using Swarm (as I am, on https://getcarina.com ) you can execute docker containers via docker run and docker exec, but you can't run nsenter on the node where the containers are running.