"file has vanished" errors with lxc-copy

[sephii]

On LXC 2.0.6, with kernel 4.8.0, creating a Debian container with overlayfs and then trying to copy it with sudo lxc-copy -n testoverlaystd -N "testoverlaystd2" -s -B overlayfs always fails with the following output:

file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/random"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/tty1"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/tty0"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/tty3"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/kmem"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/null"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/tty2"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/zero"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/mem"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/tty4"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/urandom"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/ptmx"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/tty"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/port"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/full"
file has vanished: "/var/lib/lxc/testoverlaystd2/rootfs/dev/console"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1196) [sender=3.1.2]
lxc-copy: bdev/lxcoverlay.c: ovl_rsync: 717 rsyncing /var/lib/lxc/testoverlaystd/rootfs to /var/lib/lxc/testoverlaystd2/rootfs
lxc-copy: bdev/lxcoverlay.c: ovl_do_rsync: 757 copying overlayfs delta
lxc-copy: bdev/bdev.c: bdev_copy: 388 failed getting pathnames for cloned storage: overlayfs:/var/lib/lxc/testoverlaystd/rootfs:/var/lib/lxc/testoverlaystd/delta0
lxc-copy: lxccontainer.c: copy_storage: 2856 Error copying storage.
clone failed

Here's the complete log:

lxc-copy 20170127140538.242 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
lxc-copy 20170127140538.242 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
lxc-copy 20170127140538.243 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/testoverlaystd/rootfs onto /var/lib/lxc/testoverlaystd/rootfs with options upperdir=/var/lib/lxc/testoverlaystd/delta0,lowerdir=/var/lib/lxc/testoverlaystd/rootfs,workdir=/var/lib/lxc/testoverlaystd/olwork.
lxc-copy 20170127140538.243 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/testoverlaystd/rootfs onto /var/lib/lxc/testoverlaystd2/rootfs with options upperdir=/var/lib/lxc/testoverlaystd2/delta0,lowerdir=/var/lib/lxc/testoverlaystd/rootfs,workdir=/var/lib/lxc/testoverlaystd2/olwork.
lxc-copy 20170127140538.535 ERROR    lxcoverlay - bdev/lxcoverlay.c:ovl_rsync:717 - rsyncing /var/lib/lxc/testoverlaystd/rootfs to /var/lib/lxc/testoverlaystd2/rootfs
lxc-copy 20170127140538.535 ERROR    lxcoverlay - bdev/lxcoverlay.c:ovl_do_rsync:757 - copying overlayfs delta
lxc-copy 20170127140538.535 ERROR    bdev - bdev/bdev.c:bdev_copy:388 - failed getting pathnames for cloned storage: overlayfs:/var/lib/lxc/testoverlaystd/rootfs:/var/lib/lxc/testoverlaystd/delta0
lxc-copy 20170127140538.535 ERROR    lxc_container - lxccontainer.c:copy_storage:2856 - Error copying storage.
lxc-copy 20170127140538.536 INFO     lxc_container - lxccontainer.c:container_destroy:2403 - Destroyed directory for testoverlaystd2

[brauner]

Hm, works for me here. Can you please show the config of the source container?

[sephii]

Here's the config. Also the container was created using lxc-create -t debian -n testoverlaystd -B overlayfs:

# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

lxc.network.type = veth
lxc.network.hwaddr = 00:16:3e:81:13:e7
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.rootfs = overlayfs:/var/lib/lxc/testoverlaystd/rootfs:/var/lib/lxc/testoverlaystd/delta0
lxc.rootfs.backend = overlayfs

# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf

# Container specific configuration
lxc.tty = 4
lxc.utsname = testoverlaystd
lxc.arch = amd64

[sephii]

It seems to work if the base container is created with the dir backingstore. Maybe what I'm trying to do just doesn't make sense (having an overlayfs base container)?

[alfred-sa]

Hello,

The same here... any news ?
(I made it works by changing the rsync command used in source file src/lxc/bdev/lxcrsync.c)

root@s3:/var/lib/lxc# lxc-snapshot --name vsftpd
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/ptmx"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/zero"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/tty2"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/full"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/tty"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/tty3"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/tty1"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/urandom"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/tty4"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/null"
file has vanished: "/var/lib/lxc/vsftpd/snaps/snap0/rootfs/dev/random"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1196) [sender=3.1.2]
lxc-snapshot: bdev/lxcoverlay.c: ovl_rsync: 717 rsyncing /var/lib/lxc/vsftpd/rootfs to /var/lib/lxc/vsftpd/snaps/snap0/rootfs
lxc-snapshot: bdev/lxcoverlay.c: ovl_do_rsync: 757 copying overlayfs delta
lxc-snapshot: bdev/bdev.c: bdev_copy: 388 failed getting pathnames for cloned storage: overlayfs:/var/lib/lxc/vsftpd/rootfs:/var/lib/lxc/vsftpd/delta0
lxc-snapshot: lxccontainer.c: copy_storage: 2867 Error copying storage.
lxc-snapshot: lxccontainer.c: do_lxcapi_snapshot: 3415 clone of /var/lib/lxc:vsftpd failed
lxc-snapshot: tools/lxc_snapshot.c: do_snapshot: 201 Error creating a snapshot

Thank you

[adityacloud1]

Respected @whuji,
I also face Rsync send failed error`` error while migrating LXD container under load testing environment. Could you please suggest, how to resolve this error using --ignore-missing-args`.

[stgraber]

So to clarify, has anyone seen this on LXC 3.0 or higher?

[stgraber]

(LXD doesn't count, completely unrelated logic)

[brauner]

This could happen if the container is shutdown or unmounted after rsync has built up its initial file transfer list and it seems that that's what's happening. I've reworked the overlay driver so this should be a bit more robust now. Please re-open if the issue still persists.