Error starting daemon:... invalid cross-device link

[kiemlicz]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

On Ubuntu Artful's docker image I'm unable to install and start docker service

Expected behavior

Working (enabled and started dockerd)

Actual behavior

Following message in /var/log/docker.log

time="2018-02-19T22:15:08.255302189Z" level=info msg="libcontainerd: started new docker-containerd process" pid=1311
time="2018-02-19T22:15:08Z" level=info msg="starting containerd" module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0 
time="2018-02-19T22:15:08Z" level=info msg="setting subreaper..." module=containerd 
time="2018-02-19T22:15:08Z" level=info msg="changing OOM score to -500" module=containerd 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." module=containerd type=io.containerd.content.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1 
time="2018-02-19T22:15:08Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." module=containerd type=io.containerd.snapshotter.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." module=containerd type=io.containerd.metadata.v1 
time="2018-02-19T22:15:08Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt" 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." module=containerd type=io.containerd.differ.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." module=containerd type=io.containerd.gc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." module=containerd type=io.containerd.monitor.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." module=containerd type=io.containerd.runtime.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." module=containerd type=io.containerd.grpc.v1 
time="2018-02-19T22:15:08Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug" 
time="2018-02-19T22:15:08Z" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc" 
time="2018-02-19T22:15:08Z" level=info msg="containerd successfully booted in 0.002996s" module=containerd 
Error starting daemon: rename /var/lib/docker/runtimes /var/lib/docker/runtimes-old: invalid cross-device link

Steps to reproduce the behavior

1. use any Dockerfile that uses artful:

FROM ubuntu:artful
ENTRYPOINT /bin/bash

2. install docker in running container using steps from docker-ce doc

3. docker won't start (on e.g. Debian images it works flawlessly)

Output of docker version:
dockerd is not running

Output of docker info:
dockerd is not running

Additional environment details (AWS, VirtualBox, physical, etc.)
dockerd is not running

[keyurgolani]

Also facing this same issue. Trying to run docker inside docker container. But it is failing and the log says this line.

time="2018-05-07T21:55:17.019667500Z" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: invalid cross-device link. Deleting synchronously"
Error starting daemon: rename /var/lib/docker/runtimes /var/lib/docker/runtimes-old: invalid cross-device link

[thaJeztah]

Is /var/lib/docker defined as a volume? Otherwise you may be trying to run (e.g.) overlay2 on top of overlay2, which wont work (in general, copy-on-write filesystems cannot be stacked)

Start an artful container, with /var/lib/docker as volume:

docker run -it --rm --privileged -v /var/lib/docker ubuntu:artful

inside the container;

apt-get update && apt-get install -y curl && curl -fsSL https://get.docker.com | sh

dockerd&

docker run hello-world

[aquaminer]

Solution: https://gist.github.com/Francesco149/ce376cd83d42774ed39d34816b9e21db

[aivus]

The simple fix of this issue is just to remove /var/lib/docker/runtimes directory manually before running docker.
Overlayfs doesn't support renaming.

[duzezhen]

The simple fix of this issue is just to remove /var/lib/docker/runtimes directory manually before running docker. Overlayfs doesn't support renaming.

it worked!

[Tptogiar]

Also facing this same issue. Trying to run docker inside docker container. But it is failing and the log says this line.

time="2018-05-07T21:55:17.019667500Z" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: invalid cross-device link. Deleting synchronously"
Error starting daemon: rename /var/lib/docker/runtimes /var/lib/docker/runtimes-old: invalid cross-device link

I also trouble in the problem when i try to run a docker in docker

[Tptogiar]

it worked!

The simple fix of this issue is just to remove /var/lib/docker/runtimes directory manually before running docker. Overlayfs doesn't support renaming.

┗|'O′|┛ Wha~~, you really helped me a lot, I have to thank you even if I don't eat

[aivus]

Should be fixed as moby/moby#43225 merged