failed to register layer: rename [...] directory not empty

[sweber83]

Output of docker version:

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   8acee1b
 Built: 
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   8acee1b
 Built:
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 7
 Running: 6
 Paused: 0
 Stopped: 1
Images: 10
Server Version: 1.10.3
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.5.0-coreos-r1
Operating System: CoreOS 1010.5.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.958 GiB
Name: REDACTED-01
ID: Q2H3:YKQ4:TJWM:ONUY:2DJR:NWTQ:NTHY:6DSU:D5SX:QAOU:EEPM:6WBK
Username: REDACTED
Registry: https://index.docker.io/v1/

Additional environment details (AWS, VirtualBox, physical, etc.):
Docker is running in VMs hosted by digitalocean.com

Steps to reproduce the issue:

docker pull clue/adminer

Describe the results you received:

failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-558969056 /var/lib/docker/image/overlay/layerdb/sha256/3d979a98dc302b351aec55cf58f1b2bd86880c57df9553fb4fe0f94087f04e0b: directory not empty

Describe the results you expected:

Status: Downloaded newer image for clue/adminer:latest

Additional information you deem important (e.g. issue happens only occasionally):
The issue is not easy to reproduce, just happens sometimes.
The issue occured with different docker images, I just use adminer as an example because it is in a public repository.
The issue only occurred after switching from docker 1.9.1 to docker 1.10.3 ( CoreOS 899.17.0 -> 1010.5.0 )

Clearing the Docker data directory fixes the issue temporarily but then all images need to be downloaded again, so it doesn't really solve the problem.
Here is a stackoverflow post on the issue where it says to empty the working directory which again only is a temporary fix:
https://stackoverflow.com/questions/35325103/docker-error-when-pulling-java-8-image-failed-to-register-layer

[thaJeztah]

Are you still seeing this issue on the current release (1.11.2?). Note that you're running a version of docker that is built and supported through CoreOS; it contains some modifications that are not in the official builds, and could be of influence here (see the commit that this was built from coreos@8acee1b)

[sweber83]

We aren't using the current release of docker yet.
I realize that the version of docker shipping with CoreOS is customized, so I wasn't really sure where to post this issue.
Then I'll post on the CoreOS issues, too and cross link the two posts?
Sorry for the question, but I'm pretty new to github.

[thaJeztah]

@sweber83 yes, the primary support in this case should go through CoreOS; I'll close this issue for now, but happy to reopen if there appears to be a general issue after it's been looked at by CoreOS

[petrosagg]

We've also hit this issue with an unmodified docker 1.10.3 on btrfs. I don't think this is specific to CoreOS. It looks like the operations while pulling the layers are not atomic and can result in a corrupted /var/lib/docker.

We're running docker on embedded devices so power failures happen much more often compared to a datacenter.

[xueshanf]

I did not remember to have this issue before 1.10. I also reported to coreos/bug#1397, but it doesn't seem to be limited to CoreOS. Here is another Max OX with docker machine.
Error:

failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-068187541 /var/lib/docker/image/overlay/layerdb/sha256/2d90fb8f617bf974548cbe33226ed5a677056637f02cc4196fcffac66debABCD: directory not empty

# ls -l /var/lib/docker/image/overlay/layerdb/sha256/2d90fb8f617bf974548cbe33226ed5a677056637f02cc4196fcffac66debABCD

-rw-r--r-- 1 root root 0 Jun 24 00:01 cache-id
-rw-r--r-- 1 root root 0 Jun 24 00:01 diff
-rw-r--r-- 1 root root 0 Jun 24 00:01 parent
-rw-r--r-- 1 root root 0 Jun 24 00:01 size
-rw-r--r-- 1 root root 0 Jun 24 00:01 tar-split.json.gz

The workaround is to remove the empty files under the said directory and re-pull the image.

System info:

Containers: 45
 Running: 24
 Paused: 0
 Stopped: 21
Images: 102
Server Version: 1.10.3
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.5.7-coreos
Operating System: CoreOS 1010.6.0 (MoreOS)
OSType: linux
Architecture: x86_64

[ezesculli]

any news about it? I'm getting the same on Mac OS X El Capitan

[kovalevsky]

Reproduced on Debian 8:

$ uname -a
Linux hostname 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux

$ docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:23:39 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:23:39 2016
 OS/Arch:      linux/amd64

$ docker info
Containers: 47
 Running: 3
 Paused: 0
 Stopped: 44
Images: 165
Server Version: 1.11.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 316
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 47.26 GiB
Name: hostname
ID: ZHKM:SAIE:LLZD:7SSW:FTOM:5T53:JWKK:H2CR:KBNS:NALM:GFSG:GC4P
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

[thaJeztah]

@kovalevsky actually; this looks like a duplicate of #23184, which is still open; could you add more info there?

[mpekalski]

I have the same issue. Lubuntu 16.10 running on Oracle VM Box. Clean install. And I am trying to run

% docker pull jupyter/base-notebook 
Using default tag: latest
latest: Pulling from jupyter/base-notebook
693502eb7dfb: Extracting [==================================================>] 51.36 MB/51.36 MB
24ff998bd82e: Download complete 
6e3c00741208: Download complete 
a84bb1c63b96: Download complete 
0dabba4d355e: Download complete 
3507d90284be: Download complete 
a956b54901d3: Download complete 
a940391c8107: Download complete 
f02c9823d0a9: Download complete 
0fbe701e5670: Download complete 
c4d268a619b8: Download complete 
e8aef5744fe3: Download complete 
e301e361d59a: Download complete 
failed to register layer: rename /var/lib/docker/image/aufs/layerdb/tmp/layer-051919914 /var/lib/docker/image/aufs/layerdb/sha256/d17d48b2382adda1fd94284c51d725f0226bf20b07f4d29ce09596788bed7e8e: directory not empty

% docker --version
Docker version 1.12.6, build 78d1802

[thaJeztah]

@mpekalski you're commenting on a closed issue, and the version of docker you're using is outdated, and no longer maintained; if you're still able to reproduce on the current version, please open a new issue with details.

I'm locking the conversation on this issue, because this issue is closed, and new issues being reported here can be non-related to the original issue being reported here.