Error downloading dependent layers with Btrfs storage backend

[iwinux]

Got a weird problem and no idea what was going on:

$ docker pull dockerfile/ghost 
Pulling repository dockerfile/ghost
39a76a2f0d9f: Error pulling image (latest) from dockerfile/ghost, endpoint: https://cdn-registry-1.docker.io/v1/, open /var/lib/docker/btrfs/subvolumes
/c1bd2f6276a1c1774977c43bfc68828df89113aeefdf6384e45349edd064cad8/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_restricted_source_S
39a76a2f0d9f: Error pulling image (latest) from dockerfile/ghost, open /var/lib/docker/btrfs/subvolumes/c1bd2f6276a1c1774977c43bfc68828df89113aeefdf638
4e45349edd064cad8/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_restricted_source_Sources: file exists 
173b9b8c2fc9: Download complete 
c1bd2f6276a1: Error downloading dependent layers 

Related Info

OS: Arch Linux
Kernel: 3.12.14

$ docker info  
Containers: 1
Images: 37
Storage Driver: btrfs
Execution Driver: native-0.1
Kernel Version: 3.12.14

$ docker version
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0

[vieux]

I can't reproduce /cc @crosbymichael

[iwinux]

@vieux

Here's the steps I've taken when setting up Docker (hope this would help):

1. create a logical volume: lvcreate -L 5G Shuffle -n Docker

2. create a Btrfs partition on that volume:

   $ mkfs.btrfs -L docker /dev/mapper/Shuffle-Docker 
   
   WARNING! - Btrfs v3.14 IS EXPERIMENTAL
   WARNING! - see http://btrfs.wiki.kernel.org before using
   
   Performing full device TRIM (5.00GiB) ...
   Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
   fs created label docker on /dev/mapper/Shuffle-Docker
           nodesize 16384 leafsize 16384 sectorsize 4096 size 5.00GiB
   Btrfs v3.14
   

3. mount it: mkdir /docker && mount /dev/mapper/Shuffle-Docker /docker

4. run the daemon:
   docker -d -G users --dns 172.17.42.1 -p /tmp/docker.pid -g /docker -s btrfs
   and the directory structure looks like this:

   $ tree /docker
   /docker
   ├── containers
   ├── execdriver
   │   └── native
   ├── graph
   ├── init
   │   └── dockerinit-0.10.0
   ├── linkgraph.db
   ├── repositories-btrfs
   └── volumes
   
   6 directories, 3 files
   

[iwinux]

BTW: the Btrfs partition is mounted with option rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache

[iwinux]

Update: I took out space_cache,autodefrag,inode_cache from mount option, and now it works without problem...any ideas?

[csirac2]

For what it's worth, I've been using docker's btrfs storage driver for about 3-4 months. The only issues I've encountered are related to using up all metadata blocks (fixable with a rebalance). I don't have space_cache or inode_cache, but I am using autodefrag.

[sqs]

I experienced this same issue when using a btrfs FS with mount options rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache. Removing the inode_cache,space_cache options fixed it.

My system specs:

# uname -a
Linux ip-10-221-138-19 3.10.35-43.137.amzn1.x86_64 #1 SMP Wed Apr 2 09:36:59 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# docker info
Containers: 0
Images: 5
Driver: btrfs
# docker version
Client version: 0.9.0
Go version (client): go1.2
Git commit (client): 2b3fdf2/0.9.0
Server version: 0.9.0
Git commit (server): 2b3fdf2/0.9.0
Go version (server): go1.2
Last stable version: 0.11.1, please update docker

[LK4D4]

Hmmm, I experienced this with 3.15 kernel. Switching to 3.14 helps.
Also this was with ubuntu pulling, busybox was fine.

[iwinux]

@LK4D4 can this behavior be stably reproduced? I'm still unsure about the root cause...

[LK4D4]

@iwinux Hmm, okay, I describe what I got:

noatime,compress=lzo,autodefrag,space_cache,inode_cache

in fstab.
All files in filesystem were copied before options was specified. After I removed inode_cache and did

btrfs filesystem defragment -r -v -clzo /

all seems well on kernel 3.15. I think that problem is in inode_cache, which is not recommended to set by Btrfs wiki, but appears in example from ArchWiki.

[iwinux]

@LK4D4 thx for narrowing it down 👍

[jessfraz]

All seems well on 3.17 kernel too

[jessfraz]

@iwinux are you still having this issue?

[vbatts]

i'm not seeing this issue. I'm on 3.18.7, mounted with options rw,noatime,seclabel,ssd,discard,space_cache,commit=90

[cpuguy83]

Closing since it seems like noone is having this issue anymore.