docker + btrfs: file operations trigger AUFS related warnings

[unclejack]

This is a bug in AUFS which I've managed to trigger by running docker with the /var/lib/docker folder stored on a btrfs fs.

How I can reproduce:

1. set up the machine to use btrfs
2. docker run -i -t ubuntu:12.04
3. install packages needed for building C & C++ code
4. git clone some really large repo
5. configure and compile
   During compilation, it says "stale NFS file handle" after building for a while.

It's much easier to trigger this by running make clean after the build is done.

The warning always looks like this:

98279.856904] WARNING: at fs/inode.c:281 drop_nlink+0x25/0x50()
[98279.857049] Hardware name: VMware Virtual Platform
[98279.857052] Modules linked in: isofs xt_nat veth aufs xt_addrtype xt_CLASSIFY sch_prio xt_DSCP xt_length xt_tcpudp iptable_mangle openvswitch ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc bnep rfcomm bluetooth deflate ctr twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic camellia_x86_64 serpent_sse2_x86_64 serpent_generic glue_helper blowfish_generic blowfish_x86_64 blowfish_common cast5_generic cast_common des_generic xcbc rmd160 sha512_generic crypto_null af_key xfrm_algo binfmt_misc snd_ens1371 coretemp joydev kvm_intel kvm snd_ac97_codec ghash_clmulni_intel ac97_bus aesni_intel gameport aes_x86_64 ablk_helper cryptd xts snd_pcm lrw gf128mul snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer psmouse snd ppdev soundcore serio_raw vmw_balloon microcode parport_pc vmw_vmci mac_hid i2c_piix4 shpchp lp parport btrfs xor zlib_deflate raid6_pq libcrc32c hid_generic usbhid hid floppy vmxnet3 mptsas vmwgfx mptscsih mptbase scsi_transport_sas ttm drm
[98279.969428] Pid: 26846, comm: rm Not tainted 3.9.8 #1
[98279.969434] Call Trace:
[98279.969920]  [<ffffffff81057a73>] warn_slowpath_common+0x93/0xc0
[98279.969929]  [<ffffffff81057aba>] warn_slowpath_null+0x1a/0x20
[98279.969932]  [<ffffffff811a7e15>] drop_nlink+0x25/0x50
[98279.970003]  [<ffffffffa05a0d17>] au_whtmp_rmdir+0x137/0x1a0 [aufs]
[98279.970264]  [<ffffffff811a4948>] ? d_kill+0x138/0x150
[98279.970272]  [<ffffffffa05adca8>] aufs_rmdir+0x208/0x4a0 [aufs]
[98279.970470]  [<ffffffff810efbe7>] ? call_rcu_sched+0x17/0x20
[98279.970474]  [<ffffffff8119c63b>] vfs_rmdir+0xab/0x130
[98279.970476]  [<ffffffff8119c7db>] do_rmdir+0x11b/0x1d0
[98279.970498]  [<ffffffff8107910b>] ? task_work_run+0xab/0xd0
[98279.970520]  [<ffffffff8101498c>] ? do_notify_resume+0x6c/0x90
[98279.970523]  [<ffffffff8119eefb>] sys_unlinkat+0x1b/0x50
[98279.971284]  [<ffffffff816af7d9>] system_call_fastpath+0x16/0x1b

[jpetazzo]

Alright, this one is going to be a though one.

Could you provide a full example, i.e.:

• full apt-get install command to use to install compilation environment;
• which repo you tried to compile;
• compilation parameters (if any).

I know that it's tedious, but if you could provide a full docker run ... command which exhibits the bug, that would be Very Awesome.

Thanks a lot!

[unclejack]

@jpetazzo I'll provide an image.

[creack]

@unclejack ping

[unclejack]

I've tried out the following Dockerfile and make clean triggers the bug when the docker storage is on BTRFS:

from ubuntu:12.04
run apt-get install -y wget curl openssl build-essential libncurses5-dev bc
run wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.2.tar.xz -O /root/kernel.tar.xz
run tar -xpf /root/kernel.tar.xz -C /root/
run cd /root/linux-3.10.2 && make defconfig && make -j2 && make clean

[unclejack]

Extracting and building in one step seems to trigger it every time:

from ubuntu:12.04
run apt-get install -y wget curl openssl build-essential libncurses5-dev bc
run wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.2.tar.xz -O /root/kernel.tar.xz
run tar -xpf /root/kernel.tar.xz -C /root/ && cd /root/linux-3.10.2 && make defconfig && make -j2 && make clean

output:

  CLEAN   arch/x86/boot/compressed
  CLEAN   arch/x86/boot
  CLEAN   arch/x86/tools
  CLEAN   .tmp_versions
sh: 0: getcwd() failed: No such file or directory
find: cannot get current directory: Stale NFS file handle

[shykes]

Not sure it's related to your problem, but you can use ADD to download and
unpack the tarball directly without needing to store the intermediary
tarball, and without needing to install wget :)

---

DOCKER-VERSION 0.4.8
ADD https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.2.tar.xz /root/

RUN cd /root/linux-3.10.2 && make defconfig && make -j2 && make clean

On Wed, Jul 24, 2013 at 12:05 PM, unclejack notifications@github.comwrote:

Extracting and building in one step seems to trigger it every time:

from ubuntu:12.04
run apt-get install -y wget curl openssl build-essential libncurses5-dev bc
run wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.2.tar.xz
-O /root/kernel.tar.xz
run tar -xpf /root/kernel.tar.xz -C /root/ && cd /root/linux-3.10.2 && make defconfig && make -j2 && make clean

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1075#issuecomment-21508201
.

[unclejack]

@shykes I agree. The only purpose of that Dockerfile was to trigger the bug. I was building a bit more code when I've discovered the problem and I had to find something similar which downloads and compiles less.

[jpetazzo]

This might be related to #1320.

[unclejack]

AUFS is being removed in 0.7 and this issue should be reopened when AUFS gets added back as a plugin.

I'm closing this issue for now.

[unclejack]

Reopening this as AUFS will be available in 0.7 and it'll be used if the host meets the requirements.

[zedr]

To side-step the issue, you might want to create a new partition with EXT4 and either:

• mount it on /var/lib/docker (make sure to copy over your files);
• mount it somewhere else, relocate the files, and symlink /var/lib/docker to it;
• mount it somewhere else, and run the deamon with the -g option pointing to the mountpoint.

I had the same problem, and relocating the docker folder on an EXT4 partition provided a workaround.

[unclejack]

@zedr Storing the data somewhere else is a known workaround for these problems. However, that doesn't help solve AUFS and btrfs problems.

There are also some bugs in AUFS which need to be fixed so btrfs and the other file systems may never run into them again.

[unclejack]

@paulepanter found an easier way to reproduce this issue:

    # ./configure 
    pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
    configure: error: working directory cannot be determined
    root@file:/tmp/file-5.08# pwd
    /tmp/file-5.08
    root@file:/tmp/file-5.08# /bin/bash -c pwd
    shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
    pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or director
    root@file:/tmp/file-5.08# ls
    ls: cannot open directory .: Stale NFS file handle

I'm not seeing the warning any more, but the other problems (stale NFS file handle and broken getcwd()) are still present.

[unclejack]

I'm closing this now in favor of #2056 because the warnings are gone and the getcwd() problem is the same..