docker pull fails, operation not permitted

[karalabe]

I've used docker on my system for quite a while, but after a few months of idle time when I try to pull an image now, everything fails with the following error:

$ docker pull ubuntu
[...]
Error pulling image (latest) from ubuntu, ApplyLayer exit status 1 stdout:  stderr: mkdir /usr/local/bin: operation not permitted

I actually don't even know how docker managed to conjure this error up. /usr/local/bin exists, so irrespective of root or non-root status, the mkdir error would be "already exists", definitely not operation not permitted.

On a side note, why does docker even try to create that folder? As far as I know it doesn't use it for anything?

[cpuguy83]

docker info and docker version output would help, thanks!

[karalabe]

Sorry about that :)

> docker info
Containers: 6
Images: 37
Storage Driver: btrfs
 Build Version: Btrfs v3.16.2+20141003
 Library Version: 101
Execution Driver: native-0.2
Kernel Version: 3.16.7-7-desktop
Operating System: openSUSE 13.2 (Harlequin) (x86_64)
CPUs: 2
Total Memory: 3.866 GiB
Name: turnip.karalabe.com
ID: D3PI:33LF:SE3S:OBWL:TU5Y:4HNG:KZVB:L4I7:2KDA:RHNP:VXXT:55GD
WARNING: No swap limit support

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.3.3
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.3.3
Git commit (server): a8a31ef

[karalabe]

After playing around with everything, I've tried to delete all containers and images, and got hit with a load of error messages such as:

Error response from daemon: Cannot destroy container faab0b987973: Driver btrfs failed to remove root filesystem faab0b987973cd92553c33beea20f60207dc8f4ad5005fbeda9ef8872bc3fa01: Failed to destroy btrfs snapshot: invalid argument

However, the stuff disappeared from docker. I've tried pulling again, and got hit with

Driver btrfs failed to create image rootfs 511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158: Failed to create btrfs subvolume: file exists

So I went in and deleted all leftovers manually from the docker installation folder. Now it seems to work and download the images fine.

I'm guessing the issue in in the btrfs driver whereby it doesn't notice some external changes done to the btrfs snapshots and thus reports all kinds of weird error messages. Although the issue seems to be resolved now, it would be nice to look into the driver's error messages/handling.

[cpuguy83]

These errors look to be coming from mount.
The error messages themselves should be enhanced in docker 1.6

So you modified some of the layers external to Docker?

[karalabe]

I had some issues with a Linux distro update so I rolled back the btrfs file system to a previous state, and deleted all existing snapshots as I figured they aren't really needed anyway. As it would seem, docker was using those snapshots and didn't really like that I got rid of them.

[cpuguy83]

Ok, seems like we can close this then?

FYI, the mkdir stuff is likely because ApplyLayer is happening in a chroot and it was trying to extract to something that didn't exist.

[MichaelAquilina]

I have this exact same issue now. I am running OpenSUSE Tumbleweed and after a recent rollback I get "operation not permitted" for almost every docker operation I perform. I had the same issue a while ago and had to manually delete everything in /var/lib/docker. Is there a better way to resolve this rather than nuking my docker installation after each rollback? This is particularly frustrating for large docker projects that need to bootstrap again :/