Failing to create a volume in docker compose

[mariash]

Bug Report

Concourse docker compose fails to run builds.

$ fly -t local execute -c ~/pipelines/simple-task.yml
executing build 1
initializing
failed to create volume
errored

docker version:

Version 1.13.0 (15072)
Channel: Stable
0c6d765c54

docker info:

Containers: 8
 Running: 3
 Paused: 0
 Stopped: 5
Images: 15
Server Version: 1.13.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 84
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.4-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952 GiB
Name: moby
ID: DFZX:T445:7IS3:ONLY:CJWJ:STH5:OSRS:47BB:Y7Q7:H7EH:66QL:JVJ5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 39
 Goroutines: 58
 System Time: 2017-01-27T01:22:48.092966869Z
 EventsListeners: 2
No Proxy: *.local, 169.254/16
Username: marynixie
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #138463093 Investigate: Failing to create a volume in docker compose
• #140277749 reopened: Failing to create a volume in docker compose
• #138656889 Failing to create a volume in docker compose

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[eedwardsdisco]

Getting this as well.

docker-compose on mac

$ fly -t main execute --config ci/tasks/unit.yml
executing build 1
failed to create volume
errored

[vito]

Need logs from BaggageClaim or something to be able to debug this.

[mariash]

I believe it is always reproducible with docker compose.

[berisberis]

{"timestamp":"1485479149.779026508","source":"baggageclaim","message":"baggageclaim.repository.create-volume.failed-to-materialize-strategy","log_level":2,"data":{"error":"exit status 1","handle":"63e9f9d0-83ac-4e57-5912-a74cf745f653","session":"2.51"}} {"timestamp":"1485479155.638654470","source":"tsa","message":"tsa.connection.channel.forward-worker.heartbeat.start","log_level":1,"data":{"remote":"172.17.0.4:57226","session":"1.1.1.19","worker-address":"127.0.0.1:39595","worker-platform":"linux","worker-tags":""}} {"timestamp":"1485479155.641455889","source":"guardian","message":"guardian.list-containers.starting","log_level":1,"data":{"session":"33"}} {"timestamp":"1485479155.641624689","source":"guardian","message":"guardian.list-containers.finished","log_level":1,"data":{"session":"33"}} {"timestamp":"1485479155.643083334","source":"tsa","message":"tsa.connection.channel.forward-worker.heartbeat.reached-worker","log_level":0,"data":{"remote":"172.17.0.4:57226","session":"1.1.1.19","took":"3.677294ms"}} {"timestamp":"1485479156.859302998","source":"tsa","message":"tsa.connection.channel.forward-worker.heartbeat.done","log_level":1,"data":{"remote":"172.17.0.4:57226","session":"1.1.1.19","worker-address":"127.0.0.1:39595","worker-platform":"linux","worker-tags":""}}

[eedwardsdisco]

Same error as @berisberis

I'm wondering if this is due to a change in docker or docker for mac, since I tried with 2.6.0 all the way back to 2.2.1 (the version I KNOW my task worked on) and they all exhibit this problem.

[berisberis]

It is reproducible with docker compose only if you use Docker-for-mac instead of Docker-toolbox.

[eedwardsdisco]

I've uninstalled docker-for-mac 1.13.0 and re-installed previous version https://download.docker.com/mac/stable/1.12.3.13776/Docker.dmg and the problem goes away

appears that this broke in 1.13.0

[topherbullock]

Seems to be a known issue in Docker for Mac
docker/for-mac#388

[eedwardsdisco]

@pivotal-topher-bullock Not sure I'd call it a 'known issue', it's been awaiting further comment from the concourse team:

Could you direct us to something about why btrfs is necessary for this use case? We currently support aufs and overlay2 and we would like the user experience of Docker for Mac to abstract the graph driver decision entirely. Any information about why btrfs is necessary would be really helpful for us. Thanks!

Appears that Docker for Mac is trying to find out why Concourse requires btrfs in the first place. Is it possible for concourse to use overlay2 \ aufs instead? Doesn't sound like they're going to add back btrfs support any time soon, and docker toolbox is not the recommended solution for mac any more (docker wants you to use docker-for-mac).

[thiagoramos23]

I am having the same issue

[alexeybondarenko]

Same error. I've tried with 2.5.1 and 2.6.0

[eedwardsdisco]

@vito someone from Docker did a repro on this issue and is getting the error on both versions, anything useful in the errors he posted?

see docker/for-mac#388

[vito]

This should be resolved by #1045 - I think it's time to bail on btrfs.

[gonzalomarcote]

This also happens in Linux.

$ fly -t ci execute -c build.yml
executing build 3
failed to create volume
errored

docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:09:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:09:06 2017
 OS/Arch:      linux/amd64
 Experimental: false

one simple hello-world example doesn't work wit concourse in docker-compose

[marcbachmann]

failed to create volume
errored

I guess In my case it was also a btrfs issue. @vito thanks for the hint.