docker-compose build hangs

[bfallik]

$ docker version
Client:
 Version:      1.13.0
 API version:  1.24 (downgraded from 1.25)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Wed Jan 18 16:20:26 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.6
 API version:  1.24 (minimum version )
 Go version:   go1.6.3
 Git commit:   d5236f0
 Built:        Fri Jan 20 05:47:10 2017
 OS/Arch:      linux/amd64
 Experimental: false

I'm trying to use docker-compose build but the command hangs:

$ docker-compose build --force-rm
Building integration-test-go
^CERROR: Aborting.

docker-compose.yml contains

version: '2'
services:
  integration-test-go:
    build:
      context: go
      dockerfile: go/Dockerfile

and go/.dockerignore has

bin/
coverage/

The same invocation via docker build completes in about a minute:

$ time docker build --force-rm=true go
...
Step 10 : ADD ./Makefile $GOPATH/Makefile
 ---> 7291ce227751
Removing intermediate container e69e54d81655
Step 11 : ADD ./src/ $GOPATH/src/
 ---> 690cf13fd4ae
Removing intermediate container 9649280eddff
Successfully built 690cf13fd4ae

real	1m49.268s
user	0m3.688s
sys	0m2.501s

Any idea what's going on here?

[shin-]

Can you try again with --verbose and share the output?

[shin-]

As well as the output of docker-compose version

[bfallik]

@shin-

$ docker-compose --version
docker-compose version 1.10.0, build 4bd6f1a

[bfallik]

@shin-

$ docker-compose --verbose build
compose.config.config.find: Using configuration files: ./../docker-compose.yml
docker.auth.find_config_file: Trying paths: ['/Users/bfallik/.docker/config.json', '/Users/bfallik/.dockercfg']
docker.auth.find_config_file: Found file at path: /Users/bfallik/.docker/config.json
docker.auth.load_config: Found 'auths' section
docker.auth.parse_auth: Found entry (registry=u'https://index.docker.io/v1/', username=u'bfallik')
docker.auth.parse_auth: Found entry (registry=u'https://468274692891.dkr.ecr.us-east-1.amazonaws.com', username=u'AWS')
compose.cli.command.get_client: docker-compose version 1.10.0, build 4bd6f1a
docker-py version: 2.0.1
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016
compose.cli.command.get_client: Docker base_url: http://localhost:2375
compose.cli.command.get_client: Docker version: KernelVersion=4.8.17-coreos, Os=linux, BuildTime=2017-01-20T05:47:10.598613119+00:00, ApiVersion=1.24, Version=1.12.6, GitCommit=d5236f0, Arch=amd64, GoVersion=go1.6.3
compose.service.build: Building go
compose.cli.verbose_proxy.proxy_callable: docker build <- (pull=False, stream=True, nocache=False, tag='go', buildargs=None, rm=True, forcerm=False, path='/Users/bfallik/sandbox/go', dockerfile='go/Dockerfile')

[mrjackdavis]

@shin-

I've got the same issue, with the same --verbose output.

It seems both @bfallik and I are using AWS ECR.

docker.auth.find_config_file: Trying paths: ['<REDACTED>/.docker/config.json', '<REDACTED>/.dockercfg']
docker.auth.find_config_file: Found file at path: <REDACTED>/.docker/config.json
docker.auth.load_config: Found 'auths' section
docker.auth.parse_auth: Found entry (registry=u'https://index.docker.io/v1/', username=u'<REDACTED>')
docker.auth.parse_auth: Found entry (registry=u'https://<REDACTED>.dkr.ecr.ap-southeast-2.amazonaws.com', username=u'AWS')
docker.auth.parse_auth: Found entry (registry=u'<REDACTED>.dkr.ecr.ap-southeast-2.amazonaws.com', username=u'AWS')
compose.cli.command.get_client: docker-compose version 1.10.0, build 4bd6f1a
docker-py version: 2.0.1
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.9.4-moby, Arch=amd64, BuildTime=2017-01-18T16:20:26.924957224+00:00, ApiVersion=1.25, Version=1.13.0, MinAPIVersion=1.12, GitCommit=49bf474, Os=linux, Experimental=True, GoVersion=go1.7.3
compose.project.build: mongo uses an image, skipping
compose.service.build: Building api
compose.cli.verbose_proxy.proxy_callable: docker build <- (pull=False, stream=True, nocache=False, tag='<REDACTED>', buildargs=None, rm=True, forcerm=False, path='<REDACTED>/<REDACTED>/src/api', dockerfile=None)```

[shin-]

Thanks for sharing. I must admit, I'm stumped - doesn't look like anything obvious is happening on the client side that would cause issues to occur while pulling.

Do you see anything relevant in the Docker daemon logs that could indicate a failure while processing the build request?

[bfallik]

@shin- I'm trying to follow them with syslog -w -k Sender Docker but I don't see much output when I invoke docker-compose. One line seems suspicious:

Feb  1 19:39:42 xxx Docker[1495] <Notice>: 127.0.0.1:0: dispatcher shutting down: read: Eof

Is there some other way I can view what's going on?

[shin-]

@bfallik Here's what I do when I need to dig in

1. Shut down running docker daemon (on Ubuntu, sudo service docker stop)
2. Manually start dockerd with the debug flag (sudo dockerd --debug)
3. Run docker-compose ... command in a separate term and watch logs

[bfallik]

@shin- Unfortunately I'm on OSX and I'm not as familiar with it as I might be with Linux. I don't see even a dockerd command available to me. I imagine it's buried somewhere in the Docker for Mac packaging.

[shin-]

I'm less familiar with Mac OS and D4M, but I think you can add "debug": true in the daemon options as shown here, then get the logs the same way you were doing already.

[bfallik]

@shin- so this is weird.

When I restart the daemon with "debug": true I don't see any logging from docker-compose. Here's the output:

$ syslog -w -k Sender Docker
...
Feb  1 21:34:09 Brians-MBP-4 Docker[9044] <Notice>: Docker is not responding: Get http://./info: EOF: waiting 0.5s
Feb  1 21:34:09 Brians-MBP-4 Docker[9043] <Notice>: Using protocol TwoThousand msize 8192
Feb  1 21:34:11 Brians-MBP-4 Docker[9044] <Notice>: Docker is responding
Feb  1 21:34:11 Brians-MBP-4 Docker[1496] <Notice>: VM started at 2017-02-01 21:34:11 -0500 EST


^C
$ 

I added those newlines to note when Docker completed its startup and when I invoked docker-compose.

[bfallik]

It's as if my docker client isn't talking to that daemon but I don't have any DOCKER_HOST variable set:

$ env | grep -i docker
$

[mrjackdavis]

Really odd. My colleague was able to build the project which I am unable to, on the same version of docker.

So far, I've tried:

• Restore Docker to factory settings
• Uninstall and install Docker
• Remove ~/.docker/config.json

All to no avail.

[mrjackdavis]

Sweet mother of god.

There's a SO post which mentions docker-compose build hangs when chrome is open (link)

My colleague was using Safari and his was working. I told him to open chrome and try again. It failed

Armed with this knowledge, I was able to fix by:

1. Update Chrome
2. Quit Chrome
3. rm -rf <my repository>
4. clone and setup repository
5. docker-compose build 👍
6. Open chrome
7. docker-compose build 👍

What. On. Earth.

Update
This did not last. When I rebooted docker, it started to hang again.

[bfallik]

@mrjackdavis that is super weird. I just tested docker-compose after killing Chrome but that didn't workaround my hang so maybe they're separate issues?

[mrjackdavis]

@bfallik did you also try it on an empty repository?

I updated the fix in my previous comment

[bfallik]

@mrjackdavis No, what did you mean by an empty repository? Which repository?

[mrjackdavis]

@bfallik Sorry - I wasn't clear.

I deleted my repository (my project files, with docker-compose.yml at the root). The entire folder. Then cloned it from remote. Do that right after quitting chrome.

[sschimmel]

Is there an ETA for the fix for this? I haven't been able to use Docker for my projects for over a month now. The workaround to exit Chrome is ridiculous.

[mrjackdavis]

Agreed. The chrome workaround isidiotic.

I've settled with a bash script which uses docker build directly. Terrible also, but better than the chrome fix.

[bfallik]

Does anyone have any idea why Chrome would even interact with docker-compose?

[mrjackdavis]

Docker Black Magic™

[mrjackdavis]

Does this only affect users who push their Docker repository to AWS ECR?

If it affects anybody who uses Chrome, I imagine this issue would be of higher profile.

[bfallik]

@mrjackdavis In my case I'm not pushing the images to ECR, or anywhere else for that matter.

[nathanleclaire]

I have Chrome open and I can execute a docker-compose build with the latest versions without issue.

$ docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Mar  2 01:11:00 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 07:52:04 2017
 OS/Arch:      linux/amd64
 Experimental: true

$ docker-compose --version
docker-compose version 1.11.2, build dfed245

$ docker-compose build --force-rm
Building foo
Step 1/4 : FROM alpine
 ---> 7d23b3ca3463
Step 2/4 : ENV FOO BAR
 ---> Using cache
 ---> 4e1e7e845ae3
Step 3/4 : ENV QUUX SPAM
 ---> Using cache
 ---> 15b03bf9044d
Step 4/4 : ENV EGGS BLACK
 ---> Using cache
 ---> 9f20c82b59d6
Successfully built 9f20c82b59d6

Try the latest versions of everything just to be sure. And if that doesn't work:

This bug seems specific to Docker for Mac right? (e.g., does executing it against a docker-machine work or not?)

What step of the build does it get stuck on?

What does the Docker daemon log (try something like docker run -ti -v /var/log/docker.log:/dlog alpine tail -50 /dlog to view this) say while this hang is occurring?

[bfallik]

@nathanleclaire I don't believe this is always an issue for Chrome + DfM or this ticket would have generated much more interest.

I don't know what you mean by "what step of the build does it get stuck on?" What are my options? I posted sample output above in case that helps you narrow the step down.

brian

[nathanleclaire]

Hm, I guess I had thought this might have been an issue with a build step yesterday (so I was wondering if it was hanging on a Step X/Y line), but reading over the thread it seems a little bit more likely to be an issue with registry auth? Potentially to ECR specifically?

Is your FROM line an ECR image? Public Hub image? It could be that a hang is occurring there due to ECR doing things a bit differently than Docker Hub in terms of auth.

[bfallik]

@nathanleclaire It might be related to ECR though in my case I wasn't actually building an image for ECR. The Dockerfiles in use don't reference any ECR URLs. The two images I was trying to build were based on FROM golang:1.8 and FROM postgres:9.5.

I agree it might be hanging trying to auth with ECR but it's strange that it's trying to do that in the first place. Maybe that's part of the problem.

[vohtaski]

In my case, .git folder had an issue, when I added it to .dockerignore, it started to work

[fazalerabbi]

I was facing same issue, docker-compose build hang, and reason at my end was I didn't have enough space. After creating some space its start working.

[sadjow]

I was facing this issue as well. The problem was that I needed to add the ssh private keys to ssh agent. ssh-add private-key.pem.

[augnustin]

Had the issue, and as for @dmoliveira it was because of a too heavy repo.

Adding big files to .dockerignore solved it!

Maybe a warning message when the loading context step takes more than x seconds would be helpful?

Cheers

[intellix]

I'm getting this now and it seems to be caused by recently using gcloud init. Since then, I've got about 20 entries inside my ~/.docker/config file and it's trying to authenticate with them all before building anything. I don't even reference Google Cloud whatsoever in my build file so I dunno why it's authenticating with 20 servers first

[roberthills]

This was sending me mental. In the end, I had to remove my php application from my project folder. Therefore, I'll be adding it to my .dockerignore.

[lloudermilk-ww]

I was seeing the same issue - EOF on step x/y while running docker-compose up --build.

Thanks to all you guys I looked in my repo for the largest files - found a test.log that was nearly 10GB... .gitignored, but not .dockerignored. Once I added it to the .dockerignore my build sped up abt 3x as fast and finished no problem.

Beefing up my .dockerignore now...

[w2ak]

Hey :)

Just in case, I had the same problem but very different reason. In my docker-compose folder there's a data folder that should have been ignored and is HUGE.
The hanging was because of "sending context to the docker daemon" (more than 20G of context...), and this message was not displayed by docker-compose --verbose.

[rc1021]

嗨, 我的情況跟 @w2ak 一樣。

在我的 docker-compose build 資料夾中，有一個被忽略的巨大檔案，請使用 .dockerignore 忽略它。

[glenuidev]

In my case there was a ton of extra stuff in the context directory bogging things down a bit...video output from cypress etc. git clean -dfx did quite a bit of cleanup targeting specifically the ignored files: node_modules, videos, screenshots etc. This did the trick

[ivanzotov]

Adding .dockerignore works, thanks #4396 (comment)

[rivernews]

In my case it is because I use string variable interpolation in .env but soon found out I can only use KEY=VALUE pairs, no interpolation allowed. Before this docker-compose --verbose build always stopped at something like send header send auth and got a SIGPIPE(13).

[MatteoGioioso]

In my case using --verbose it just hangs and says nothing...

[mistval]

My build was always hanging right at the start ("building xxxx"). I saw the suggestion to close Chrome, tried it, and the build immediately got unstuck.

I couldn't believe it worked.

[vluzrmos]

I just realized that my context (actual folder) was too big. So I moved my dockerfile to another folder and change my docker-compose.yml:

#...
services:
   app:
     build:
       context: ./another-folder-here
       dockerfile: YourDockerfileNameThatIsInsideAnotherFolderHereOrRemoveThisPropertyIfTheNameIsDockerfile

# ....

[wakeuphate]

I just realized that my context (actual folder) was too big. So I moved my dockerfile to another folder and change my docker-compose.yml:

#...
services:
   app:
     build:
       context: ./another-folder-here
       dockerfile: YourDockerfileNameThatIsInsideAnotherFolderHereOrRemoveThisPropertyIfTheNameIsDockerfile

# ....

This just fixed my build too. It seems maybe building is searching through directories for some reason and hanging, even when told which files to read? As soon as I moved my dockerfiles into a subfolder and modified the context rather than sitting in the root, my builds stopped hanging and ran perfectly.

[LiamAttClarke]

In my case, docker-compose was hanging on a massive node_modules tree. Ignoring node_modules with a .dockerignore file solved the issue.

[gaberogan]

in my case I was using WSL 2. it hanged sometimes. i think putting my files in the WSL file tree worked

[igorantun]

In my case (using WSL 2 as well), signing out from my Docker ID account fixed the issue. Using --verbose showed docker-compose was hanging at some account auth step.

[patricknazar]

I am having the symptom described by @igorantun , where it stops at an auth step, in my case docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'https://index.docker.io/v1/' I've tried signing in and out, uninstalling, reinstalling, reset factory settings, no go. Only running as root works. I beg for help.

Edit: I THOUGHT I found a fix. In an elevated powershell echo https://index.docker.io/v1/ | docker-credential-desktop.exe erase. I guessed the credentials helper was getting stuck when not running as root. Tried to run this command as root on my ubuntu WSL but apparently that wasn't enough, had to do it in admin windows. It worked once but now it's no longer working. Hope someone gains insight from this for a fix..

Edit: Refer to docker/for-win#7808

[igorantun]

For anyone having the same issue with WSL, the correct Docker for Windows issue @patricknazar mentioned is the following: docker/for-win#7808

As it seems, a recent Windows (Insider) release broke clip.exe, consequentially breaking docker-credential-desktop.exe.
This bug is being discussed at microsoft/WSL#5648

Also, this note present at the Docker for Windows issue worked for me:

Observation:
Removing ~/.docker/config.json seems to solve the issue, but this file is re-created after restarting Docker or rebooting Windows.

[gordonbanderson]

I ran into this issue, for a docker-compose build that processes images to animate sequences. I had a subdir of 66GB, adding that to .dockerignore resolved my hanging build

[ramytamer]

I ran to this issue, the problem was the files inside my log folder in a Rails application was too big.

Deleting these big files and adding them to .dockerignore fixed it.

[jensneuhaus]

Had the issue, and as for @dmoliveira it was because of a too heavy repo.
Adding big files to .dockerignore solved it!

@augnustin Wow, thanks a lot! This is a really tricky, hard to find error but your comment helped!

[saiabinesh]

I just realized that my context (actual folder) was too big. So I moved my dockerfile to another folder and change my docker-compose.yml:

#...
services:
   app:
     build:
       context: ./another-folder-here
       dockerfile: YourDockerfileNameThatIsInsideAnotherFolderHereOrRemoveThisPropertyIfTheNameIsDockerfile

# ....

You just saved my life. In 2021!