Docker 1.9.1 hanging at build step "Setting up ca-certificates-java"

[jredl-va]

A few of us within the office upgraded to the latest version of docker toolbox backed by Docker 1.9.1 and builds are hanging as per the below build output.

docker version:

 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

docker info:

Containers: 10
Images: 57
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 77
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: vbootstrap-vm
ID: LLM6:CASZ:KOD3:646A:XPRK:PIVB:VGJ5:JSDB:ZKAN:OUC4:E2AK:FFTC
Debug mode (server): true
 File Descriptors: 13
 Goroutines: 18
 System Time: 2015-11-24T02:03:35.597772191Z
 EventsListeners: 0
 Init SHA1: 
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

uname -a:

Darwin JRedl-MB-Pro.local 15.0.0 Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64

Here is a snippet from the docker build uppet that hangs on the Setting up ca-certificates-java line. Something to do with the latest version of docker and openjdk?

update-alternatives: using /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
Setting up ca-certificates-java (20140324) ...

Docker file example:

FROM gcr.io/google_appengine/base

# Prepare the image.
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y -qq --no-install-recommends build-essential wget curl unzip python python-dev php5-mysql php5-cli php5-cgi openjdk-7-jre-headless openssh-client python-openssl && apt-get clean

I can confirm that this is not an issue with Docker 1.9.0 or Docker Toolbox 1.9.0d. Let me know if I can provide any further information but this feels like a regression of some sort within the new version.

[bsao]

I am facing same problem. I am investigating.

[Jberlinsky]

We're facing the same problem as well.

[bsao]

Yep, it is a problem em docker 1.9. I had downgraded to 1.8.3 and all problems solved. Now i am investigating a workarround. will post here! Tks

[alex-jestin-taylor]

I'm having the same issue with docker 1.9.1a

[bean5]

I have docker 1.8.3, so maybe the process of installing a different version of docker remedies the situation. @bsao.

[loebpaul]

having this same issue with docker version 1.9.1, build a34a1d5

[crosbymichael]

Are you only seeing this on boot2docker?

[crosbymichael]

I cannot repo on a stock ubuntu with aufs or on my machine. let me try with boot2docker to see if I can repo there.

[marianmoldovan]

+1 in Docker 1.9.1 for ubuntu:14.10 using OSX

[bean5]

This is an issue that started appearing after I turned on VPN for work. Even after I turned off VPN and restarted the docker machine on OSX it continued to have this problem. I re-installed Docker 1.9.1 and then 1.8.3, still seeing the issue. Blocks me from using most if not all of my dockers on the Mac.

[maxwellpeterson-wf]

+1 in Docker 1.9.1 for ubuntu 12.04 using OS X 10.11

[chico1198]

Developers in my office came across this by accident too.

This version/build worked: Docker version 1.9.0, build 76d6bc9

This version/build hung:Docker version 1.9.1, build a34a1d5

[jredl-va]

@crosbymichael I unfortunately have not tried it on any other environment than Boot2Docker.

[bean5]

Someone with the know-how of git-bisecting and docker could use the build IDs provided by @chico1198!

[abuechler]

I experienced the same problem with 1.9.1 on OSX El Capitan, downgrading to 1.9.0 didn't help.

[dannymcpherson]

Same issue here on OSX 10.9.3 with:
Docker version 1.9.1, build a34a1d5
Docker version 1.9.0, build 76d6bc9

[sevein]

@crosbymichael I logged in boot2docker and ran ps auxf, this is what I saw:

root      1290  0.4  1.8 1346656 75692 ?       Sl   Nov27   4:53 /usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// -H tcp://0.0.0.0:2376 [...]
root      8556  0.0  0.0      0     0 ?        Ss   05:12   0:00  \_ [sh]
root     24221 99.8  0.0      0     0 ?        Zl   05:33  64:17  |   \_ [java] <defunct>
root     24657  0.0  0.0      0     0 ?        Ss   06:07   0:00  \_ [sh]
root      6174 79.6  0.0      0     0 ?        Zl   06:22  12:33      \_ [java] <defunct>
root      7295 49.3  0.0      0     0 ?        Zl   06:32   2:49      \_ [java] <defunct>

[osterman]

+1 with docker 1.9.1 on OSX 10.11 with attempting to build image from ubuntu 14.04

[junxy]

+1
use DockerToolbox-1.9.1a.pkg

docker version                                                                                      2 master?
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

[osterman]

Downgrading to Docker 1.8.3 is my temporary workaround. Here's the target I use in my Makefile.

downgrade-docker:
  docker-machine ssh $(DOCKER_MACHINE_NAME) sudo /etc/init.d/docker stop
  docker-machine ssh $(DOCKER_MACHINE_NAME) "while sudo /etc/init.d/docker status ; do sleep 1; done"
  docker-machine ssh $(DOCKER_MACHINE_NAME) "sudo curl 'https://get.docker.com/builds/Linux/x86_64/docker-1.8.3' -o /usr/local/bin/docker-1.8.3"
  docker-machine ssh $(DOCKER_MACHINE_NAME) "sudo ln -sf /usr/local/bin/docker-1.8.3 /usr/local/bin/docker"
  # FIXME: Starting machine is not enough; always fails with message like "Need TLS certs for 127.0.0.1,10.0.2.15,192.168.99.100"
  #docker-machine ssh $(DOCKER_MACHINE_NAME) sudo /etc/init.d/docker start
  docker-machine stop $(DOCKER_MACHINE_NAME) 
  docker-machine start $(DOCKER_MACHINE_NAME) 

[tiborvass]

I couldn't reproduce this. Does it always hang at "setting up certificates" ? Did you try sending a ^D to close some pipe? Can you also try sending a SIGUSR1 to the daemon and paste the stack trace here when it's stuck?

[leafjiang]

+1 with docker 1.9.1 on OS X 10.10

I tried downgrading to 1.8.3 using @osterman 's Makefile and also had troubles with the SSH key:

ip-10-100-0-211:docker-dev leaf$ docker-machine start default
(default) OUT | Starting VM...
Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded

[carsten-ulrich-saitow-ag]

Tested it by doing different openjdk installs inside debian:jessie and ubuntu
OSX 10.11.1, boot2docker 1.9.1: hangs
OSX 10.11.1, boot2docker 1.9.0: works
Ubuntu 14.04 with docker 1.9.1: works

The boot2docker vms were created with:
docker-machine create -d virtualbox --virtualbox-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v1.9.0/boot2docker.iso
and
docker-machine create -d virtualbox --virtualbox-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v1.9.1/boot2docker.iso

On Ubuntu 14.04 docker was installed following the documentation on https://docs.docker.com/engine/installation/ubuntulinux/

[Lohhari]

+1, running docker 1.9.1 build a34a1d5 on OSX Yosemite 10.10.5.

[cpuguy83]

I can't reproduce this.

[patsa]

Same issue here.
Is there any way to downgrade to an earlier version on Windows?

[patsa]

Found it myself.
https://github.com/docker/docker/releases

[noisy]

upgrading kernel of 14.04LTS worked for me 👍

[Eyjafjallajokull]

I'm on OSX on Boot2Docker version 1.10.2, build master : 611be10, Docker version 1.10.2, build c3959b1 and first got this from docker-compose:

Recreating docker_preview_1
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Then tried docker kill 38e1e2590dfa but process hangs forever. docker.log:

time="2016-03-09T14:49:13.053004077Z" level=debug msg="Calling POST /v1.21/containers/38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b/stop"
time="2016-03-09T14:49:13.053058084Z" level=debug msg="POST /v1.21/containers/38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b/stop?t=10"
time="2016-03-09T14:49:13.053097711Z" level=debug msg="Sending 15 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:49:23.053530062Z" level=info msg="Container 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b failed to exit within 10 seconds of SIGTERM - using the force"
time="2016-03-09T14:49:23.053720529Z" level=debug msg="Sending 9 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:49:33.054082100Z" level=info msg="Container 38e1e2590dfa failed to exit within 10 seconds of kill - trying direct SIGKILL"
time="2016-03-09T14:49:34.254353402Z" level=debug msg="Calling GET /v1.22/containers/json"
time="2016-03-09T14:49:34.254413283Z" level=debug msg="GET /v1.22/containers/json"
time="2016-03-09T14:49:54.293708866Z" level=debug msg="Calling POST /v1.22/containers/38e1e2590dfa/kill"
time="2016-03-09T14:49:54.293752784Z" level=debug msg="POST /v1.22/containers/38e1e2590dfa/kill?signal=KILL"
time="2016-03-09T14:49:54.293802705Z" level=debug msg="Sending 9 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:50:04.294276946Z" level=info msg="Container 38e1e2590dfa failed to exit within 10 seconds of kill - trying direct SIGKILL"
time="2016-03-09T14:50:26.678957119Z" level=debug msg="clean 3 unused exec commands"

[einhverfr]

Just as a note (I know this is closed but not sure if it makes sense to open as a new issue). I was having the same issue on a later version until I switched to devmapper.

$ docker info
Containers: 4
 Running: 3
 Paused: 0
 Stopped: 1
Images: 81
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-8:1-9044034-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.726 GB
 Data Space Total: 107.4 GB
 Data Space Available: 96.43 GB
 Metadata Space Used: 4.387 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.56 GiB
Name: ravn
ID: L2WX:3RQ7:W6IC:7MY3:M3ZC:7MP2:3ZMP:VHW4:TLXM:VLYO:NNZ5:2FVW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

[AkihiroSuda]

@einhverfr The issue is fixed in kernel 3.13.0-79.123 (your one seems to be 3.13.0-77)

[martinm82]

Can this issue really be solved with a Kernel upgrade? We are encountering the same problem with Docker 1.9.1 on Ubuntu 14.04 with Kernel 3.13.0-83-generic.

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

[thaJeztah]

@martinm82 yes, this issue was a kernel issue. It's possible something else can result in a similar behavior, or if there's a regression in the kernel. However, please open a new issue if you're having issues on the current release; keep in mind that docker 1.9.1 is EOL, so won't be receiving updates anymore.

[thaJeztah]

I am locking the discussion on this issue, because the original issue here was resolved, and I want to prevent this issue from collecting possibly unrelated issues. See this comment; #18180 (comment) for the kernel versions needed to fix this issue