Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker 1.9.1 hanging at build step "Setting up ca-certificates-java" #18180

Closed
jredl-va opened this issue Nov 24, 2015 · 258 comments
Closed

Docker 1.9.1 hanging at build step "Setting up ca-certificates-java" #18180

jredl-va opened this issue Nov 24, 2015 · 258 comments

Comments

@jredl-va
Copy link

@jredl-va jredl-va commented Nov 24, 2015

A few of us within the office upgraded to the latest version of docker toolbox backed by Docker 1.9.1 and builds are hanging as per the below build output.

docker version:

 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

docker info:

Containers: 10
Images: 57
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 77
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: vbootstrap-vm
ID: LLM6:CASZ:KOD3:646A:XPRK:PIVB:VGJ5:JSDB:ZKAN:OUC4:E2AK:FFTC
Debug mode (server): true
 File Descriptors: 13
 Goroutines: 18
 System Time: 2015-11-24T02:03:35.597772191Z
 EventsListeners: 0
 Init SHA1: 
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

uname -a:

Darwin JRedl-MB-Pro.local 15.0.0 Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64

Here is a snippet from the docker build uppet that hangs on the Setting up ca-certificates-java line. Something to do with the latest version of docker and openjdk?

update-alternatives: using /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
Setting up ca-certificates-java (20140324) ...

Docker file example:

FROM gcr.io/google_appengine/base

# Prepare the image.
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y -qq --no-install-recommends build-essential wget curl unzip python python-dev php5-mysql php5-cli php5-cgi openjdk-7-jre-headless openssh-client python-openssl && apt-get clean

I can confirm that this is not an issue with Docker 1.9.0 or Docker Toolbox 1.9.0d. Let me know if I can provide any further information but this feels like a regression of some sort within the new version.

@bsao
Copy link

@bsao bsao commented Nov 24, 2015

I am facing same problem. I am investigating.

@Jberlinsky
Copy link

@Jberlinsky Jberlinsky commented Nov 24, 2015

We're facing the same problem as well.

@bsao
Copy link

@bsao bsao commented Nov 24, 2015

Yep, it is a problem em docker 1.9. I had downgraded to 1.8.3 and all problems solved. Now i am investigating a workarround. will post here! Tks

@alex-jestin-taylor
Copy link

@alex-jestin-taylor alex-jestin-taylor commented Nov 24, 2015

I'm having the same issue with docker 1.9.1a

@bean5
Copy link

@bean5 bean5 commented Nov 24, 2015

I have docker 1.8.3, so maybe the process of installing a different version of docker remedies the situation. @bsao.

@loebpaul
Copy link

@loebpaul loebpaul commented Nov 24, 2015

having this same issue with docker version 1.9.1, build a34a1d5

@crosbymichael
Copy link
Contributor

@crosbymichael crosbymichael commented Nov 25, 2015

Are you only seeing this on boot2docker?

@crosbymichael
Copy link
Contributor

@crosbymichael crosbymichael commented Nov 25, 2015

I cannot repo on a stock ubuntu with aufs or on my machine. let me try with boot2docker to see if I can repo there.

@marianmoldovan
Copy link

@marianmoldovan marianmoldovan commented Nov 25, 2015

+1 in Docker 1.9.1 for ubuntu:14.10 using OSX

@bean5
Copy link

@bean5 bean5 commented Nov 25, 2015

This is an issue that started appearing after I turned on VPN for work. Even after I turned off VPN and restarted the docker machine on OSX it continued to have this problem. I re-installed Docker 1.9.1 and then 1.8.3, still seeing the issue. Blocks me from using most if not all of my dockers on the Mac.

@maxwellpeterson-wf
Copy link

@maxwellpeterson-wf maxwellpeterson-wf commented Nov 25, 2015

+1 in Docker 1.9.1 for ubuntu 12.04 using OS X 10.11

@chico1198
Copy link

@chico1198 chico1198 commented Nov 25, 2015

Developers in my office came across this by accident too.

This version/build worked: Docker version 1.9.0, build 76d6bc9

This version/build hung:Docker version 1.9.1, build a34a1d5

@jredl-va
Copy link
Author

@jredl-va jredl-va commented Nov 26, 2015

@crosbymichael I unfortunately have not tried it on any other environment than Boot2Docker.

@bean5
Copy link

@bean5 bean5 commented Nov 27, 2015

Someone with the know-how of git-bisecting and docker could use the build IDs provided by @chico1198!

@abuechler
Copy link

@abuechler abuechler commented Nov 27, 2015

I experienced the same problem with 1.9.1 on OSX El Capitan, downgrading to 1.9.0 didn't help.

@dannymcpherson
Copy link

@dannymcpherson dannymcpherson commented Nov 27, 2015

Same issue here on OSX 10.9.3 with:
Docker version 1.9.1, build a34a1d5
Docker version 1.9.0, build 76d6bc9

@sevein
Copy link

@sevein sevein commented Nov 28, 2015

@crosbymichael I logged in boot2docker and ran ps auxf, this is what I saw:

root      1290  0.4  1.8 1346656 75692 ?       Sl   Nov27   4:53 /usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// -H tcp://0.0.0.0:2376 [...]
root      8556  0.0  0.0      0     0 ?        Ss   05:12   0:00  \_ [sh]
root     24221 99.8  0.0      0     0 ?        Zl   05:33  64:17  |   \_ [java] <defunct>
root     24657  0.0  0.0      0     0 ?        Ss   06:07   0:00  \_ [sh]
root      6174 79.6  0.0      0     0 ?        Zl   06:22  12:33      \_ [java] <defunct>
root      7295 49.3  0.0      0     0 ?        Zl   06:32   2:49      \_ [java] <defunct>
@osterman
Copy link

@osterman osterman commented Nov 29, 2015

+1 with docker 1.9.1 on OSX 10.11 with attempting to build image from ubuntu 14.04

@junxy
Copy link

@junxy junxy commented Nov 30, 2015

+1
use DockerToolbox-1.9.1a.pkg

docker version                                                                                      2 master?
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64
@osterman
Copy link

@osterman osterman commented Nov 30, 2015

Downgrading to Docker 1.8.3 is my temporary workaround. Here's the target I use in my Makefile.

downgrade-docker:
  docker-machine ssh $(DOCKER_MACHINE_NAME) sudo /etc/init.d/docker stop
  docker-machine ssh $(DOCKER_MACHINE_NAME) "while sudo /etc/init.d/docker status ; do sleep 1; done"
  docker-machine ssh $(DOCKER_MACHINE_NAME) "sudo curl 'https://get.docker.com/builds/Linux/x86_64/docker-1.8.3' -o /usr/local/bin/docker-1.8.3"
  docker-machine ssh $(DOCKER_MACHINE_NAME) "sudo ln -sf /usr/local/bin/docker-1.8.3 /usr/local/bin/docker"
  # FIXME: Starting machine is not enough; always fails with message like "Need TLS certs for 127.0.0.1,10.0.2.15,192.168.99.100"
  #docker-machine ssh $(DOCKER_MACHINE_NAME) sudo /etc/init.d/docker start
  docker-machine stop $(DOCKER_MACHINE_NAME) 
  docker-machine start $(DOCKER_MACHINE_NAME) 
@thaJeztah thaJeztah added this to the 1.9.2 milestone Nov 30, 2015
@tiborvass
Copy link
Collaborator

@tiborvass tiborvass commented Nov 30, 2015

I couldn't reproduce this. Does it always hang at "setting up certificates" ? Did you try sending a ^D to close some pipe? Can you also try sending a SIGUSR1 to the daemon and paste the stack trace here when it's stuck?

@leafjiang
Copy link

@leafjiang leafjiang commented Nov 30, 2015

+1 with docker 1.9.1 on OS X 10.10

I tried downgrading to 1.8.3 using @osterman 's Makefile and also had troubles with the SSH key:

ip-10-100-0-211:docker-dev leaf$ docker-machine start default
(default) OUT | Starting VM...
Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded
@carsten-ulrich-saitow-ag

Tested it by doing different openjdk installs inside debian:jessie and ubuntu
OSX 10.11.1, boot2docker 1.9.1: hangs
OSX 10.11.1, boot2docker 1.9.0: works
Ubuntu 14.04 with docker 1.9.1: works

The boot2docker vms were created with:
docker-machine create -d virtualbox --virtualbox-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v1.9.0/boot2docker.iso
and
docker-machine create -d virtualbox --virtualbox-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v1.9.1/boot2docker.iso

On Ubuntu 14.04 docker was installed following the documentation on https://docs.docker.com/engine/installation/ubuntulinux/

@Lohhari
Copy link

@Lohhari Lohhari commented Nov 30, 2015

+1, running docker 1.9.1 build a34a1d5 on OSX Yosemite 10.10.5.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Nov 30, 2015

I can't reproduce this.

@patsa
Copy link

@patsa patsa commented Nov 30, 2015

Same issue here.
Is there any way to downgrade to an earlier version on Windows?

@patsa
Copy link

@patsa patsa commented Nov 30, 2015

@noisy
Copy link

@noisy noisy commented Mar 8, 2016

upgrading kernel of 14.04LTS worked for me 👍

@Eyjafjallajokull
Copy link

@Eyjafjallajokull Eyjafjallajokull commented Mar 9, 2016

I'm on OSX on Boot2Docker version 1.10.2, build master : 611be10, Docker version 1.10.2, build c3959b1 and first got this from docker-compose:

Recreating docker_preview_1
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Then tried docker kill 38e1e2590dfa but process hangs forever. docker.log:

time="2016-03-09T14:49:13.053004077Z" level=debug msg="Calling POST /v1.21/containers/38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b/stop"
time="2016-03-09T14:49:13.053058084Z" level=debug msg="POST /v1.21/containers/38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b/stop?t=10"
time="2016-03-09T14:49:13.053097711Z" level=debug msg="Sending 15 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:49:23.053530062Z" level=info msg="Container 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b failed to exit within 10 seconds of SIGTERM - using the force"
time="2016-03-09T14:49:23.053720529Z" level=debug msg="Sending 9 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:49:33.054082100Z" level=info msg="Container 38e1e2590dfa failed to exit within 10 seconds of kill - trying direct SIGKILL"
time="2016-03-09T14:49:34.254353402Z" level=debug msg="Calling GET /v1.22/containers/json"
time="2016-03-09T14:49:34.254413283Z" level=debug msg="GET /v1.22/containers/json"
time="2016-03-09T14:49:54.293708866Z" level=debug msg="Calling POST /v1.22/containers/38e1e2590dfa/kill"
time="2016-03-09T14:49:54.293752784Z" level=debug msg="POST /v1.22/containers/38e1e2590dfa/kill?signal=KILL"
time="2016-03-09T14:49:54.293802705Z" level=debug msg="Sending 9 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:50:04.294276946Z" level=info msg="Container 38e1e2590dfa failed to exit within 10 seconds of kill - trying direct SIGKILL"
time="2016-03-09T14:50:26.678957119Z" level=debug msg="clean 3 unused exec commands"
@einhverfr
Copy link

@einhverfr einhverfr commented Aug 22, 2016

Just as a note (I know this is closed but not sure if it makes sense to open as a new issue). I was having the same issue on a later version until I switched to devmapper.

$ docker info
Containers: 4
 Running: 3
 Paused: 0
 Stopped: 1
Images: 81
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-8:1-9044034-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.726 GB
 Data Space Total: 107.4 GB
 Data Space Available: 96.43 GB
 Metadata Space Used: 4.387 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.56 GiB
Name: ravn
ID: L2WX:3RQ7:W6IC:7MY3:M3ZC:7MP2:3ZMP:VHW4:TLXM:VLYO:NNZ5:2FVW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
@AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda commented Aug 22, 2016

@einhverfr The issue is fixed in kernel 3.13.0-79.123 (your one seems to be 3.13.0-77)

@martinm82
Copy link

@martinm82 martinm82 commented Feb 14, 2017

Can this issue really be solved with a Kernel upgrade? We are encountering the same problem with Docker 1.9.1 on Ubuntu 14.04 with Kernel 3.13.0-83-generic.

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Feb 14, 2017

@martinm82 yes, this issue was a kernel issue. It's possible something else can result in a similar behavior, or if there's a regression in the kernel. However, please open a new issue if you're having issues on the current release; keep in mind that docker 1.9.1 is EOL, so won't be receiving updates anymore.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Feb 14, 2017

I am locking the discussion on this issue, because the original issue here was resolved, and I want to prevent this issue from collecting possibly unrelated issues. See this comment; #18180 (comment) for the kernel versions needed to fix this issue

@moby moby locked and limited conversation to collaborators Feb 14, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.