New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker 1.9.1 hanging at build step "Setting up ca-certificates-java" #18180

Closed
jredl-va opened this Issue Nov 24, 2015 · 258 comments

Comments

Projects
None yet
@jredl-va

jredl-va commented Nov 24, 2015

A few of us within the office upgraded to the latest version of docker toolbox backed by Docker 1.9.1 and builds are hanging as per the below build output.

docker version:

 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

docker info:

Containers: 10
Images: 57
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 77
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: vbootstrap-vm
ID: LLM6:CASZ:KOD3:646A:XPRK:PIVB:VGJ5:JSDB:ZKAN:OUC4:E2AK:FFTC
Debug mode (server): true
 File Descriptors: 13
 Goroutines: 18
 System Time: 2015-11-24T02:03:35.597772191Z
 EventsListeners: 0
 Init SHA1: 
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

uname -a:

Darwin JRedl-MB-Pro.local 15.0.0 Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64

Here is a snippet from the docker build uppet that hangs on the Setting up ca-certificates-java line. Something to do with the latest version of docker and openjdk?

update-alternatives: using /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
Setting up ca-certificates-java (20140324) ...

Docker file example:

FROM gcr.io/google_appengine/base

# Prepare the image.
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y -qq --no-install-recommends build-essential wget curl unzip python python-dev php5-mysql php5-cli php5-cgi openjdk-7-jre-headless openssh-client python-openssl && apt-get clean

I can confirm that this is not an issue with Docker 1.9.0 or Docker Toolbox 1.9.0d. Let me know if I can provide any further information but this feels like a regression of some sort within the new version.

@bsao

This comment has been minimized.

Show comment
Hide comment
@bsao

bsao Nov 24, 2015

I am facing same problem. I am investigating.

bsao commented Nov 24, 2015

I am facing same problem. I am investigating.

@Jberlinsky

This comment has been minimized.

Show comment
Hide comment
@Jberlinsky

Jberlinsky Nov 24, 2015

We're facing the same problem as well.

Jberlinsky commented Nov 24, 2015

We're facing the same problem as well.

@bsao

This comment has been minimized.

Show comment
Hide comment
@bsao

bsao Nov 24, 2015

Yep, it is a problem em docker 1.9. I had downgraded to 1.8.3 and all problems solved. Now i am investigating a workarround. will post here! Tks

bsao commented Nov 24, 2015

Yep, it is a problem em docker 1.9. I had downgraded to 1.8.3 and all problems solved. Now i am investigating a workarround. will post here! Tks

@alex-jestin-taylor

This comment has been minimized.

Show comment
Hide comment
@alex-jestin-taylor

alex-jestin-taylor Nov 24, 2015

I'm having the same issue with docker 1.9.1a

alex-jestin-taylor commented Nov 24, 2015

I'm having the same issue with docker 1.9.1a

@bean5

This comment has been minimized.

Show comment
Hide comment
@bean5

bean5 Nov 24, 2015

I have docker 1.8.3, so maybe the process of installing a different version of docker remedies the situation. @bsao.

bean5 commented Nov 24, 2015

I have docker 1.8.3, so maybe the process of installing a different version of docker remedies the situation. @bsao.

@loebpaul

This comment has been minimized.

Show comment
Hide comment
@loebpaul

loebpaul Nov 24, 2015

having this same issue with docker version 1.9.1, build a34a1d5

loebpaul commented Nov 24, 2015

having this same issue with docker version 1.9.1, build a34a1d5

@crosbymichael

This comment has been minimized.

Show comment
Hide comment
@crosbymichael

crosbymichael Nov 25, 2015

Contributor

Are you only seeing this on boot2docker?

Contributor

crosbymichael commented Nov 25, 2015

Are you only seeing this on boot2docker?

@crosbymichael

This comment has been minimized.

Show comment
Hide comment
@crosbymichael

crosbymichael Nov 25, 2015

Contributor

I cannot repo on a stock ubuntu with aufs or on my machine. let me try with boot2docker to see if I can repo there.

Contributor

crosbymichael commented Nov 25, 2015

I cannot repo on a stock ubuntu with aufs or on my machine. let me try with boot2docker to see if I can repo there.

@marianmoldovan

This comment has been minimized.

Show comment
Hide comment
@marianmoldovan

marianmoldovan Nov 25, 2015

+1 in Docker 1.9.1 for ubuntu:14.10 using OSX

marianmoldovan commented Nov 25, 2015

+1 in Docker 1.9.1 for ubuntu:14.10 using OSX

@bean5

This comment has been minimized.

Show comment
Hide comment
@bean5

bean5 Nov 25, 2015

This is an issue that started appearing after I turned on VPN for work. Even after I turned off VPN and restarted the docker machine on OSX it continued to have this problem. I re-installed Docker 1.9.1 and then 1.8.3, still seeing the issue. Blocks me from using most if not all of my dockers on the Mac.

bean5 commented Nov 25, 2015

This is an issue that started appearing after I turned on VPN for work. Even after I turned off VPN and restarted the docker machine on OSX it continued to have this problem. I re-installed Docker 1.9.1 and then 1.8.3, still seeing the issue. Blocks me from using most if not all of my dockers on the Mac.

@maxwellpeterson-wf

This comment has been minimized.

Show comment
Hide comment
@maxwellpeterson-wf

maxwellpeterson-wf Nov 25, 2015

+1 in Docker 1.9.1 for ubuntu 12.04 using OS X 10.11

maxwellpeterson-wf commented Nov 25, 2015

+1 in Docker 1.9.1 for ubuntu 12.04 using OS X 10.11

@chico1198

This comment has been minimized.

Show comment
Hide comment
@chico1198

chico1198 Nov 25, 2015

Developers in my office came across this by accident too.

This version/build worked: Docker version 1.9.0, build 76d6bc9

This version/build hung:Docker version 1.9.1, build a34a1d5

chico1198 commented Nov 25, 2015

Developers in my office came across this by accident too.

This version/build worked: Docker version 1.9.0, build 76d6bc9

This version/build hung:Docker version 1.9.1, build a34a1d5

@jredl-va

This comment has been minimized.

Show comment
Hide comment
@jredl-va

jredl-va Nov 26, 2015

@crosbymichael I unfortunately have not tried it on any other environment than Boot2Docker.

jredl-va commented Nov 26, 2015

@crosbymichael I unfortunately have not tried it on any other environment than Boot2Docker.

@bean5

This comment has been minimized.

Show comment
Hide comment
@bean5

bean5 Nov 27, 2015

Someone with the know-how of git-bisecting and docker could use the build IDs provided by @chico1198!

bean5 commented Nov 27, 2015

Someone with the know-how of git-bisecting and docker could use the build IDs provided by @chico1198!

@abuechler

This comment has been minimized.

Show comment
Hide comment
@abuechler

abuechler Nov 27, 2015

I experienced the same problem with 1.9.1 on OSX El Capitan, downgrading to 1.9.0 didn't help.

abuechler commented Nov 27, 2015

I experienced the same problem with 1.9.1 on OSX El Capitan, downgrading to 1.9.0 didn't help.

@dannymcpherson

This comment has been minimized.

Show comment
Hide comment
@dannymcpherson

dannymcpherson Nov 27, 2015

Same issue here on OSX 10.9.3 with:
Docker version 1.9.1, build a34a1d5
Docker version 1.9.0, build 76d6bc9

dannymcpherson commented Nov 27, 2015

Same issue here on OSX 10.9.3 with:
Docker version 1.9.1, build a34a1d5
Docker version 1.9.0, build 76d6bc9

@sevein

This comment has been minimized.

Show comment
Hide comment
@sevein

sevein Nov 28, 2015

@crosbymichael I logged in boot2docker and ran ps auxf, this is what I saw:

root      1290  0.4  1.8 1346656 75692 ?       Sl   Nov27   4:53 /usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// -H tcp://0.0.0.0:2376 [...]
root      8556  0.0  0.0      0     0 ?        Ss   05:12   0:00  \_ [sh]
root     24221 99.8  0.0      0     0 ?        Zl   05:33  64:17  |   \_ [java] <defunct>
root     24657  0.0  0.0      0     0 ?        Ss   06:07   0:00  \_ [sh]
root      6174 79.6  0.0      0     0 ?        Zl   06:22  12:33      \_ [java] <defunct>
root      7295 49.3  0.0      0     0 ?        Zl   06:32   2:49      \_ [java] <defunct>

sevein commented Nov 28, 2015

@crosbymichael I logged in boot2docker and ran ps auxf, this is what I saw:

root      1290  0.4  1.8 1346656 75692 ?       Sl   Nov27   4:53 /usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// -H tcp://0.0.0.0:2376 [...]
root      8556  0.0  0.0      0     0 ?        Ss   05:12   0:00  \_ [sh]
root     24221 99.8  0.0      0     0 ?        Zl   05:33  64:17  |   \_ [java] <defunct>
root     24657  0.0  0.0      0     0 ?        Ss   06:07   0:00  \_ [sh]
root      6174 79.6  0.0      0     0 ?        Zl   06:22  12:33      \_ [java] <defunct>
root      7295 49.3  0.0      0     0 ?        Zl   06:32   2:49      \_ [java] <defunct>
@osterman

This comment has been minimized.

Show comment
Hide comment
@osterman

osterman Nov 29, 2015

+1 with docker 1.9.1 on OSX 10.11 with attempting to build image from ubuntu 14.04

osterman commented Nov 29, 2015

+1 with docker 1.9.1 on OSX 10.11 with attempting to build image from ubuntu 14.04

@junxy

This comment has been minimized.

Show comment
Hide comment
@junxy

junxy Nov 30, 2015

+1
use DockerToolbox-1.9.1a.pkg

docker version                                                                                      2 master?
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

junxy commented Nov 30, 2015

+1
use DockerToolbox-1.9.1a.pkg

docker version                                                                                      2 master?
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64
@osterman

This comment has been minimized.

Show comment
Hide comment
@osterman

osterman Nov 30, 2015

Downgrading to Docker 1.8.3 is my temporary workaround. Here's the target I use in my Makefile.

downgrade-docker:
  docker-machine ssh $(DOCKER_MACHINE_NAME) sudo /etc/init.d/docker stop
  docker-machine ssh $(DOCKER_MACHINE_NAME) "while sudo /etc/init.d/docker status ; do sleep 1; done"
  docker-machine ssh $(DOCKER_MACHINE_NAME) "sudo curl 'https://get.docker.com/builds/Linux/x86_64/docker-1.8.3' -o /usr/local/bin/docker-1.8.3"
  docker-machine ssh $(DOCKER_MACHINE_NAME) "sudo ln -sf /usr/local/bin/docker-1.8.3 /usr/local/bin/docker"
  # FIXME: Starting machine is not enough; always fails with message like "Need TLS certs for 127.0.0.1,10.0.2.15,192.168.99.100"
  #docker-machine ssh $(DOCKER_MACHINE_NAME) sudo /etc/init.d/docker start
  docker-machine stop $(DOCKER_MACHINE_NAME) 
  docker-machine start $(DOCKER_MACHINE_NAME) 

osterman commented Nov 30, 2015

Downgrading to Docker 1.8.3 is my temporary workaround. Here's the target I use in my Makefile.

downgrade-docker:
  docker-machine ssh $(DOCKER_MACHINE_NAME) sudo /etc/init.d/docker stop
  docker-machine ssh $(DOCKER_MACHINE_NAME) "while sudo /etc/init.d/docker status ; do sleep 1; done"
  docker-machine ssh $(DOCKER_MACHINE_NAME) "sudo curl 'https://get.docker.com/builds/Linux/x86_64/docker-1.8.3' -o /usr/local/bin/docker-1.8.3"
  docker-machine ssh $(DOCKER_MACHINE_NAME) "sudo ln -sf /usr/local/bin/docker-1.8.3 /usr/local/bin/docker"
  # FIXME: Starting machine is not enough; always fails with message like "Need TLS certs for 127.0.0.1,10.0.2.15,192.168.99.100"
  #docker-machine ssh $(DOCKER_MACHINE_NAME) sudo /etc/init.d/docker start
  docker-machine stop $(DOCKER_MACHINE_NAME) 
  docker-machine start $(DOCKER_MACHINE_NAME) 

@thaJeztah thaJeztah added this to the 1.9.2 milestone Nov 30, 2015

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Nov 30, 2015

Collaborator

I couldn't reproduce this. Does it always hang at "setting up certificates" ? Did you try sending a ^D to close some pipe? Can you also try sending a SIGUSR1 to the daemon and paste the stack trace here when it's stuck?

Collaborator

tiborvass commented Nov 30, 2015

I couldn't reproduce this. Does it always hang at "setting up certificates" ? Did you try sending a ^D to close some pipe? Can you also try sending a SIGUSR1 to the daemon and paste the stack trace here when it's stuck?

@leafjiang

This comment has been minimized.

Show comment
Hide comment
@leafjiang

leafjiang Nov 30, 2015

+1 with docker 1.9.1 on OS X 10.10

I tried downgrading to 1.8.3 using @osterman 's Makefile and also had troubles with the SSH key:

ip-10-100-0-211:docker-dev leaf$ docker-machine start default
(default) OUT | Starting VM...
Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded

leafjiang commented Nov 30, 2015

+1 with docker 1.9.1 on OS X 10.10

I tried downgrading to 1.8.3 using @osterman 's Makefile and also had troubles with the SSH key:

ip-10-100-0-211:docker-dev leaf$ docker-machine start default
(default) OUT | Starting VM...
Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded
@carsten-ulrich-saitow-ag

This comment has been minimized.

Show comment
Hide comment
@carsten-ulrich-saitow-ag

carsten-ulrich-saitow-ag Nov 30, 2015

Tested it by doing different openjdk installs inside debian:jessie and ubuntu
OSX 10.11.1, boot2docker 1.9.1: hangs
OSX 10.11.1, boot2docker 1.9.0: works
Ubuntu 14.04 with docker 1.9.1: works

The boot2docker vms were created with:
docker-machine create -d virtualbox --virtualbox-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v1.9.0/boot2docker.iso
and
docker-machine create -d virtualbox --virtualbox-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v1.9.1/boot2docker.iso

On Ubuntu 14.04 docker was installed following the documentation on https://docs.docker.com/engine/installation/ubuntulinux/

Tested it by doing different openjdk installs inside debian:jessie and ubuntu
OSX 10.11.1, boot2docker 1.9.1: hangs
OSX 10.11.1, boot2docker 1.9.0: works
Ubuntu 14.04 with docker 1.9.1: works

The boot2docker vms were created with:
docker-machine create -d virtualbox --virtualbox-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v1.9.0/boot2docker.iso
and
docker-machine create -d virtualbox --virtualbox-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/download/v1.9.1/boot2docker.iso

On Ubuntu 14.04 docker was installed following the documentation on https://docs.docker.com/engine/installation/ubuntulinux/

@Lohhari

This comment has been minimized.

Show comment
Hide comment
@Lohhari

Lohhari Nov 30, 2015

+1, running docker 1.9.1 build a34a1d5 on OSX Yosemite 10.10.5.

Lohhari commented Nov 30, 2015

+1, running docker 1.9.1 build a34a1d5 on OSX Yosemite 10.10.5.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Nov 30, 2015

Contributor

I can't reproduce this.

Contributor

cpuguy83 commented Nov 30, 2015

I can't reproduce this.

@patsa

This comment has been minimized.

Show comment
Hide comment
@patsa

patsa Nov 30, 2015

Same issue here.
Is there any way to downgrade to an earlier version on Windows?

patsa commented Nov 30, 2015

Same issue here.
Is there any way to downgrade to an earlier version on Windows?

@patsa

This comment has been minimized.

Show comment
Hide comment

patsa commented Nov 30, 2015

@psanders

This comment has been minimized.

Show comment
Hide comment
@psanders

psanders Dec 1, 2015

+1, docker 1.9.1 @ El Capitan

psanders commented Dec 1, 2015

+1, docker 1.9.1 @ El Capitan

@virtualzone

This comment has been minimized.

Show comment
Hide comment
@virtualzone

virtualzone Dec 1, 2015

+1, Docker 1.9.1 on OS X 10.11.1

virtualzone commented Dec 1, 2015

+1, Docker 1.9.1 on OS X 10.11.1

@patrikjensen

This comment has been minimized.

Show comment
Hide comment
@patrikjensen

patrikjensen Dec 1, 2015

+1, Docker 1.9.1a, OS X 10.10.5

patrikjensen commented Dec 1, 2015

+1, Docker 1.9.1a, OS X 10.10.5

@jebbench

This comment has been minimized.

Show comment
Hide comment
@jebbench

jebbench Dec 1, 2015

+1, Docker 1.9.1 build a34a1d5, Windows 10

jebbench commented Dec 1, 2015

+1, Docker 1.9.1 build a34a1d5, Windows 10

@crazyball

This comment has been minimized.

Show comment
Hide comment
@crazyball

crazyball Dec 1, 2015

+1, Docker 1.9.1 build a34a1d5, OS X 10.11.1, Docker-Machine 0.5.1 build 7e8e38e

crazyball commented Dec 1, 2015

+1, Docker 1.9.1 build a34a1d5, OS X 10.11.1, Docker-Machine 0.5.1 build 7e8e38e

@rogierslag

This comment has been minimized.

Show comment
Hide comment
@rogierslag

rogierslag Dec 1, 2015

+1

Same on Docker-machine on OSX 10.11.1
Docker version 1.9.1, build a34a1d5
docker-machine version 0.5.1 (HEAD)

rogierslag commented Dec 1, 2015

+1

Same on Docker-machine on OSX 10.11.1
Docker version 1.9.1, build a34a1d5
docker-machine version 0.5.1 (HEAD)

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 1, 2015

Member

I'm able to reproduce this on docker-machine, OS X 10.10.5, so this may be something related to boot2docker. docker top also gives me <defunct> for a java process;

docker top dreamy_sammet                                                                  Tue Dec  1 15:58:47 2015
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                2538                1023                0                   14:44               ?                   00:00:00            /bin/sh -c apt-get update && apt-get install -y -qq --no-install-recommends build-essential wget curl unzip python python-dev php5-mysql php5-cli php5-cgi openjdk-7-jre-headless openssh-client python-openssl && apt-get clean
root                2566                2538                1                   14:44               ?                   00:00:16            apt-get install -y -qq --no-install-recommends build-essential wget curl unzip python python-dev php5-mysql php5-cli php5-cgi openjdk-7-jre-headless openssh-client python-openssl
root                4830                2566                0                   14:46               pts/0               00:00:00            /usr/bin/dpkg --status-fd 14 --configure libgdbm3:amd64 libjson-c2:amd64 libbsd0:amd64 libedit2:amd64 libkeyutils1:amd64 libkrb5support0:amd64 libk5crypto3:amd64 libkrb5-3:amd64 libgssapi-krb5-2:amd64 libidn11:amd64 libsasl2-modules-db:amd64 libsasl2-2:amd64 libldap-2.4-2:amd64 libmagic1:amd64 libsqlite3-0:amd64 libwrap0:amd64 libxml2:amd64 perl-modules:all perl:amd64 mime-support:all libexpat1:amd64 libpython2.7-stdlib:amd64 python2.7:amd64 libpython-stdlib:amd64 python:amd64 libasan1:amd64 libasyncns0:amd64 libatomic1:amd64 libavahi-common-data:amd64 libavahi-common3:amd64 libdbus-1-3:amd64 libavahi-client3:amd64 libcilkrts5:amd64 libisl10:amd64 libcloog-isl4:amd64 libcups2:amd64 librtmp1:amd64 libssh2-1:amd64 libcurl3:amd64 libogg0:amd64 libflac8:amd64 libpng12-0:amd64 libfreetype6:amd64 ucf:all fonts-dejavu-core:all fontconfig-config:all libfontconfig1:amd64 libglib2.0-0:amd64 libgomp1:amd64 x11-common:all libice6:amd64 libicu52:amd64 libitm1:amd64 liblcms2-2:amd64 liblsan0:amd64 libmpfr4:amd64 mysql-common:all libmysqlclient18:amd64 libnspr4:amd64 libnss3:amd64 libonig2:amd64 libpcsclite1:amd64 libsm6:amd64 libvorbis0a:amd64 libvorbisenc2:amd64 libsndfile1:amd64 libxau6:amd64 libxdmcp6:amd64 libxcb1:amd64 libx11-data:all libx11-6:amd64 libx11-xcb1:amd64 libxext6:amd64 libxi6:amd64 libxtst6:amd64 libpulse0:amd64 libpython2.7:amd64 libc-dev-bin:amd64 linux-libc-dev:amd64 libc6-dev:amd64 libexpat1-dev:amd64 libpython2.7-dev:amd64 libquadmath0:amd64 libsctp1:amd64 libtsan0:amd64 libubsan0:amd64 tzdata-java:all java-common:all libjpeg62-turbo:amd64 ca-certificates-java:all openjdk-7-jre-headless:amd64 libmpc3:amd64 libpsl0:amd64 wget:amd64 bzip2:amd64 libperl4-corelibs-perl:all lsof:amd64 openssh-client:amd64 patch:amd64 xz-utils:amd64 binutils:amd64 cpp-4.9:amd64 cpp:amd64 libgcc-4.9-dev:amd64 gcc-4.9:amd64 gcc:amd64 libstdc++-4.9-dev:amd64 g++-4.9:amd64 g++:amd64 make:amd64 libtimedate-perl:all libdpkg-perl:all dpkg-dev:all build-essential:amd64 curl:amd64 libpython-dev:amd64 libqdbm14:amd64 psmisc:amd64 php5-common:amd64 php5-json:amd64 php5-cli:amd64 php5-cgi:amd64 php5-mysql:amd64 python-ply:all python-pycparser:all python-cffi:amd64 python-pkg-resources:all python-six:all python-cryptography:amd64 python2.7-dev:amd64 python-dev:amd64 python-openssl:all unzip:amd64
root                6711                4830                0                   14:46               pts/0               00:00:00            /bin/bash /var/lib/dpkg/info/ca-certificates-java.postinst configure
root                6725                6711                97                  14:46               pts/0               00:12:25            [java] <defunct>
Member

thaJeztah commented Dec 1, 2015

I'm able to reproduce this on docker-machine, OS X 10.10.5, so this may be something related to boot2docker. docker top also gives me <defunct> for a java process;

docker top dreamy_sammet                                                                  Tue Dec  1 15:58:47 2015
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                2538                1023                0                   14:44               ?                   00:00:00            /bin/sh -c apt-get update && apt-get install -y -qq --no-install-recommends build-essential wget curl unzip python python-dev php5-mysql php5-cli php5-cgi openjdk-7-jre-headless openssh-client python-openssl && apt-get clean
root                2566                2538                1                   14:44               ?                   00:00:16            apt-get install -y -qq --no-install-recommends build-essential wget curl unzip python python-dev php5-mysql php5-cli php5-cgi openjdk-7-jre-headless openssh-client python-openssl
root                4830                2566                0                   14:46               pts/0               00:00:00            /usr/bin/dpkg --status-fd 14 --configure libgdbm3:amd64 libjson-c2:amd64 libbsd0:amd64 libedit2:amd64 libkeyutils1:amd64 libkrb5support0:amd64 libk5crypto3:amd64 libkrb5-3:amd64 libgssapi-krb5-2:amd64 libidn11:amd64 libsasl2-modules-db:amd64 libsasl2-2:amd64 libldap-2.4-2:amd64 libmagic1:amd64 libsqlite3-0:amd64 libwrap0:amd64 libxml2:amd64 perl-modules:all perl:amd64 mime-support:all libexpat1:amd64 libpython2.7-stdlib:amd64 python2.7:amd64 libpython-stdlib:amd64 python:amd64 libasan1:amd64 libasyncns0:amd64 libatomic1:amd64 libavahi-common-data:amd64 libavahi-common3:amd64 libdbus-1-3:amd64 libavahi-client3:amd64 libcilkrts5:amd64 libisl10:amd64 libcloog-isl4:amd64 libcups2:amd64 librtmp1:amd64 libssh2-1:amd64 libcurl3:amd64 libogg0:amd64 libflac8:amd64 libpng12-0:amd64 libfreetype6:amd64 ucf:all fonts-dejavu-core:all fontconfig-config:all libfontconfig1:amd64 libglib2.0-0:amd64 libgomp1:amd64 x11-common:all libice6:amd64 libicu52:amd64 libitm1:amd64 liblcms2-2:amd64 liblsan0:amd64 libmpfr4:amd64 mysql-common:all libmysqlclient18:amd64 libnspr4:amd64 libnss3:amd64 libonig2:amd64 libpcsclite1:amd64 libsm6:amd64 libvorbis0a:amd64 libvorbisenc2:amd64 libsndfile1:amd64 libxau6:amd64 libxdmcp6:amd64 libxcb1:amd64 libx11-data:all libx11-6:amd64 libx11-xcb1:amd64 libxext6:amd64 libxi6:amd64 libxtst6:amd64 libpulse0:amd64 libpython2.7:amd64 libc-dev-bin:amd64 linux-libc-dev:amd64 libc6-dev:amd64 libexpat1-dev:amd64 libpython2.7-dev:amd64 libquadmath0:amd64 libsctp1:amd64 libtsan0:amd64 libubsan0:amd64 tzdata-java:all java-common:all libjpeg62-turbo:amd64 ca-certificates-java:all openjdk-7-jre-headless:amd64 libmpc3:amd64 libpsl0:amd64 wget:amd64 bzip2:amd64 libperl4-corelibs-perl:all lsof:amd64 openssh-client:amd64 patch:amd64 xz-utils:amd64 binutils:amd64 cpp-4.9:amd64 cpp:amd64 libgcc-4.9-dev:amd64 gcc-4.9:amd64 gcc:amd64 libstdc++-4.9-dev:amd64 g++-4.9:amd64 g++:amd64 make:amd64 libtimedate-perl:all libdpkg-perl:all dpkg-dev:all build-essential:amd64 curl:amd64 libpython-dev:amd64 libqdbm14:amd64 psmisc:amd64 php5-common:amd64 php5-json:amd64 php5-cli:amd64 php5-cgi:amd64 php5-mysql:amd64 python-ply:all python-pycparser:all python-cffi:amd64 python-pkg-resources:all python-six:all python-cryptography:amd64 python2.7-dev:amd64 python-dev:amd64 python-openssl:all unzip:amd64
root                6711                4830                0                   14:46               pts/0               00:00:00            /bin/bash /var/lib/dpkg/info/ca-certificates-java.postinst configure
root                6725                6711                97                  14:46               pts/0               00:12:25            [java] <defunct>
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 1, 2015

Member

/cc @tianon @nathanleclaire @jeffdm perhaps any of you has an idea where to look, or what to debug, I couldn't really find something

Member

thaJeztah commented Dec 1, 2015

/cc @tianon @nathanleclaire @jeffdm perhaps any of you has an idea where to look, or what to debug, I couldn't really find something

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Dec 1, 2015

Member
Member

tianon commented Dec 1, 2015

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 1, 2015

Member

Looks like memory is not the problem, however the <defunct> process does consume 100% CPU;

CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O               BLOCK I/O
d263da116bfd        99.51%              689.3 MB / 2.1 GB   32.82%              157.9 MB / 2.754 MB   25.15 MB / 130.4 MB

The container seems to be stuck as well, and I had to reboot the vm to get it killed

Member

thaJeztah commented Dec 1, 2015

Looks like memory is not the problem, however the <defunct> process does consume 100% CPU;

CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O               BLOCK I/O
d263da116bfd        99.51%              689.3 MB / 2.1 GB   32.82%              157.9 MB / 2.754 MB   25.15 MB / 130.4 MB

The container seems to be stuck as well, and I had to reboot the vm to get it killed

@jiangty-addepar

This comment has been minimized.

Show comment
Hide comment
@jiangty-addepar

jiangty-addepar Feb 23, 2016

amazing! I've been having this problem for a few weeks now, and I guess the fix for Ubuntu was just released yesterday :P

jiangty-addepar commented Feb 23, 2016

amazing! I've been having this problem for a few weeks now, and I guess the fix for Ubuntu was just released yesterday :P

@pdsouza

This comment has been minimized.

Show comment
Hide comment
@pdsouza

pdsouza Feb 24, 2016

Confirming that the latest 14.04LTS kernel update to 3.19.0-51 puts an end to my java zombies. Thanks!

pdsouza commented Feb 24, 2016

Confirming that the latest 14.04LTS kernel update to 3.19.0-51 puts an end to my java zombies. Thanks!

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Mar 8, 2016

Member

Debian supported this issue.

LATEST QUICK WORKAROUNDS

Distro Workaround
General Use devicemapper/overlay/btrfs (but it may cause another problem..).
If you can upgrade AUFS and build the kernel manually, you can also use AUFS v20160111 or later.
Boot2Docker Upgrade to v1.10.0 or later
Ubuntu 14.04LTS Upgrade kernel to 3.13.0-79.123 or later
Ubuntu 15.04 Upgrade kernel to 3.19.0-51.57 or later
Ubuntu 15.10 Upgrade kernel to 4.2.0-30.35 or later
Debian 7 Upgrade kernel to 3.2.73-2+deb7u3 (of linux-image-3.2.0-4-amd64 package) or later
Debian 8 Upgrade kernel to 3.16.7-ckt20-1+deb8u4 (of linux-image-3.16.0-4-amd64 package) or later
Debian 9 (does not support AUFS since kernel 3.18-1~exp1)
Gentoo Upgrade to recent ones (⚠️ not tested)
RHEL/CentOS (does not support AUFS)
openSUSE (does not support AUFS)

Distributors Issue Tickets

Distro Status Issue URL
Boot2Docker Closed boot2docker/boot2docker#1113
Ubuntu Closed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1533043
Debian Closed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812207
Member

AkihiroSuda commented Mar 8, 2016

Debian supported this issue.

LATEST QUICK WORKAROUNDS

Distro Workaround
General Use devicemapper/overlay/btrfs (but it may cause another problem..).
If you can upgrade AUFS and build the kernel manually, you can also use AUFS v20160111 or later.
Boot2Docker Upgrade to v1.10.0 or later
Ubuntu 14.04LTS Upgrade kernel to 3.13.0-79.123 or later
Ubuntu 15.04 Upgrade kernel to 3.19.0-51.57 or later
Ubuntu 15.10 Upgrade kernel to 4.2.0-30.35 or later
Debian 7 Upgrade kernel to 3.2.73-2+deb7u3 (of linux-image-3.2.0-4-amd64 package) or later
Debian 8 Upgrade kernel to 3.16.7-ckt20-1+deb8u4 (of linux-image-3.16.0-4-amd64 package) or later
Debian 9 (does not support AUFS since kernel 3.18-1~exp1)
Gentoo Upgrade to recent ones (⚠️ not tested)
RHEL/CentOS (does not support AUFS)
openSUSE (does not support AUFS)

Distributors Issue Tickets

Distro Status Issue URL
Boot2Docker Closed boot2docker/boot2docker#1113
Ubuntu Closed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1533043
Debian Closed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812207
@noisy

This comment has been minimized.

Show comment
Hide comment
@noisy

noisy Mar 8, 2016

upgrading kernel of 14.04LTS worked for me 👍

noisy commented Mar 8, 2016

upgrading kernel of 14.04LTS worked for me 👍

@Eyjafjallajokull

This comment has been minimized.

Show comment
Hide comment
@Eyjafjallajokull

Eyjafjallajokull Mar 9, 2016

I'm on OSX on Boot2Docker version 1.10.2, build master : 611be10, Docker version 1.10.2, build c3959b1 and first got this from docker-compose:

Recreating docker_preview_1
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Then tried docker kill 38e1e2590dfa but process hangs forever. docker.log:

time="2016-03-09T14:49:13.053004077Z" level=debug msg="Calling POST /v1.21/containers/38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b/stop"
time="2016-03-09T14:49:13.053058084Z" level=debug msg="POST /v1.21/containers/38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b/stop?t=10"
time="2016-03-09T14:49:13.053097711Z" level=debug msg="Sending 15 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:49:23.053530062Z" level=info msg="Container 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b failed to exit within 10 seconds of SIGTERM - using the force"
time="2016-03-09T14:49:23.053720529Z" level=debug msg="Sending 9 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:49:33.054082100Z" level=info msg="Container 38e1e2590dfa failed to exit within 10 seconds of kill - trying direct SIGKILL"
time="2016-03-09T14:49:34.254353402Z" level=debug msg="Calling GET /v1.22/containers/json"
time="2016-03-09T14:49:34.254413283Z" level=debug msg="GET /v1.22/containers/json"
time="2016-03-09T14:49:54.293708866Z" level=debug msg="Calling POST /v1.22/containers/38e1e2590dfa/kill"
time="2016-03-09T14:49:54.293752784Z" level=debug msg="POST /v1.22/containers/38e1e2590dfa/kill?signal=KILL"
time="2016-03-09T14:49:54.293802705Z" level=debug msg="Sending 9 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:50:04.294276946Z" level=info msg="Container 38e1e2590dfa failed to exit within 10 seconds of kill - trying direct SIGKILL"
time="2016-03-09T14:50:26.678957119Z" level=debug msg="clean 3 unused exec commands"

Eyjafjallajokull commented Mar 9, 2016

I'm on OSX on Boot2Docker version 1.10.2, build master : 611be10, Docker version 1.10.2, build c3959b1 and first got this from docker-compose:

Recreating docker_preview_1
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Then tried docker kill 38e1e2590dfa but process hangs forever. docker.log:

time="2016-03-09T14:49:13.053004077Z" level=debug msg="Calling POST /v1.21/containers/38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b/stop"
time="2016-03-09T14:49:13.053058084Z" level=debug msg="POST /v1.21/containers/38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b/stop?t=10"
time="2016-03-09T14:49:13.053097711Z" level=debug msg="Sending 15 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:49:23.053530062Z" level=info msg="Container 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b failed to exit within 10 seconds of SIGTERM - using the force"
time="2016-03-09T14:49:23.053720529Z" level=debug msg="Sending 9 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:49:33.054082100Z" level=info msg="Container 38e1e2590dfa failed to exit within 10 seconds of kill - trying direct SIGKILL"
time="2016-03-09T14:49:34.254353402Z" level=debug msg="Calling GET /v1.22/containers/json"
time="2016-03-09T14:49:34.254413283Z" level=debug msg="GET /v1.22/containers/json"
time="2016-03-09T14:49:54.293708866Z" level=debug msg="Calling POST /v1.22/containers/38e1e2590dfa/kill"
time="2016-03-09T14:49:54.293752784Z" level=debug msg="POST /v1.22/containers/38e1e2590dfa/kill?signal=KILL"
time="2016-03-09T14:49:54.293802705Z" level=debug msg="Sending 9 to 38e1e2590dfa5d77482b8fbf6b14f01e8d5278622b8e5d7262cd2cdeb777690b"
time="2016-03-09T14:50:04.294276946Z" level=info msg="Container 38e1e2590dfa failed to exit within 10 seconds of kill - trying direct SIGKILL"
time="2016-03-09T14:50:26.678957119Z" level=debug msg="clean 3 unused exec commands"
@einhverfr

This comment has been minimized.

Show comment
Hide comment
@einhverfr

einhverfr Aug 22, 2016

Just as a note (I know this is closed but not sure if it makes sense to open as a new issue). I was having the same issue on a later version until I switched to devmapper.

$ docker info
Containers: 4
 Running: 3
 Paused: 0
 Stopped: 1
Images: 81
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-8:1-9044034-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.726 GB
 Data Space Total: 107.4 GB
 Data Space Available: 96.43 GB
 Metadata Space Used: 4.387 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.56 GiB
Name: ravn
ID: L2WX:3RQ7:W6IC:7MY3:M3ZC:7MP2:3ZMP:VHW4:TLXM:VLYO:NNZ5:2FVW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

einhverfr commented Aug 22, 2016

Just as a note (I know this is closed but not sure if it makes sense to open as a new issue). I was having the same issue on a later version until I switched to devmapper.

$ docker info
Containers: 4
 Running: 3
 Paused: 0
 Stopped: 1
Images: 81
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-8:1-9044034-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.726 GB
 Data Space Total: 107.4 GB
 Data Space Available: 96.43 GB
 Metadata Space Used: 4.387 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.56 GiB
Name: ravn
ID: L2WX:3RQ7:W6IC:7MY3:M3ZC:7MP2:3ZMP:VHW4:TLXM:VLYO:NNZ5:2FVW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Aug 22, 2016

Member

@einhverfr The issue is fixed in kernel 3.13.0-79.123 (your one seems to be 3.13.0-77)

Member

AkihiroSuda commented Aug 22, 2016

@einhverfr The issue is fixed in kernel 3.13.0-79.123 (your one seems to be 3.13.0-77)

@martinm82

This comment has been minimized.

Show comment
Hide comment
@martinm82

martinm82 Feb 14, 2017

Can this issue really be solved with a Kernel upgrade? We are encountering the same problem with Docker 1.9.1 on Ubuntu 14.04 with Kernel 3.13.0-83-generic.

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

martinm82 commented Feb 14, 2017

Can this issue really be solved with a Kernel upgrade? We are encountering the same problem with Docker 1.9.1 on Ubuntu 14.04 with Kernel 3.13.0-83-generic.

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 14, 2017

Member

@martinm82 yes, this issue was a kernel issue. It's possible something else can result in a similar behavior, or if there's a regression in the kernel. However, please open a new issue if you're having issues on the current release; keep in mind that docker 1.9.1 is EOL, so won't be receiving updates anymore.

Member

thaJeztah commented Feb 14, 2017

@martinm82 yes, this issue was a kernel issue. It's possible something else can result in a similar behavior, or if there's a regression in the kernel. However, please open a new issue if you're having issues on the current release; keep in mind that docker 1.9.1 is EOL, so won't be receiving updates anymore.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 14, 2017

Member

I am locking the discussion on this issue, because the original issue here was resolved, and I want to prevent this issue from collecting possibly unrelated issues. See this comment; #18180 (comment) for the kernel versions needed to fix this issue

Member

thaJeztah commented Feb 14, 2017

I am locking the discussion on this issue, because the original issue here was resolved, and I want to prevent this issue from collecting possibly unrelated issues. See this comment; #18180 (comment) for the kernel versions needed to fix this issue

@moby moby locked and limited conversation to collaborators Feb 14, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.