starting test-containers hangs

[kamkie]

starting test-containers version 1.9.x sometimes hangs on server with multiple repositoris defined in config.json

log from failed attempt

10:58:01.331 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/var/lib/jenkins/jobs/wejit-backend-mr/workspace/target/test-classes/testcontainers.properties
10:58:01.340 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/var/lib/jenkins/jobs/wejit-backend-mr/workspace/target/test-classes/testcontainers.properties
10:58:01.340 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
10:58:01.341 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides loaded from TestcontainersConfiguration(properties={docker.client.strategy=org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy, ryuk.container.privileged=true})
10:58:01.404 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy from ~/.testcontainers.properties, will try it first
10:58:01.531 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Will use 'okhttp' transport
10:58:02.020 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
10:58:02.547 [ducttape-0] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: org.testcontainers.dockerclient.transport.okhttp.OkHttpDockerCmdExecFactory$1@ee75499
10:58:02.706 [main] INFO  org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
10:58:02.726 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

10:58:02.734 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=alpine:3.5,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@797f97e3]
10:58:02.941 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: alpine:3.5
10:58:02.942 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (exists) and commandPathPrefix: 
10:58:02.950 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [alpine:3.5]
10:58:02.951 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - no matching Auth Configs - falling back to defaultAuthConfig [null]
10:58:02.951 [main] DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [null]
10:58:02.977 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@5d94ac8a[name=<null>,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd={sh,-c,ip route|awk '/default/ { print $3 }'},entrypoint=<null>,image=alpine:3.5,volumes=com.github.dockerjava.api.model.Volumes@2bc8caa7,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@582ea164,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@2fccf49e[binds=<null>,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=<null>,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=<null>,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true, org.testcontainers.sessionId=95d81324-95d9-434e-80eb-ea96ae4137ad},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@31de27c]
10:58:03.530 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 5a621d03dac6ed4acc5fcb40949eba41a3066e4d7082521121307f86d207c45a,com.github.dockerjava.core.exec.StartContainerCmdExec@6d45dd4
10:58:04.025 [tc-okhttp-stream-968448640] DEBUG com.github.dockerjava.core.command.LogContainerResultCallback - STDOUT: 172.17.0.1

logs from successful attempt

10:56:13.321 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/var/lib/jenkins/jobs/wejit-backend-mr/workspace/target/test-classes/testcontainers.properties
10:56:13.330 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/var/lib/jenkins/jobs/wejit-backend-mr/workspace/target/test-classes/testcontainers.properties
10:56:13.330 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
10:56:13.330 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides loaded from TestcontainersConfiguration(properties={docker.client.strategy=org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy, ryuk.container.privileged=true})
10:56:13.384 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy from ~/.testcontainers.properties, will try it first
10:56:13.499 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Will use 'okhttp' transport
10:56:13.873 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
10:56:14.407 [ducttape-0] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: org.testcontainers.dockerclient.transport.okhttp.OkHttpDockerCmdExecFactory$1@28ed8a8a
10:56:14.579 [main] INFO  org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
10:56:14.602 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

10:56:14.610 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=alpine:3.5,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@797f97e3]
10:56:14.855 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: alpine:3.5
10:56:14.856 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (exists) and commandPathPrefix: 
10:56:14.871 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [alpine:3.5]
10:56:14.872 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - no matching Auth Configs - falling back to defaultAuthConfig [null]
10:56:14.873 [main] DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [null]
10:56:14.906 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@5d94ac8a[name=<null>,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd={sh,-c,ip route|awk '/default/ { print $3 }'},entrypoint=<null>,image=alpine:3.5,volumes=com.github.dockerjava.api.model.Volumes@2bc8caa7,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@582ea164,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@2fccf49e[binds=<null>,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=<null>,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=<null>,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true, org.testcontainers.sessionId=718c7e37-bc40-48d3-adca-3976aa7459c8},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@31de27c]
10:56:15.519 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 57f7356d26280a7b0ae72c3297a05359b596751d7054991b3373f2f6facfd449,com.github.dockerjava.core.exec.StartContainerCmdExec@6d45dd4
10:56:15.959 [tc-okhttp-stream-968448640] DEBUG com.github.dockerjava.core.command.LogContainerResultCallback - STDOUT: 172.17.0.1
10:56:15.967 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 57f7356d26280a7b0ae72c3297a05359b596751d7054991b3373f2f6facfd449,true,true,com.github.dockerjava.core.exec.RemoveContainerCmdExec@31a4db4f
10:56:16.222 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
10:56:16.224 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.exec.InfoCmdExec@691eb389
10:56:16.386 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.exec.VersionCmdExec@1adfb5b8
10:56:16.392 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 1.12.6
  API Version: 1.24
  Operating System: CentOS Linux 7 (Core)
  Total Memory: 32012 MB
10:56:16.396 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=quay.io/testcontainers/ryuk:0.2.2,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@364c93e6]
10:56:16.454 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
10:56:16.454 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (exists) and commandPathPrefix: 
10:56:16.455 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.2]
10:56:16.455 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - no matching Auth Configs - falling back to defaultAuthConfig [null]
10:56:16.455 [main] DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [null]

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[kiview]

Is this still something you are observing @kamkie?
The logs don't much information for this issue.

[kamkie]

yes, i still have this problem but only on openshift. i tried to investigate it further but find nothing

[kiview]

What the detailed behavior? The Testcontainers code hangs, or the Docker container is not started?

[kamkie]

container getting host ip is started and when i run docker logs on it it has proper ip there
but test container code does not continuing to start my postgres container

[kiview]

So basically your complete test suite hangs indefinitely?

[kamkie]

yes, I'm using test containers 1.5.3 all projects that are build and hosted on openshift

[kiview]

In your initial post you said:

starting test-containers version 1.9.x sometimes hangs on server with multiple repositoris defined in config.json

So which version are you using?

[kamkie]

I tried every version from 1.9.0 before release to 1.10.2

[kiview]

So you mean, it works fine for you with 1.5.3 and stopped working with every version >= 1.9.0?
And your JVM hangs?

[kamkie]

yes

[kiview]

Can you please provide an anonymized version of your config.json file, so we can try to reproduce the bug?