New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't pull from docker.io explicitly, 500 error or #845

Closed
ernestm opened this Issue Aug 21, 2017 · 10 comments

Comments

Projects
None yet
4 participants
@ernestm

ernestm commented Aug 21, 2017

Description

I have a maven configuration that was working until this weekend, where we're setting a custom registry to pull from/push to by default in the parent, and then for a couple specific images that inherit from public images, we set a
<docker.pull.registry>docker.io</docker.pull.registry> in their pom. Now all these builds fail.

For example, I have a Dockerfile that's FROM debian:sid. As of the 20th, I get 500 errors when the plugin tries to pull it, both from my Mac and from a Ubuntu bamboo build server in AWS (completely different networks, etc.). The build server tells me "Unable to pull 'debian:sid' from registry 'docker.io' : Could not reach any registry endpoint" and the Mac tells me "Unable to pull 'library/debian:sid' from registry 'docker.io' : received unexpected HTTP status: 500 Server Error (Internal Server Error: 500)".

Mac mvn -X output:

[DEBUG] http-outgoing-0 >> POST /v1.21/images/create?fromImage=docker.io%2Flibrary%2Fdebian&tag=sid HTTP/1.1
[DEBUG] http-outgoing-0 >> Accept: */*
[DEBUG] http-outgoing-0 >> Content-Type: application/json; charset=utf-8
[DEBUG] http-outgoing-0 >> X-Registry-Auth: eyJwYXNzd29yZCI6IiIsImF1dGgiOiIiLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoiIn0=
[DEBUG] http-outgoing-0 >> Content-Length: 0
[DEBUG] http-outgoing-0 >> Host: 127.0.0.1:1
[DEBUG] http-outgoing-0 >> Connection: Keep-Alive
[DEBUG] http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_112)
[DEBUG] http-outgoing-0 >> Accept-Encoding: gzip,deflate
[DEBUG] http-outgoing-0 >> "POST /v1.21/images/create?fromImage=docker.io%2Flibrary%2Fdebian&tag=sid HTTP/1.1[\r][\n]"
[DEBUG] http-outgoing-0 >> "Accept: */*[\r][\n]"
[DEBUG] http-outgoing-0 >> "Content-Type: application/json; charset=utf-8[\r][\n]"
[DEBUG] http-outgoing-0 >> "X-Registry-Auth: eyJwYXNzd29yZCI6IiIsImF1dGgiOiIiLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoiIn0=[\r][\n]"
[DEBUG] http-outgoing-0 >> "Content-Length: 0[\r][\n]"
[DEBUG] http-outgoing-0 >> "Host: 127.0.0.1:1[\r][\n]"
[DEBUG] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
[DEBUG] http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_112)[\r][\n]"
[DEBUG] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[DEBUG] http-outgoing-0 >> "[\r][\n]"
[DEBUG] http-outgoing-0 << "HTTP/1.1 500 Internal Server Error[\r][\n]"
[DEBUG] http-outgoing-0 << "Api-Version: 1.31[\r][\n]"
[DEBUG] http-outgoing-0 << "Content-Length: 50[\r][\n]"
[DEBUG] http-outgoing-0 << "Content-Type: text/plain; charset=utf-8[\r][\n]"
[DEBUG] http-outgoing-0 << "Date: Mon, 21 Aug 2017 19:42:00 GMT[\r][\n]"
[DEBUG] http-outgoing-0 << "Docker-Experimental: true[\r][\n]"
[DEBUG] http-outgoing-0 << "Ostype: linux[\r][\n]"
[DEBUG] http-outgoing-0 << "Server: Docker/17.07.0-ce-rc3 (linux)[\r][\n]"
[DEBUG] http-outgoing-0 << "X-Content-Type-Options: nosniff[\r][\n]"
[DEBUG] http-outgoing-0 << "[\r][\n]"
[DEBUG] http-outgoing-0 << HTTP/1.1 500 Internal Server Error
[DEBUG] http-outgoing-0 << Api-Version: 1.31
[DEBUG] http-outgoing-0 << Content-Length: 50
[DEBUG] http-outgoing-0 << Content-Type: text/plain; charset=utf-8
[DEBUG] http-outgoing-0 << Date: Mon, 21 Aug 2017 19:42:00 GMT
[DEBUG] http-outgoing-0 << Docker-Experimental: true
[DEBUG] http-outgoing-0 << Ostype: linux
[DEBUG] http-outgoing-0 << Server: Docker/17.07.0-ce-rc3 (linux)
[DEBUG] http-outgoing-0 << X-Content-Type-Options: nosniff
[DEBUG] Connection can be kept alive indefinitely
[DEBUG] http-outgoing-0 << "received unexpected HTTP status: 500 Server Error[\n]"
[DEBUG] Connection [id: 0][route: {}->unix://127.0.0.1:1] can be kept alive indefinitely
[DEBUG] Connection released: [id: 0][route: {}->unix://127.0.0.1:1][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 20]
[ERROR] DOCKER> Unable to pull 'library/debian:sid' from registry 'docker.io' : received unexpected HTTP status: 500 Server Error (Internal Server Error: 500) [received unexpected HTTP status: 500 Server Error (Internal Server Error: 500)]
[DEBUG] Connection manager is shutting down
[DEBUG] http-outgoing-0: Close connection
[DEBUG] Connection manager shut down
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.169 s
[INFO] Finished at: 2017-08-21T14:42:00-05:00
[INFO] Final Memory: 23M/358M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.20.1:build (default-build) on project otxb-core-base-container: Unable to pull 'library/debian:sid' from registry 'docker.io' : received unexpected HTTP status: 500 Server Error (Internal Server Error: 500) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.fabric8:docker-maven-plugin:0.20.1:build (default-build) on project otxb-core-base-container: Unable to pull 'library/debian:sid' from registry 'docker.io' : received unexpected HTTP status: 500 Server Error (Internal Server Error: 500)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Unable to pull 'library/debian:sid' from registry 'docker.io' : received unexpected HTTP status: 500 Server Error (Internal Server Error: 500)
	at io.fabric8.maven.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:226)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
	... 20 more
Caused by: io.fabric8.maven.docker.access.DockerAccessException: Unable to pull 'library/debian:sid' from registry 'docker.io' : received unexpected HTTP status: 500 Server Error (Internal Server Error: 500)
	at io.fabric8.maven.docker.access.hc.DockerAccessWithHcClient.pullImage(DockerAccessWithHcClient.java:358)
	at io.fabric8.maven.docker.service.RegistryService.checkImageWithAutoPull(RegistryService.java:81)
	at io.fabric8.maven.docker.service.BuildService.autoPullBaseImage(BuildService.java:211)
	at io.fabric8.maven.docker.service.BuildService.buildImage(BuildService.java:58)
	at io.fabric8.maven.docker.BuildMojo.buildAndTag(BuildMojo.java:58)
	at io.fabric8.maven.docker.BuildMojo.executeInternal(BuildMojo.java:44)
	at io.fabric8.maven.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:223)
	... 22 more
Caused by: io.fabric8.maven.docker.access.hc.http.HttpRequestException: received unexpected HTTP status: 500 Server Error (Internal Server Error: 500)
	at io.fabric8.maven.docker.access.hc.ApacheHttpClientDelegate$StatusCodeCheckerResponseHandler.handleResponse(ApacheHttpClientDelegate.java:177)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:222)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
	at io.fabric8.maven.docker.access.hc.ApacheHttpClientDelegate.post(ApacheHttpClientDelegate.java:97)
	at io.fabric8.maven.docker.access.hc.DockerAccessWithHcClient.pullImage(DockerAccessWithHcClient.java:355)
	... 28 more

Ubuntu bamboo mvn -X output:

build	21-Aug-2017 16:14:35	[DEBUG] Connection leased: [id: 2][route: {}->unix://127.0.0.1:1][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 20]
build	21-Aug-2017 16:14:35	[DEBUG] Executing request POST /v1.21/images/create?fromImage=docker.io%2Fdebian&tag=sid HTTP/1.1
build	21-Aug-2017 16:14:35	[DEBUG] Target auth state: UNCHALLENGED
build	21-Aug-2017 16:14:35	[DEBUG] Proxy auth state: UNCHALLENGED
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> POST /v1.21/images/create?fromImage=docker.io%2Fdebian&tag=sid HTTP/1.1
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> Accept: */*
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> Content-Type: application/json; charset=utf-8
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> X-Registry-Auth: eyJwYXNzd29yZCI6IiIsImF1dGgiOiIiLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoiIn0=
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> Content-Length: 0
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> Host: 127.0.0.1:1
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> Connection: Keep-Alive
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_112)
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> Accept-Encoding: gzip,deflate
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "POST /v1.21/images/create?fromImage=docker.io%2Fdebian&tag=sid HTTP/1.1[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "Accept: */*[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "Content-Type: application/json; charset=utf-8[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "X-Registry-Auth: eyJwYXNzd29yZCI6IiIsImF1dGgiOiIiLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoiIn0=[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "Content-Length: 0[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "Host: 127.0.0.1:1[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "Connection: Keep-Alive[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_112)[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "Accept-Encoding: gzip,deflate[\r][\n]"
build	21-Aug-2017 16:14:35	[DEBUG] http-outgoing-2 >> "[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "HTTP/1.1 200 OK[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "Content-Type: application/json[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "Server: Docker/1.12.6 (linux)[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "Date: Mon, 21 Aug 2017 16:14:40 GMT[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "Transfer-Encoding: chunked[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "3a[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "{"status":"Pulling repository docker.io/library/debian"}[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << "[\r][\n]"
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << HTTP/1.1 200 OK
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << Content-Type: application/json
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << Server: Docker/1.12.6 (linux)
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << Date: Mon, 21 Aug 2017 16:14:40 GMT
build	21-Aug-2017 16:14:40	[DEBUG] http-outgoing-2 << Transfer-Encoding: chunked
build	21-Aug-2017 16:14:40	[DEBUG] Connection can be kept alive indefinitely
build	21-Aug-2017 16:14:40	[INFO] DOCKER> Pulling repository docker.io/library/debian
build	21-Aug-2017 16:14:42	[DEBUG] http-outgoing-2 << "75[\r][\n]"
build	21-Aug-2017 16:14:42	[DEBUG] http-outgoing-2 << "{"errorDetail":{"message":"Could not reach any registry endpoint"},"error":"Could not reach any registry endpoint"}[\r][\n]"
build	21-Aug-2017 16:14:42	[DEBUG] http-outgoing-2 << "[\r][\n]"
build	21-Aug-2017 16:14:42	[DEBUG] http-outgoing-2 << "0[\r][\n]"
build	21-Aug-2017 16:14:42	[DEBUG] http-outgoing-2 << "[\r][\n]"
build	21-Aug-2017 16:14:42	[DEBUG] Connection [id: 2][route: {}->unix://127.0.0.1:1] can be kept alive indefinitely
build	21-Aug-2017 16:14:42	[DEBUG] Connection released: [id: 2][route: {}->unix://127.0.0.1:1][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 20]
build	21-Aug-2017 16:14:42	[ERROR] DOCKER> Unable to pull 'debian:sid' from registry 'docker.io' : Could not reach any registry endpoint  [Could not reach any registry endpoint ]
build	21-Aug-2017 16:14:42	[DEBUG] Connection manager is shutting down
build	21-Aug-2017 16:14:42	[DEBUG] http-outgoing-2: Close connection
build	21-Aug-2017 16:14:42	[DEBUG] Connection manager shut down
build	21-Aug-2017 16:14:42	[INFO] ------------------------------------------------------------------------
build	21-Aug-2017 16:14:42	[INFO] BUILD FAILURE
build	21-Aug-2017 16:14:42	[INFO] ------------------------------------------------------------------------
build	21-Aug-2017 16:14:42	[INFO] Total time: 9.548 s
build	21-Aug-2017 16:14:42	[INFO] Finished at: 2017-08-21T16:14:42+00:00
build	21-Aug-2017 16:14:42	[INFO] Final Memory: 26M/492M
build	21-Aug-2017 16:14:42	[INFO] ------------------------------------------------------------------------
build	21-Aug-2017 16:14:42	[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.20.1:build (default-build) on project otxb-core-base-container: Unable to pull 'debian:sid' from registry 'docker.io' : Could not reach any registry endpoint -> [Help 1]
build	21-Aug-2017 16:14:42	org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.fabric8:docker-maven-plugin:0.20.1:build (default-build) on project otxb-core-base-container: Unable to pull 'debian:sid' from registry 'docker.io' : Could not reach any registry endpoint 
build	21-Aug-2017 16:14:42		at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
build	21-Aug-2017 16:14:42		at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
build	21-Aug-2017 16:14:42		at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
build	21-Aug-2017 16:14:42		at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
build	21-Aug-2017 16:14:42		at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
build	21-Aug-2017 16:14:42		at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
build	21-Aug-2017 16:14:42		at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
build	21-Aug-2017 16:14:42		at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
build	21-Aug-2017 16:14:42		at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
build	21-Aug-2017 16:14:42		at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
build	21-Aug-2017 16:14:42		at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
build	21-Aug-2017 16:14:42		at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
build	21-Aug-2017 16:14:42		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
build	21-Aug-2017 16:14:42		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
build	21-Aug-2017 16:14:42		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
build	21-Aug-2017 16:14:42		at java.lang.reflect.Method.invoke(Method.java:498)
build	21-Aug-2017 16:14:42		at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
build	21-Aug-2017 16:14:42		at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
build	21-Aug-2017 16:14:42		at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
build	21-Aug-2017 16:14:42		at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
build	21-Aug-2017 16:14:42	Caused by: org.apache.maven.plugin.MojoExecutionException: Unable to pull 'debian:sid' from registry 'docker.io' : Could not reach any registry endpoint 
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:226)
build	21-Aug-2017 16:14:42		at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
build	21-Aug-2017 16:14:42		at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
build	21-Aug-2017 16:14:42		... 19 more
build	21-Aug-2017 16:14:42	Caused by: io.fabric8.maven.docker.access.DockerAccessException: Unable to pull 'debian:sid' from registry 'docker.io' : Could not reach any registry endpoint 
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.access.hc.DockerAccessWithHcClient.pullImage(DockerAccessWithHcClient.java:358)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.service.RegistryService.checkImageWithAutoPull(RegistryService.java:81)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.service.BuildService.autoPullBaseImage(BuildService.java:211)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.service.BuildService.buildImage(BuildService.java:58)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.BuildMojo.buildAndTag(BuildMojo.java:58)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.BuildMojo.executeInternal(BuildMojo.java:44)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:223)
build	21-Aug-2017 16:14:42		... 21 more
build	21-Aug-2017 16:14:42	Caused by: io.fabric8.maven.docker.access.DockerAccessException: Could not reach any registry endpoint 
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.access.chunked.PullOrPushResponseJsonHandler.throwDockerAccessException(PullOrPushResponseJsonHandler.java:45)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.access.chunked.PullOrPushResponseJsonHandler.process(PullOrPushResponseJsonHandler.java:22)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.access.chunked.EntityStreamReaderUtil.processJsonStream(EntityStreamReaderUtil.java:26)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.access.hc.DockerAccessWithHcClient$HcChunkedResponseHandlerWrapper.handleResponse(DockerAccessWithHcClient.java:652)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.access.hc.ApacheHttpClientDelegate$StatusCodeCheckerResponseHandler.handleResponse(ApacheHttpClientDelegate.java:172)
build	21-Aug-2017 16:14:42		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:222)
build	21-Aug-2017 16:14:42		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
build	21-Aug-2017 16:14:42		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.access.hc.ApacheHttpClientDelegate.post(ApacheHttpClientDelegate.java:97)
build	21-Aug-2017 16:14:42		at io.fabric8.maven.docker.access.hc.DockerAccessWithHcClient.pullImage(DockerAccessWithHcClient.java:355)
build	21-Aug-2017 16:14:42		... 27 more

I can pull manually just fine on both the boxes:

ubuntu@ip-10-42-128-244:~$ docker pull docker.io/debian:sid
sid: Pulling from library/debian
b5499bef70c5: Pull complete 
Digest: sha256:c5e3138644b0b6116f95c05b211ac2a456f575c200b4edac5878d78144aebef4
Status: Downloaded newer image for debian:sid

As you can see it definitely forgets our custom registry and understands it's trying to get from docker.io - it just fails. I have tried registry-1.docker.com, registry.docker.com, registry.hub.docker.com, and various other permutations of .io/.com and hostnames to no good effect.

This was working until the 19th; we didn't change code, poms, or versions. Then it started failing on the 20th. Any thoughts are welcome.

  • Happening on my mac from home and from bamboo server in AWS - not network, maven version, docker version
  • POMs not changed
  • Builds not changed
  • Code not changed in any meaningful way
  • Different parent poms
  • Different base images (debian, ubuntu)
  • Multiple builds
  • Different default repos (otxb is in AWS ECR, otxp is in jfrog Artifactory) that are getting overridden for the pull

Info

  • d-m-p version : 0.20.1
  • Maven version (mvn -v) for my Mac:
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T14:39:06-05:00)
Maven home: /usr/local/Cellar/maven/3.5.0/libexec
Java version: 1.8.0_112, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac"

for the ubuntu bamboo server, it's Maven 3.3.

  • Docker version on Mac:
Client:
 Version:      17.07.0-ce-rc3
 API version:  1.31
 Go version:   go1.8.3
 Git commit:   665d244
 Built:        Thu Aug 17 00:56:49 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.07.0-ce-rc3
 API version:  1.31 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   665d244
 Built:        Thu Aug 17 01:32:35 2017
 OS/Arch:      linux/amd64
 Experimental: true
  • Docker version on build server:
ubuntu@ip-10-42-128-244:~$ docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:38:45 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:38:45 2017
 OS/Arch:      linux/amd64
  • If it's a bug, how to reproduce : Just mvn install with our POMs/any pom setting a docker.pull.registry to docker.io.
@kogitant

This comment has been minimized.

Show comment
Hide comment
@kogitant

kogitant Aug 22, 2017

I seem to have the same or very similar problem. My docker daemon debug logs show this when a Jenkins build attempts "mvn docker-maven-plugin:0.21.0:build"

Aug 22 06:41:47 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:47.826115678Z" level=debug msg="Calling GET /version"
Aug 22 06:41:47 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:47.851874009Z" level=debug msg="Calling GET /v1.30/images/frolvlad%2Falpine-oraclejdk8%3Aslim/json"
Aug 22 06:41:47 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:47.860980746Z" level=debug msg="Calling POST /v1.30/images/create?fromImage=docker.io%2Ffrolvlad%2Falpine-oraclejdk8&tag=slim"
Aug 22 06:41:47 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:47.861127432Z" level=debug msg="Trying to pull frolvlad/alpine-oraclejdk8 from https://registry-1.docker.io v2"
Aug 22 06:41:52 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:52.505829798Z" level=info msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 500 Server Error"
Aug 22 06:41:52 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:52.505877667Z" level=error msg="Handler for POST /v1.30/images/create returned error: received unexpected HTTP status: 500 Server Error"

If ssh to the jenkins slave and run the same exact mvn command in the workspace of the jenkins build, it succeeds and docker daemon log shows this:

Aug 22 06:17:51 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:51.690993816Z" level=debug msg="Calling GET /version"
Aug 22 06:17:51 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:51.715278332Z" level=debug msg="Calling GET /v1.30/images/frolvlad%2Falpine-oraclejdk8%3Aslim/json"
Aug 22 06:17:51 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:51.722393341Z" level=debug msg="Calling POST /v1.30/images/create?fromImage=frolvlad%2Falpine-oraclejdk8&tag=slim"
Aug 22 06:17:51 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:51.722521644Z" level=debug msg="Trying to pull frolvlad/alpine-oraclejdk8 from https://registry-1.docker.io v2"
Aug 22 06:17:52 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:52.809160005Z" level=debug msg="Pulling ref from V2 registry: frolvlad/alpine-oraclejdk8:slim"
Aug 22 06:17:52 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:52.809409368Z" level=debug msg="pulling blob \"sha256:77f0b91330fc657e9b5a5a4ce1e4fff708b1a52a0a421deb7613ffb5cf5f0c9a\""

The build broke during the weekend, no changes were made to the project, to Jenkins setup etc.

kogitant commented Aug 22, 2017

I seem to have the same or very similar problem. My docker daemon debug logs show this when a Jenkins build attempts "mvn docker-maven-plugin:0.21.0:build"

Aug 22 06:41:47 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:47.826115678Z" level=debug msg="Calling GET /version"
Aug 22 06:41:47 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:47.851874009Z" level=debug msg="Calling GET /v1.30/images/frolvlad%2Falpine-oraclejdk8%3Aslim/json"
Aug 22 06:41:47 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:47.860980746Z" level=debug msg="Calling POST /v1.30/images/create?fromImage=docker.io%2Ffrolvlad%2Falpine-oraclejdk8&tag=slim"
Aug 22 06:41:47 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:47.861127432Z" level=debug msg="Trying to pull frolvlad/alpine-oraclejdk8 from https://registry-1.docker.io v2"
Aug 22 06:41:52 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:52.505829798Z" level=info msg="Attempting next endpoint for pull after error: received unexpected HTTP status: 500 Server Error"
Aug 22 06:41:52 10-224-125-29 dockerd[23405]: time="2017-08-22T06:41:52.505877667Z" level=error msg="Handler for POST /v1.30/images/create returned error: received unexpected HTTP status: 500 Server Error"

If ssh to the jenkins slave and run the same exact mvn command in the workspace of the jenkins build, it succeeds and docker daemon log shows this:

Aug 22 06:17:51 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:51.690993816Z" level=debug msg="Calling GET /version"
Aug 22 06:17:51 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:51.715278332Z" level=debug msg="Calling GET /v1.30/images/frolvlad%2Falpine-oraclejdk8%3Aslim/json"
Aug 22 06:17:51 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:51.722393341Z" level=debug msg="Calling POST /v1.30/images/create?fromImage=frolvlad%2Falpine-oraclejdk8&tag=slim"
Aug 22 06:17:51 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:51.722521644Z" level=debug msg="Trying to pull frolvlad/alpine-oraclejdk8 from https://registry-1.docker.io v2"
Aug 22 06:17:52 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:52.809160005Z" level=debug msg="Pulling ref from V2 registry: frolvlad/alpine-oraclejdk8:slim"
Aug 22 06:17:52 10-224-125-29 dockerd[23405]: time="2017-08-22T06:17:52.809409368Z" level=debug msg="pulling blob \"sha256:77f0b91330fc657e9b5a5a4ce1e4fff708b1a52a0a421deb7613ffb5cf5f0c9a\""

The build broke during the weekend, no changes were made to the project, to Jenkins setup etc.

@rhuss

This comment has been minimized.

Show comment
Hide comment
@rhuss

rhuss Aug 22, 2017

Collaborator

Thanks for the report, but unfortunately I can't reproduce it here (with Docker for macOS). Do you have a sample pom.xml how I could reproduce the error ?

The logs above ("Unable to pull 'debian:sid' from registry 'docker.io' : Could not reach any registry endpoint") sound like that the Docker daemon cannot reach out to docker.io (so might be a network issue, too). For registry related issues it is important to note that its the Docker daemon which tries to contact the registry, never the client (cli, d-m-p) itself. So fo comparing runs its important to use the same Docker daemons (just mentioning it).

Collaborator

rhuss commented Aug 22, 2017

Thanks for the report, but unfortunately I can't reproduce it here (with Docker for macOS). Do you have a sample pom.xml how I could reproduce the error ?

The logs above ("Unable to pull 'debian:sid' from registry 'docker.io' : Could not reach any registry endpoint") sound like that the Docker daemon cannot reach out to docker.io (so might be a network issue, too). For registry related issues it is important to note that its the Docker daemon which tries to contact the registry, never the client (cli, d-m-p) itself. So fo comparing runs its important to use the same Docker daemons (just mentioning it).

@kogitant

This comment has been minimized.

Show comment
Hide comment
@kogitant

kogitant Aug 22, 2017

My problem was resolved by upgrading the Jenkins pipeline to use JDK8u144 instead of JDK8u112 that it was using. When I ssh'd to the jenkins slave, the slave machine itself had JDK8u144 installed, and thus the build worked when executed directly on the machine.

kogitant commented Aug 22, 2017

My problem was resolved by upgrading the Jenkins pipeline to use JDK8u144 instead of JDK8u112 that it was using. When I ssh'd to the jenkins slave, the slave machine itself had JDK8u144 installed, and thus the build worked when executed directly on the machine.

@sylvainlaurent

This comment has been minimized.

Show comment
Hide comment
@sylvainlaurent

sylvainlaurent Aug 22, 2017

thanks @kogitant this worked for me too.
How do you explain that this problem occurred all of a sudden without any change to neither docker nor maven nor java nor the plugin ??

sylvainlaurent commented Aug 22, 2017

thanks @kogitant this worked for me too.
How do you explain that this problem occurred all of a sudden without any change to neither docker nor maven nor java nor the plugin ??

@kogitant

This comment has been minimized.

Show comment
Hide comment
@kogitant

kogitant Aug 22, 2017

I suspect it's related to SSL certificates somehow. As far as I know a JDK contains details of trusted root certificate authorities. Those can become outdated. But that's just a guess, I don't know for sure what the problem was here.

kogitant commented Aug 22, 2017

I suspect it's related to SSL certificates somehow. As far as I know a JDK contains details of trusted root certificate authorities. Those can become outdated. But that's just a guess, I don't know for sure what the problem was here.

@rhuss

This comment has been minimized.

Show comment
Hide comment
@rhuss

rhuss Aug 22, 2017

Collaborator

@sylvainlaurent @kogitant Is it ok for you to close the issue because it seems not to be directly dmp related ?

Collaborator

rhuss commented Aug 22, 2017

@sylvainlaurent @kogitant Is it ok for you to close the issue because it seems not to be directly dmp related ?

@ernestm

This comment has been minimized.

Show comment
Hide comment
@ernestm

ernestm Aug 22, 2017

Well dip me in honey and call the bears, that fixes it! I was on 1.8.0_112-b16 but upgrading to 1.8.0_144 has made the issue go away. Thanks all! So for future searchers, if you're using docker-maven-plugin, or I guess any other maven plugin or java scheme for downloading container images (?) from docker.io, upgrading your Java version to at least .144 will get you going again.

Thanks to everyone who pitched in to find this.

ernestm commented Aug 22, 2017

Well dip me in honey and call the bears, that fixes it! I was on 1.8.0_112-b16 but upgrading to 1.8.0_144 has made the issue go away. Thanks all! So for future searchers, if you're using docker-maven-plugin, or I guess any other maven plugin or java scheme for downloading container images (?) from docker.io, upgrading your Java version to at least .144 will get you going again.

Thanks to everyone who pitched in to find this.

@ernestm ernestm closed this Aug 22, 2017

@rhuss

This comment has been minimized.

Show comment
Hide comment
@rhuss

rhuss Aug 23, 2017

Collaborator

fwiw, there was a TLS issue with Java 8u121 and 8u131 (and prior). On Alpine Linux only, though (see e.g. docker-library/openjdk#115)

wonder whether your Jenkins images are based on Alpine ?

Collaborator

rhuss commented Aug 23, 2017

fwiw, there was a TLS issue with Java 8u121 and 8u131 (and prior). On Alpine Linux only, though (see e.g. docker-library/openjdk#115)

wonder whether your Jenkins images are based on Alpine ?

@ernestm

This comment has been minimized.

Show comment
Hide comment
@ernestm

ernestm Aug 23, 2017

I had this happening to me on plain Ubuntu and OSX both, not inside docker, so it must be more widespread than that.

ernestm commented Aug 23, 2017

I had this happening to me on plain Ubuntu and OSX both, not inside docker, so it must be more widespread than that.

@kogitant

This comment has been minimized.

Show comment
Hide comment
@kogitant

kogitant Aug 23, 2017

In my case I was using AWS EC2 instances as jenkins slaves. The base ami was Ubuntu Zesty amd64 ami-1446b66d. Jdk8u112 was installed by Jenkins from Oracle site.

kogitant commented Aug 23, 2017

In my case I was using AWS EC2 instances as jenkins slaves. The base ami was Ubuntu Zesty amd64 ami-1446b66d. Jdk8u112 was installed by Jenkins from Oracle site.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment