Boot2Docker and create command throws an exception

[lordofthejars]

Hi I don't know if you have tested the library using boot2docker, but we have tried to use and an exception is thrown.

The exception is something like:

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: javax.ws.rs.ProcessingException: java.net.SocketException: Unexpected end of file from server
at org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:229)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:246)
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:683)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:424)

Maybe it is a boot2docker configuration problem, but we have tried the same code with a linux docker and runs perfectly and also we have tried using curl directly to boot2docker and it worked as well.

Any idea?

[marcuslinke]

@lordofthejars Usually i run the integration tests against boot2docker. Saying this all tests passed with boot2docker 1.3.2 and current v0.10.4 of docker-java

bash-3.2$ boot2docker version
Boot2Docker-cli version: v1.3.2
Git commit: e41a9ae

bash-3.2$ docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa
OS/Arch (client): darwin/amd64
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa

[marcuslinke]

@lordofthejars Could you give me some more details? What kind of operation leads to the stacktrace above?

[lordofthejars]

arquillian/arquillian-cube#49

[marcuslinke]

@lordofthejars To be sure, it's only the case when creating a container?

[lordofthejars]

yes PING works

[marcuslinke]

@lordofthejars Hmmm.... Just a guess. Maybe this is somehow related to TLS/SSL security added with docker 1.3.0? Latest boot2Docker versions switched to TLS/SSL by default. Don't know if PING works without TLS/SSL, so it could be some issue with docker-java configuration of ports or certificates...

[tnine]

@marcuslinke I'm also experiencing this. Try the latest version of boot2docker, v1.4.1

I'm using the properties file as suggested in the documentation with the following properties. Note I'm using TLS and it works fine on all commands with the first client.

~/.docker.io.properties

docker.io.url=https://192.168.59.103:2376
docker.io.version=1.16
docker.io.dockerCertPath=/Users/tnine/.boot2docker/certs/boot2docker-vm

I seem to be experiencing this only when 2 threads in invoke the same client instance (not concurrently)

Thread 1 performs the following

1. Ping
2. Create Image
3. Start Image
   (sleeps)

Thread 2 performs the following

1. Ping
2. Create Image (hangs). Ultimately prints out this error.

Feb 05, 2015 3:40:27 PM ClientCommunicatorAdmin Checker-run
WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out

EDIT:

Just an update, and I changed the test to use only 1 thread. Even if it's the same thread, it always fails on the second invocation.

[tnine]

@marcuslinke @lordofthejars It looks like something is leaking connections. The problem appears to be buried within the apache pool. Specifically, in AbstractConnPool.getPoolEntryBlocking:219. The pool of the state shows this in my debugger

[route: {s}->https://192.168.59.103:2376][leased: 2][available: 0][pending: 0]

As you can see, there's nothing left in the pool, so we're just timing out on borrow. When you create a command, do you explicitly have to close it?

[lordofthejars]

Good question, maybe this is exactly the problem, but if it was then it would not work any test, but in our case there are some tests that are passing.

[aslakknutsen]

@tnine Seen the same behavior in the docker-java test suite. You can configure the maxroute pool in docker-java that makes it work, but.. possible just a short workaround.

If you set configBuilder.withMaxPerRouteConnections(20) here https://github.com/arquillian/arquillian-cube/blob/master/docker/src/main/java/org/arquillian/cube/impl/docker/DockerClientExecutor.java#L108

(And I just woke up, missed this was a docker-java issue, not a Cube issue.. but same same :)

[marcuslinke]

Just found http://phillbarber.blogspot.de/2014/02/lessons-learned-from-connection-leak-in.html. Seems when using javax.ws.core.Response it needs to call close() after consumption of the response. We have plenty of commands that doesn't handle this correctly. Just preparing a fixing commit.

[marcuslinke]

@tnine Could you test with current Snapshot please?