docker for mac compatibility

[jeantil]

Hello,

I am beta testing docker 4 mac and when I disable docker-machine my test suite fails with

java.lang.UnsatisfiedLinkError: Could not find library in classpath, tried: [libjunixsocket-macosx-1.8-x86_64.dylib, libjunixsocket-macosx-1.5-x86_64.dylib]
    at org.newsclub.net.unix.NativeUnixSocket.load(NativeUnixSocket.java:81)
    at org.newsclub.net.unix.NativeUnixSocket.<clinit>(NativeUnixSocket.java:112)
    at org.newsclub.net.unix.AFUNIXSocket.<init>(AFUNIXSocket.java:36)
    at org.newsclub.net.unix.AFUNIXSocket.newInstance(AFUNIXSocket.java:50)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.ApacheUnixSocket.<init>(ApacheUnixSocket.java:53)
    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:65)

The latest version of test containers (1.0.3 at the time of this writing) uses docker-java 2.2.0 (https://github.com/testcontainers/testcontainers-java/blob/2e227e29eded31c0b2d45a5351cc108994327394/core/pom.xml#L20) so I assume it is safe to say that docker-java 2.2.0 is not compatible with docker 4 mac. (I tried adding the dylib to my classpath and got further errors, also the compiled dylib seems hard to find and it feels very cumbersome to have to add it to the classpath manually).

I want to create a bug report to org.testcontainers asking them to upgrade docker-java to a docker4mac compatible version so I checked out the repository and tried to build master using only docker4mac.
I first got an error in com.github.dockerjava.client.DockerClientTest which seems to be the integration test. The error was that $HOME/.docker/certs did not exist. After creating that directory, I get the following output :

beforeTest(com.github.dockerjava.client.DockerClientTest)  Time elapsed: 2.359 sec  <<< FAILURE!
javax.ws.rs.ProcessingException: org.apache.http.conn.UnsupportedSchemeException: https protocol is not supported
    at com.github.dockerjava.client.DockerClientTest.beforeTest(DockerClientTest.java:32)
Caused by: org.apache.http.conn.UnsupportedSchemeException: https protocol is not supported
    at com.github.dockerjava.client.DockerClientTest.beforeTest(DockerClientTest.java:32)


Results :

Failed tests:
com.github.dockerjava.client.DockerClientTest.beforeTest(com.github.dockerjava.client.DockerClientTest)
  Run 1: DockerClientTest.beforeTest:32->AbstractDockerClientTest.beforeTest:60 » Processing
  Run 2: PASS
  Run 3: PASS
  Run 4: PASS
  Run 5: PASS
  Run 6: PASS
  Run 7: PASS
  Run 8: PASS
  Run 9: PASS
  Run 10: PASS
  Run 11: PASS
  Run 12: PASS
  Run 13: PASS
  Run 14: PASS
  Run 15: PASS
  Run 16: PASS
  Run 17: PASS
  Run 18: PASS
  Run 19: PASS
  Run 20: PASS
  Run 21: PASS
  Run 22: PASS
  Run 23: PASS
  Run 24: PASS
  Run 25: PASS
  Run 26: PASS
  Run 27: PASS
  Run 28: PASS
  Run 29: PASS
  Run 30: PASS
  Run 31: PASS
  Run 32: PASS
  Run 33: PASS
  Run 34: PASS
  Run 35: PASS
  Run 36: PASS
  Run 37: PASS
  Run 38: PASS
  Run 39: PASS
  Run 40: PASS
  Run 41: PASS
  Run 42: PASS
  Run 43: PASS
  Run 44: PASS
  Run 45: PASS
  Run 46: PASS
  Run 47: PASS
  Run 48: PASS
  Run 49: PASS
  Run 50: PASS
  Run 51: PASS
  Run 52: PASS
  Run 53: PASS
  Run 54: PASS
  Run 55: PASS
  Run 56: PASS
  Run 57: PASS
  Run 58: PASS
  Run 59: PASS
  Run 60: PASS
  Run 61: PASS
  Run 62: PASS
  Run 63: PASS
  Run 64: PASS
  Run 65: PASS
  Run 66: PASS
  Run 67: PASS
  Run 68: PASS
  Run 69: PASS
  Run 70: PASS
  Run 71: PASS
  Run 72: PASS
  Run 73: PASS
  Run 74: PASS
  Run 75: PASS
  Run 76: PASS
  Run 77: PASS
  Run 78: PASS
  Run 79: PASS
  Run 80: PASS
  Run 81: PASS
  Run 82: PASS
  Run 83: PASS
  Run 84: PASS
  Run 85: PASS
  Run 86: PASS
  Run 87: PASS
  Run 88: PASS
  Run 89: PASS


Tests run: 462, Failures: 1, Errors: 0, Skipped: 458

And I am unsure what to conclude, the test failure seems to suggest that the lastest master is not docker4mac compatible either but a single failed test seems surprising ...

Can you help me troubleshoot or maybe you already know if master is compatible with docker4mac ...

Thanks for your help
jean

[KostyaSha]

2.2.3 is latest

[KostyaSha]

Do you mean docker 4 mac beta?

[KostyaSha]

java.lang.UnsatisfiedLinkError: Could not find library in classpath, tried: [libjunixsocket-macosx-1.8-x86_64.dylib, libjunixsocket-macosx-1.5-x86_64.dylib]
So NativeUnixSocket from library requires system library that is not available because nobody tried to connect it via socket. As workaround you can try expose rest API and connect via it.

[KostyaSha]

I would look at libraries when i get access to beta :)

[jeantil]

Hello,
Yes I mean docker 4 Mac beta :)

I have no idea how to expose a rest API from the Unix socket or using docker 4mac...

I'll provide further details on the link error tomorrow

[gesellix]

@jeantil I assume you configured the docker client to connect via unix socket at unix:///var/tmp/docker.sock, right?

[gesellix]

@KostyaSha docker-java transitively depends on an older version of junixsocket. An updated release of https://bintray.com/gesellix/docker-utils/unix-socket-factory/2016-04-06T22-21-19/view should show up in Maven Central, soon. Since you're not actually using unix-socket-factory anymore you could also consider to replace the unix-socket-factory dependency with the ones below (Gradle notation). AFAIK this should be a drop in replacement, but beware that it will also pull log4j into your dependencies.

compile 'com.kohlschutter.junixsocket:junixsocket-common:2.0.4'
compile 'com.kohlschutter.junixsocket:junixsocket-native-common:2.0.4'

[KostyaSha]

Project uses https://github.com/docker-java/docker-java/blob/master/pom.xml#L102-L105
Oh, cool, author is here.
@gesellix could make git tags next time https://github.com/gesellix/unix-socket-factory ?

[gesellix]

@KostyaSha next time ;-) but as suggested I would replace the current dependency on unix-socket-factory with the direct dependency on junixsocket.

[gesellix]

as incentive to update: the https://github.com/gesellix/docker-client/ works for me on Docker for Mac using junixsocket 2.0.4.

[KostyaSha]

@gesellix am i right understand that it's groovy client library? 0_o

[gesellix]

if you mean docker-client: yes
junixsocket: no

[marcuslinke]

Please note that this would affect the netty impl also.

[marcuslinke]

Related: netty/netty#2448

[jeantil]

@gesellix

I assume you configured the docker client to connect via unix socket at > unix:///var/tmp/docker.sock, right?

When going through org.testcontainers I have not configured (or tried to configure) anything and I am not sure I could configure. org.testcontainers first tries to find docker-machine, start it, get the ip for the docker host and connect to it using https, if that fails it automatically falls back to unix socket which yields the UnsatisfiedLinkError I mentionned in my initial post.

When building docker-java master from source and running the integration tests I have tried the following in this order:

• mvn clean verify
• DOCKER_OPTS="-H unix:///var/run/docker.sock" mvn clean verify
• DOCKER_OPTS="-H unix:///var/run/docker.sock" mvn -Ddocker.io.url=unix:///var/run/docker.sock clean verify
  (so basically I have no idea what I am doing and I am trying whatever tips I can find in the readme :) )
  None of the above worked. They all fail on the same test with the same error.

@KostyaSha

So NativeUnixSocket from library requires system library that is not available because nobody tried to connect it via socket.

As mentionned in my initial report, I tried manually adding the library to my app classpath (I find it weird that the lib is not embedded in the jar but that might be an artifact of the shading from org.testcontainers).
I downloaded the lib from https://github.com/sleuthkit/hadoop_framework/blob/master/lib/libjunixsocket-macosx-1.5-x86_64.dylib (thank you google)
I used debug to find out that the junixsocket doesn't try to load the native library from the java.library.path, but instead looks it up at a specific location in the classpath. Here is the error I get when I try with the library in place:

DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "POST /containers/108ddc8c37eca70703d01b1c40f4e02b8ab1f1f9836f2e7721c3e26d0f8af83e/start HTTP/1.1[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "Accept: application/json[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "Content-Type: application/json[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "User-Agent: Jersey/2.11 (Apache HttpClient 4.3.1)[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "Transfer-Encoding: chunked[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "Host: localhost:80[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "Connection: Keep-Alive[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "Accept-Encoding: gzip,deflate[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "2[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "{}[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "0[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 >> "[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 << "HTTP/1.1 500 Internal Server Error[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 << "Content-Length: 4[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 << "Content-Type: text/plain; charset=utf-8[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 << "Date: Thu, 07 Apr 2016 08:17:56 GMT[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 << "Server: Docker/1.11.0-rc3 (linux)[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 << "X-Content-Type-Options: nosniff[\r][\n]"
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 << "[\r][\n]"
DEBUG o.t.s.o.a.h.headers - http-outgoing-6 << HTTP/1.1 500 Internal Server Error
DEBUG o.t.s.o.a.h.headers - http-outgoing-6 << Content-Length: 4
DEBUG o.t.s.o.a.h.headers - http-outgoing-6 << Content-Type: text/plain; charset=utf-8
DEBUG o.t.s.o.a.h.headers - http-outgoing-6 << Date: Thu, 07 Apr 2016 08:17:56 GMT
DEBUG o.t.s.o.a.h.headers - http-outgoing-6 << Server: Docker/1.11.0-rc3 (linux)
DEBUG o.t.s.o.a.h.headers - http-outgoing-6 << X-Content-Type-Options: nosniff
DEBUG o.t.s.o.a.h.i.e.MainClientExec - Connection can be kept alive indefinitely
DEBUG o.t.s.o.a.h.wire - http-outgoing-6 << "EOF[\n]"
DEBUG o.t.s.o.a.h.i.c.PoolingHttpClientConnectionManager - Connection [id: 6][route: {}->unix://localhost:80] can be kept alive indefinitely
DEBUG o.t.s.o.a.h.i.c.PoolingHttpClientConnectionManager - Connection released: [id: 6][route: {}->unix://localhost:80][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
ERROR 🐳 [mysql:latest] - Could not start container
org.testcontainers.shaded.com.github.dockerjava.api.InternalServerErrorException: EOF

    at org.testcontainers.shaded.com.github.dockerjava.jaxrs.filter.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:53) ~[testcontainers-1.0.3.jar:na]
    at org.testcontainers.shaded.org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:134) ~[testcontainers-1.0.3.jar:na]
    at org.testcontainers.shaded.org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:123) ~[testcontainers-1.0.3.jar:na]
    at org.testcontainers.shaded.org.glassfish.jersey.process.internal.Stages.process(Stages.java:171) ~[testcontainers-1.0.3.jar:na]
    at org.testcontainers.shaded.org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:251) ~[testcontainers-1.0.3.jar:na]

honestly I gave up at that point ... :) But reading through the detailed logs, it looks like we get an internal server error response from the docker host so maybe I should log this with the docker team. I just want to make sure it's not an API version mismatch (this is using docker-client 2.2.0 and docker4mac is using the very latest docker maybe the API changed in an incompatible way I would love to get confirmation though ... )

If anyone has testcases to run to check out connectivity I would be glad to help and report results here. (I can't give access to the beta, but I am pretty sure if you ask the docker team saying you are a dev of a docker tool you will get access pretty fast)

As workaround you can try expose rest API and connect via it.

I have no idea how to make docker4mac expose a REST API, not sure it is even possible

[KostyaSha]

dylib is system library, i guess that you should place it to LD paths https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/UsingDynamicLibraries.html

[jeantil]

@KostyaSha I know that dylib is a system library, (at least a native library for the jvm) :)

However, as I said, the version of junixsocket used by docker-java 2.2.0 which is embedded in org.testcontainers, _does NOT_ load the native library from the usual java.library.path (which is the list of locations the jvm is trying to resolve native libraries from).
Instead it explicitely looks it up at a specific location in the classpath : /org/newsclub/net/unix
You can check out a decompiled version of org.newsclub.net.unix.NativeUnixSocket to confirm that, I copy the related code below :

            for(int fos = 0; fos < e; ++fos) {
                String r = tmpLib[fos];
                libId = "libjunixsocket-" + os + "-" + r + "-" + arch;
                paths.add(libId + suffix);
                in = NativeUnixSocket.class.getResourceAsStream(libId + suffix);
                if(in != null) {
                    break;
                }
            }

            if(in == null) {
                throw new UnsatisfiedLinkError("Could not find library in classpath, tried: " + paths);
            }

Notice the line in = NativeUnixSocket.class.getResourceAsStream(libId + suffix); this loads the file from the classloader as a byte stream, not as a native library.

It then proceeds to copy the file to a temp directory and tells the jvm to load the file from that location with an absolute path (which bypasses the lookup in java.library.path):

 File var15;
            try {
                var15 = File.createTempFile(libId, suffix);
                var15.deleteOnExit();
                byte[] var16 = new byte[4096];
                FileOutputStream var17 = new FileOutputStream(var15);

                while(true) {
                    int var18;
                    if((var18 = in.read(var16)) == -1) {
                        var17.close();
                        in.close();
                        break;
                    }

                    var17.write(var16, 0, var18);
                }
            } catch (IOException var14) {
                throw (UnsatisfiedLinkError)(new UnsatisfiedLinkError("Could not create temporary file for library")).initCause(var14);
            }

            System.load(var15.getAbsolutePath());

The last line will actually tell the JVM to link with the native library.
This is to be able to load the native library dependencies bundled in a jar file (System.load will only accept an actual path on the filesystem).
Unfortunately, the dependency unix-socket-factory-2015-01-27T15-02-14.jar drawn in by the docker-java 2.2.0 shaded by testcontainers only includes the linux version of the native extension, not the osx version which forces me to download the native extension manually and place it at the expected point on the classpath. (and by the way: unix-socket-factory-2015-01-27T15-02-14.jar contains the classes of junixsocket)

@gesellix mentioned earlier that docker-java should not depend on unix-socket-factory anymore but instead add a direct dependency on junixsocket 2.0.4.

I am not sure what the link to netty is though.

cheers

[gesellix]

@jeantil please note the slight difference between the unix addresses: we have unix:///var/run/docker.sock for docker-machine or boot2docker setups, while Docker for Mac/Windows uses unix:///var/tmp/docker.sock.

I'm not sure about the DOCKER_OPTS variable - the official name would be DOCKER_HOST. Is that a docker-java specific option? ( @marcuslinke @KostyaSha )
I'm also not sure about the status 500 - it could mean more or less anything, but I wouldn't expect it to be a bug. We might infer that you had a connection to the Docker engine, though. For testing basic correctness, I wouldn't try to start a container, but request the /_ping, /info or /version endpoints.

[jeantil]

@gesellix thanks for that pointer this might explain why my tests failed
regarding DOCKER OPTS I followed the readme at the end of the build with maven section

For testing basic correctness, I wouldn't try to start a container, but request the /_ping, /info or /version endpoints.

I haven't had the time to dig deeper into this, I just wanted to make a sanity check on whether my test suite would work without docker machine. the test suite uses testcontainers to start a container. I only pasted the log of the container start failing, there are many others before.

When my project stops being such a mess, I'll look into this deeper (unless someone beats me to it).

[gesellix]

@jeantil ah, yes. It's not very obvious, but the docs say that DOCKER_OPTS are meant to configure your Docker engine/daemon:

To allow Docker server to use TCP add the following line to /etc/default/docker

[gesellix]

So, I'd suggest to give DOCKER_HOST="unix:///var/tmp/docker.sock" a try when running a client (still a wild guess, I'm not a docker-java maintainer :-) ).

[gesellix]

another low level test to verify basic correctness of your Docker for Mac env:

curl --unix-socket /var/run/docker.sock -XGET http://localhost/info

[jeantil]

It's not very obvious, but the docs say that DOCKER_OPTS are meant to configure your Docker engine/daemon:

I am not sure I understand what you mean and

$>ll /etc/default/docker
ls: /etc/default/docker: No such file or directory

So i guess this is a linux specific thing ..

The build is running as I write with DOCKER_HOST set manually but docker4mac doesn't seem to export it automatically to the shells (would be nice if the client could check both unix sockets)

the docker4mac environment works fine, I can start and stop containers with the cli without any problem and the low level test works too :

$>curl --unix-socket /var/run/docker.sock -XGET http://localhost/info | jq '.'
{
  "ID": "ZWCF:GAI4:CLS4:5YFG:X567:3FJ4:2HHM:FS7Y:K6SJ:LBDP:FENO:KBEG",
  "Containers": 7,
  "ContainersRunning": 0,
  "ContainersPaused": 0,
  "ContainersStopped": 7,
  "Images": 4,
  "Driver": "aufs",
  "DriverStatus": [
    [
      "Root Dir",
      "/var/lib/docker/aufs"
    ],
    [
      "Backing Filesystem",
      "extfs"
    ],
    [
      "Dirs",
      "41"
    ],
    [
      "Dirperm1 Supported",
      "true"
    ]
  ],
  "SystemStatus": null,
  "Plugins": {
    "Volume": [
      "local"
    ],
    "Network": [
      "bridge",
      "null",
      "host"
    ],
    "Authorization": null
  },
  "MemoryLimit": true,
  "SwapLimit": true,
  "KernelMemory": true,
  "CpuCfsPeriod": true,
  "CpuCfsQuota": true,
  "CPUShares": true,
  "CPUSet": true,
  "IPv4Forwarding": true,
  "BridgeNfIptables": true,
  "BridgeNfIp6tables": true,
  "Debug": true,
  "NFd": 16,
  "OomKillDisable": true,
  "NGoroutines": 36,
  "SystemTime": "2016-04-07T09:42:16.681950304Z",
  "ExecutionDriver": "",
  "LoggingDriver": "json-file",
  "CgroupDriver": "cgroupfs",
  "NEventsListener": 1,
  "KernelVersion": "4.4.6",
  "OperatingSystem": "Alpine Linux v3.3",
  "OSType": "linux",
  "Architecture": "x86_64",
  "IndexServerAddress": "https://index.docker.io/v1/",
  "RegistryConfig": {
    "InsecureRegistryCIDRs": [
      "127.0.0.0/8"
    ],
    "IndexConfigs": {
      "docker.io": {
        "Name": "docker.io",
        "Mirrors": null,
        "Secure": true,
        "Official": true
      }
    },
    "Mirrors": null
  },
  "NCPU": 4,
  "MemTotal": 2100342784,
  "DockerRootDir": "/var/lib/docker",
  "HttpProxy": "",
  "HttpsProxy": "",
  "NoProxy": "",
  "Name": "docker",
  "Labels": null,
  "ExperimentalBuild": false,
  "ServerVersion": "1.11.0-rc3",
  "ClusterStore": "",
  "ClusterAdvertise": ""
}

[jeantil]

$>ll /var/tmp/docker.sock
srwxr-xr-x  1 jean  wheel     0B  6 avr 11:26 /var/tmp/docker.sock
$>export DOCKER_HOST="unix:///var/tmp/docker.sock"
$>mvn clean integration-test
beforeTest(com.github.dockerjava.client.DockerClientTest)  Time elapsed: 2.833 sec  <<< FAILURE!
javax.ws.rs.ProcessingException: Could not initialize class org.newsclub.net.unix.NativeUnixSocket
    at com.github.dockerjava.client.DockerClientTest.beforeTest(DockerClientTest.java:32)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.newsclub.net.unix.NativeUnixSocket
    at com.github.dockerjava.client.DockerClientTest.beforeTest(DockerClientTest.java:32)


Results :

Failed tests:
com.github.dockerjava.client.DockerClientTest.beforeTest(com.github.dockerjava.client.DockerClientTest)
  Run 1: DockerClientTest.beforeTest:32->AbstractDockerClientTest.beforeTest:60 » Processing
  Run 2: PASS
  Run 3: PASS
  Run 4: PASS
  Run 5: PASS
  Run 6: PASS
  Run 7: PASS
  Run 8: PASS
  Run 9: PASS
  Run 10: PASS
  Run 11: PASS
  Run 12: PASS
  Run 13: PASS
  Run 14: PASS
  Run 15: PASS
  Run 16: PASS
  Run 17: PASS
  Run 18: PASS
  Run 19: PASS
  Run 20: PASS
  Run 21: PASS
  Run 22: PASS
  Run 23: PASS
  Run 24: PASS
  Run 25: PASS
  Run 26: PASS
  Run 27: PASS
  Run 28: PASS
  Run 29: PASS
  Run 30: PASS
  Run 31: PASS
  Run 32: PASS
  Run 33: PASS
  Run 34: PASS
  Run 35: PASS
  Run 36: PASS
  Run 37: PASS
  Run 38: PASS
  Run 39: PASS
  Run 40: PASS
  Run 41: PASS
  Run 42: PASS
  Run 43: PASS
  Run 44: PASS
  Run 45: PASS
  Run 46: PASS
  Run 47: PASS
  Run 48: PASS
  Run 49: PASS
  Run 50: PASS
  Run 51: PASS
  Run 52: PASS
  Run 53: PASS
  Run 54: PASS
  Run 55: PASS
  Run 56: PASS
  Run 57: PASS
  Run 58: PASS
  Run 59: PASS
  Run 60: PASS
  Run 61: PASS
  Run 62: PASS
  Run 63: PASS
  Run 64: PASS
  Run 65: PASS
  Run 66: PASS
  Run 67: PASS
  Run 68: PASS
  Run 69: PASS
  Run 70: PASS
  Run 71: PASS
  Run 72: PASS
  Run 73: PASS
  Run 74: PASS
  Run 75: PASS
  Run 76: PASS
  Run 77: PASS
  Run 78: PASS
  Run 79: PASS
  Run 80: PASS
  Run 81: PASS
  Run 82: PASS
  Run 83: PASS
  Run 84: PASS
  Run 85: PASS
  Run 86: PASS
  Run 87: PASS
  Run 88: PASS
  Run 89: PASS


Tests run: 462, Failures: 1, Errors: 0, Skipped: 458

Same error in the integration tests. docker-java doesn't seem to care about DOCKER_HOST :)

[jeantil]

Regarding DOCKER_OPTS, I think I see what you mean, this would alter the way the docker daemon works. I don't think it is desirable in this case. As a user I would expect the client to try the various default urls until it finds one which works.
Maybe what I got wrong is the

However you can force docker-java to use UNIX socket communication by configuring the following url:

unix:///var/run/docker.sock

section. It doesn't say how to configure the famous url (which in my case would be unix:///var/tmp/docker.sock) and explicitely setting DOCKER_HOST doesn't seem to work, so it would seem I have to look elsewher.

Thanks for your pointers though :)

[gesellix]

@jeantil yeah, I also don't know how to apply the configuration to docker-java, so I guess one of the maintainers can help.

About the clients checking different well known defaults I wouldn't be so content, because the official/native docker client also doesn't behave that way. I'll give the Docker for Mac maintainers feedback about that, though.

[jeantil]

About the clients checking different well known defaults I wouldn't be so content,

Maybe you are right and this should be the responsibility of the tools using the client to specify how they want to handle fallback. That's the case at the moment ( testcontainers is the one implementing the fallback strategies using the various strategies proposed by docker-java )

[gesellix]

request for a consistent default connection url: https://forums.docker.com/t/can-we-have-unique-default-connection-urls-across-platforms/8633

[gesellix]

you can safely forget the stuff with /var/run/docker/sock vs. /var/tmp/docker.sock. Both are available (sym-linked) on Docker for Mac, so you can expect to find /var/run/docker/sock as ultimate default.

[andyp1per]

Ok I just added these as dependencies to the docker-maven-plugin:

                            <dependency>
                                <groupId>com.kohlschutter.junixsocket</groupId>
                                <artifactId>junixsocket-common</artifactId>
                                <version>2.0.4</version>
                            </dependency>
                            <dependency>
                                <groupId>com.kohlschutter.junixsocket</groupId>
                                <artifactId>junixsocket-native-common</artifactId>
                                <version>2.0.4</version>
                            </dependency>

Built the 2.0.4 library and installed the jni libraries

sudo cp ./junixsocket-native/target/nar/junixsocket-native-2.0.4-x86_64-MacOSX-gpp-jni/lib/x86_64-MacOSX-gpp/jni/libjunixsocket-native-2.0.4.* /Library/Java/Extensions/

and all was good. So simply updating the dependencies moves this forward a fair way for me.

[gesellix]

@andyp1per you might consider to enforce it via dependency management - but as @marcuslinke mentioned this won't be the fix for the Netty based implementation.

@normanmaurer would it be an option to start with a blocking implementation in Netty and "fix" that as soon as the other projects have tackled their issues Netty depends on? I'm aware that such a hack cannot be considered as a solution, but I would also hope that from the user's perspective the interface won't change when switching from the hack to a better implementation.

Maybe we can find a way to make it just work without the risk to break a consumer's code too much.

[normanmaurer]

@gesellix let me see what I can do... Super busy atm tho :( Can you link me to the code in docker-java that uses the netty epoll stuff for domain sockets ?

[gesellix]

@normanmaurer Since I'm not involved in the docker-java development, I'd ask @KostyaSha or @marcuslinke for that code pointer :-)

[KostyaSha]

@normanmaurer @gesellix maybe here https://github.com/docker-java/docker-https://github.com/docker-java/docker-java/blob/master/src/main/java/com/github/dockerjava/netty/NettyDockerCmdExecFactory.java#L217 ?

[andyp1per]

Further to my previous update I'm happy to report that updating the dependencies is the only thing that needs to be done to make this work on OSX. Can I request that you release a patch release with this change since it fixes the broken current state of affairs.

[KostyaSha]

@andyp1per for netty?

[andyp1per]

No, using the existing mechanisms. If you remember the issue that started this was java.lang.UnsatisfiedLinkError - turns out this is simply because the code is transitively referencing an old version of junixsocket. If you fix this then life is good :)

[KostyaSha]

No, using the existing mechanisms.

What is the existing mechanisms? Jersey was planned for removal, netty need to be fixed. junixsocket update failed some dependencies, so i didn't PR/merge it. If it works now, then feel free to PR change.

[andyp1per]

Can you point me at the PR?
From the stack this looks like its using jersey + junixsocket

[andyp1per]

Ok I created a PR that works for me:
#677

[KostyaSha]

Thanks

On 23 Aug 2016 2:30 p.m., "Andy Piper" notifications@github.com wrote:

Ok I created a PR that works for me:
#677 #677

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#537 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOIu4pzh_v5aqJrcbGDoLKH_DCPj8IEks5qitnpgaJpZM4IA_rm
.

[marcuslinke]

@KostyaSha, @normanmaurer I've created an experimental branch that uses https://github.com/jnr/jnr-unixsocket for netty implementation. Basically it seems to run on linux but should support macos also. So any interested people might try out https://github.com/docker-java/docker-java/tree/unixsocket-experiment now. If it works with macos in general then maybe we can fix the issues mentioned by @normanmaurer here netty/netty#2448

[KostyaSha]

@marcuslinke how it can work on macos if there is no kqueue support anywhere?
Open PR to see real changes, nobody will investigate branch commits.
Also there is already #677.

[marcuslinke]

@KostyaSha https://github.com/jnr/jnr-unixsocket supports macos via kqueue. I'm working on implementation of an java.nio.channel.SocketChannel which is supported by netty. You or someone with native docker for mac have to try if InfoCmdExecTest runs with https://github.com/docker-java/docker-java/tree/unixsocket-experiment. The branch is experimental and not up-to-date so a PR will make no sense at this point.

As I mentioned before it works BASICALLY on linux but most tests still failing because the implementation is (still) buggy.

[KostyaSha]

@marcuslinke rebase it to get matrix travis builds against different versions.

[marcuslinke]

for reference: #692

[tamershahin]

any update on this? I

[tamershahin]

@andyp1per : you wrote:

Ok I just added these as dependencies to the docker-maven-plugin:

                        <dependency>
                            <groupId>com.kohlschutter.junixsocket</groupId>
                            <artifactId>junixsocket-common</artifactId>
                            <version>2.0.4</version>
                        </dependency>
                        <dependency>
                            <groupId>com.kohlschutter.junixsocket</groupId>
                            <artifactId>junixsocket-native-common</artifactId>
                            <version>2.0.4</version>
                        </dependency>

Built the 2.0.4 library and installed the jni libraries

sudo cp ./junixsocket-native/target/nar/junixsocket-native-2.0.4-x86_64-MacOSX-gpp-jni/lib/x86_64-MacOSX-gpp/jni/libjunixsocket-native-2.0.4.* /Library/Java/Extensions/

and all was good. So simply updating the dependencies moves this forward a fair way for me.

---

can you please add some more info about how you built the 2.0.4 library?

[andyp1per]

A colleague reported that they didn't need to build the library, so I think just updating the dependencies works.

[sscarduzio]

I'm trying to run my tests in OSX (built against testcontainers:1.1.10, based on docker-java:3.0.7).
The tests succeed in Linux, but I have this in OSX:

  2> Exception in thread "Thread-2" java.lang.ExceptionInInitializerError
  2> 	at org.newsclub.net.unix.AFUNIXSocket.<init>(AFUNIXSocket.java:36)
  2> 	at org.newsclub.net.unix.AFUNIXSocket.newInstance(AFUNIXSocket.java:54)
  2> 	at org.rnorth.tcpunixsocketproxy.TcpToUnixSocketProxy.lambda$start$0(TcpToUnixSocketProxy.java:86)
  2> 	at java.lang.Thread.run(Thread.java:745)
  2> Caused by: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException
  2> 	at org.newsclub.net.unix.NativeUnixSocket.<clinit>(NativeUnixSocket.java:42)
  2> 	... 4 more
  2> Caused by: java.lang.reflect.InvocationTargetException
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.lang.reflect.Method.invoke(Method.java:498)
  2> 	at org.newsclub.net.unix.NativeUnixSocket.<clinit>(NativeUnixSocket.java:35)
  2> 	... 4 more
  2> Caused by: java.lang.UnsatisfiedLinkError: Expecting an absolute path of the library: ./temp/libjunixsocket-native-2.0.49049338348292982436.dylib
  2> 	at java.lang.Runtime.load0(Runtime.java:806)
  2> 	at java.lang.System.load(System.java:1086)
  2> 	at org.newsclub.net.unix.NarSystem.loadLibrary(NarSystem.java:39)
  2> 	... 9 more

NB: This happens even if I already:

• added 2.0.4 deps to my project as shown above
• cloned, compiled junixsockets 2.0.4 and sudo-copied the files to /Library/Java/Extensions

Did I miss some steps?

[vyshkov]

Hi, i have similar issue on windows 10.
http://stackoverflow.com/questions/42875510/cant-build-docker-image-via-gradle-on-windows-10
Any ideas ?

[deinspanjer]

With the latest version of Docker for Mac (17.04) and IntelliJ IDEA (2017.1), socket communication now works.

I just had to edit the configuration for my local Docker connection in IDEA and change the API URL back to unix:///var/run/docker.sock .

[xuchenCN]

Hi @deinspanjer I encountered exception Only supported on Linux; Could you give some suggestion?

[deinspanjer]

@xuchenCN What version of Docker for Mac and IntelliJ IDEA are you using? Is IDEA community or Ultimate? I forgot to mention in my comment above, but I am using the Ultimate version. Not sure if that matters. I am also running OSX 10.11.6.

Did you change the local Docker connection API URL?

[kohlschuetter]

junixsocket 2.2 now supports Windows 10 (as well as macOS, and Linux on Intel and ARM), please give it a try!