Support for arbitrary url schemes?

[gesellix]

I'm trying to use OkHttp to connect to the Docker remote api. Since they support several ways of connecting to their daemon, including unix sockets, they use several url schemes. They still use HTTP, though, but connect via TCP, Unix Domain Sockets or even Names Pipes.

When trying to use a url like unix:///var/run/docker.sock, OkHttp refuses to use that url, because it enforces us to use either http or https.

I know that OkHttp wasn't supposed to handle such hacky ways of connecting via Unix Sockets, which is why I won't ask for a solution directly in OkHttp itself. My concern is more about the strict parser to only accept http(s). Would you be willing to add an option to the url parser to accept arbitrary schemes? I guess it would even be enough to support some kind of overrides. For me it would be enough to tell OkHttp that I know what I'm doing.

For the bigger picture: given the url parser to accept arbitrary schemes, I still need to provide my own SocketFactory (to connect via unix or npipe). I assume that's possible? Essentially I only want to use OkHttp to leverage the HTTP stuff, but also be able to connect via different transport mechanisms. If you now tell me that I'm crazy and that you don't want OkHttp to also go crazy, please do so :-)

If you think opening OkHttp to support different connection strategies is a good thing, I'd be happy to provide pull requests, with the first one obviously making the url parser less strict.

[gesellix]

P.S.: I'm talking about this one:

okhttp/okhttp/src/main/java/okhttp3/Request.java

Line 151 in ed943ce

public Builder url(URL url) {

[swankjesse]

Unfortunately the URL standard gets pretty awkward if you need to support schemes like mailto: and isbn: and file:. Not supporting the other schemes means the API can be simpler for HTTP.

There's a simple workaround. Just manually do string replacement to force it to be a HTTP URL.

One problem you might face with using HTTP over UNIX sockets is that the socket factory isn't provided the file path. Though maybe that's a constant anyway.

[gesellix]

@swankjesse thanks for the quick reply.

I don't fully understand your proposed workaround. Do you mean I should use something like the following: http://unix/var/run/docker.sock? I would then have to handle the special url in my SocketFactory, right? I already do similar hacks in my library.

The linked docker-client library internally uses the plain JDK api (HTTPUrlConnection stuff) along with its built in hooks to provide protocol handlers for unix and npipe schemes. The reason I'm trying to replace that implementation with OkHttp simply lies in the fact that I don't want to re-implement another HTTP client, but currently use the sun.net. internal HttpClient. Other libraries are too over-engineered compared to OkHttp, which is why I wanted to give OkHttp a try.

It's ok for me to handle the special unix/npipe/SocketFactory stuff on my own, I simply ask for a more flexible handling of request URLs. Would it be possible to make the static HttpUrl parsed = HttpUrl.get(url) call configurable?

[shekhargulati]

@swankjesse @gesellix I am also trying to do the same thing for my rx-docker-client. I have to support unix socket using okhttp but unable to figure out how to make it work. I created by custom socket factory using the jnr-unixsocket and set that in the OkHttpClient. But, when I execute code it does not use the custom factory and fails.

[swankjesse]

@shekhargulati I’m unsure. Can you provide a test case?

[gesellix]

@swankjesse I tried to use the encoded unix socket filename as hostname. Now I'm a bit lost thinking about a good layer where to hook into. A custom SocketFactory would only be used when the RouteSelector has tried to resolve my hacked hostname. Sadly, the RouteSelector makes some assumptions in regard to DNS lookups.

This is my test case:

    OkHttpClient client = new OkHttpClient();

    Request request = new Request.Builder()
        .url("http://unix--var-run-docker.sock/_ping")
        .build();

    Response response = client.newCall(request).execute();

... yielding that error:

Exception in thread "main" java.net.UnknownHostException: unix--var-run-docker.sock: nodename nor servname provided, or not known
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1295)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1248)
    at java.net.InetAddress.getAllByName(InetAddress.java:1164)
    at java.net.InetAddress.getAllByName(InetAddress.java:1098)
    at okhttp3.Dns$1.lookup(Dns.java:39)
    at okhttp3.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:173)

Where would you suggest me to customise the client? I see several options like HttpEngine, StreamAllocation, and RouteSelector. Reading the docs I feel like hooking/configuring the StreamAllocation should be a good level. I wouldn't like to hook into the HttpEngine, because that feels a bit too high level, while the RouteSelector doesn't even seem to be necessary for my needs.
What do you think?

[gesellix]

Ah, sorry to bother yourself... I tried it with localhost (http://localhost/unix--var-run-docker.sock/_ping) and now it uses my SocketFactory.

Now I'm where you probably have already seen the actual problem:

One problem you might face with using HTTP over UNIX sockets is that the socket factory isn't provided the file path. Though maybe that's a constant anyway.

Yes, the information is lost. I assume it's a no-go to pass the path down into RealConnection, right?

[gesellix]

In case someone is interested in the WIP, it can be found there https://github.com/gesellix/okhttp/blob/master/samples/simple-client/src/main/java/okhttp3/sample/OkDocker.java

[swankjesse]

Here’s a class that might help:
https://gist.github.com/swankjesse/7ffbc7f7ec9d6fc9deb026d8726eae84

Use it like this:

    UnixSocketFactory socketFactory = new UnixSocketFactory();

    OkHttpClient client = new OkHttpClient.Builder()
        .socketFactory(socketFactory)
        .dns(socketFactory)
        .build();

    HttpUrl url = socketFactory.urlForPath("/var/run/docker.sock");
    Call call = client.newCall(new Request.Builder()
        .url(url)
        .build());
    Response response = call.execute();

You’ll still need to figure out what else goes inside the connect() method. All this does is retrieve the path through a backchannel.

[gesellix]

well, that's nice, thanks!

[gesellix]

@swankjesse I decided to encode the hostname - as suggested. Thanks for the hint to also hook into the Dns lookups. Your example worked great and I don't even need any change in OkHttp :-)

I guess I'm now able to continue by myself, so I'm closing this issue.

Thanks to the whole team for that great library!

[swankjesse]

Great. Also you might find Okio’s ByteString to be a little more natural to go between hex and utf8:

      static String encode(String text) {
        return ByteString.encodeUtf8(text).hex();
      }

      static String decode(String hex) {
        return ByteString.decodeHex(hex).utf8();
      }

[gesellix]

done - looks like I need to browse even more of your api...