Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for arbitrary url schemes? #2576

Closed
gesellix opened this issue May 23, 2016 · 13 comments
Closed

Support for arbitrary url schemes? #2576

gesellix opened this issue May 23, 2016 · 13 comments

Comments

@gesellix
Copy link

gesellix commented May 23, 2016

I'm trying to use OkHttp to connect to the Docker remote api. Since they support several ways of connecting to their daemon, including unix sockets, they use several url schemes. They still use HTTP, though, but connect via TCP, Unix Domain Sockets or even Names Pipes.

When trying to use a url like unix:///var/run/docker.sock, OkHttp refuses to use that url, because it enforces us to use either http or https.

I know that OkHttp wasn't supposed to handle such hacky ways of connecting via Unix Sockets, which is why I won't ask for a solution directly in OkHttp itself. My concern is more about the strict parser to only accept http(s). Would you be willing to add an option to the url parser to accept arbitrary schemes? I guess it would even be enough to support some kind of overrides. For me it would be enough to tell OkHttp that I know what I'm doing.

For the bigger picture: given the url parser to accept arbitrary schemes, I still need to provide my own SocketFactory (to connect via unix or npipe). I assume that's possible? Essentially I only want to use OkHttp to leverage the HTTP stuff, but also be able to connect via different transport mechanisms. If you now tell me that I'm crazy and that you don't want OkHttp to also go crazy, please do so :-)

If you think opening OkHttp to support different connection strategies is a good thing, I'd be happy to provide pull requests, with the first one obviously making the url parser less strict.

@gesellix
Copy link
Author

P.S.: I'm talking about this one:

public Builder url(URL url) {

@swankjesse
Copy link
Member

Unfortunately the URL standard gets pretty awkward if you need to support schemes like mailto: and isbn: and file:. Not supporting the other schemes means the API can be simpler for HTTP.

There's a simple workaround. Just manually do string replacement to force it to be a HTTP URL.

One problem you might face with using HTTP over UNIX sockets is that the socket factory isn't provided the file path. Though maybe that's a constant anyway.

@gesellix
Copy link
Author

@swankjesse thanks for the quick reply.

I don't fully understand your proposed workaround. Do you mean I should use something like the following: http://unix/var/run/docker.sock? I would then have to handle the special url in my SocketFactory, right? I already do similar hacks in my library.

The linked docker-client library internally uses the plain JDK api (HTTPUrlConnection stuff) along with its built in hooks to provide protocol handlers for unix and npipe schemes. The reason I'm trying to replace that implementation with OkHttp simply lies in the fact that I don't want to re-implement another HTTP client, but currently use the sun.net. internal HttpClient. Other libraries are too over-engineered compared to OkHttp, which is why I wanted to give OkHttp a try.

It's ok for me to handle the special unix/npipe/SocketFactory stuff on my own, I simply ask for a more flexible handling of request URLs. Would it be possible to make the static HttpUrl parsed = HttpUrl.get(url) call configurable?

@shekhargulati
Copy link

@swankjesse @gesellix I am also trying to do the same thing for my rx-docker-client. I have to support unix socket using okhttp but unable to figure out how to make it work. I created by custom socket factory using the jnr-unixsocket and set that in the OkHttpClient. But, when I execute code it does not use the custom factory and fails.

@swankjesse
Copy link
Member

@shekhargulati I’m unsure. Can you provide a test case?

@gesellix
Copy link
Author

@swankjesse I tried to use the encoded unix socket filename as hostname. Now I'm a bit lost thinking about a good layer where to hook into. A custom SocketFactory would only be used when the RouteSelector has tried to resolve my hacked hostname. Sadly, the RouteSelector makes some assumptions in regard to DNS lookups.

This is my test case:

    OkHttpClient client = new OkHttpClient();

    Request request = new Request.Builder()
        .url("http://unix--var-run-docker.sock/_ping")
        .build();

    Response response = client.newCall(request).execute();

... yielding that error:

Exception in thread "main" java.net.UnknownHostException: unix--var-run-docker.sock: nodename nor servname provided, or not known
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1295)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1248)
    at java.net.InetAddress.getAllByName(InetAddress.java:1164)
    at java.net.InetAddress.getAllByName(InetAddress.java:1098)
    at okhttp3.Dns$1.lookup(Dns.java:39)
    at okhttp3.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:173)

Where would you suggest me to customise the client? I see several options like HttpEngine, StreamAllocation, and RouteSelector. Reading the docs I feel like hooking/configuring the StreamAllocation should be a good level. I wouldn't like to hook into the HttpEngine, because that feels a bit too high level, while the RouteSelector doesn't even seem to be necessary for my needs.
What do you think?

gesellix added a commit to gesellix/okhttp that referenced this issue May 27, 2016
@gesellix
Copy link
Author

Ah, sorry to bother yourself... I tried it with localhost (http://localhost/unix--var-run-docker.sock/_ping) and now it uses my SocketFactory.

Now I'm where you probably have already seen the actual problem:

One problem you might face with using HTTP over UNIX sockets is that the socket factory isn't provided the file path. Though maybe that's a constant anyway.

Yes, the information is lost. I assume it's a no-go to pass the path down into RealConnection, right?

@gesellix
Copy link
Author

In case someone is interested in the WIP, it can be found there https://github.com/gesellix/okhttp/blob/master/samples/simple-client/src/main/java/okhttp3/sample/OkDocker.java

@swankjesse
Copy link
Member

Here’s a class that might help:
https://gist.github.com/swankjesse/7ffbc7f7ec9d6fc9deb026d8726eae84

Use it like this:

    UnixSocketFactory socketFactory = new UnixSocketFactory();

    OkHttpClient client = new OkHttpClient.Builder()
        .socketFactory(socketFactory)
        .dns(socketFactory)
        .build();

    HttpUrl url = socketFactory.urlForPath("/var/run/docker.sock");
    Call call = client.newCall(new Request.Builder()
        .url(url)
        .build());
    Response response = call.execute();

You’ll still need to figure out what else goes inside the connect() method. All this does is retrieve the path through a backchannel.

@gesellix
Copy link
Author

well, that's nice, thanks!

@gesellix
Copy link
Author

gesellix commented May 28, 2016

@swankjesse I decided to encode the hostname - as suggested. Thanks for the hint to also hook into the Dns lookups. Your example worked great and I don't even need any change in OkHttp :-)

I guess I'm now able to continue by myself, so I'm closing this issue.

Thanks to the whole team for that great library!

@swankjesse
Copy link
Member

swankjesse commented May 28, 2016

Great. Also you might find Okio’s ByteString to be a little more natural to go between hex and utf8:

      static String encode(String text) {
        return ByteString.encodeUtf8(text).hex();
      }

      static String decode(String hex) {
        return ByteString.decodeHex(hex).utf8();
      }

@gesellix
Copy link
Author

done - looks like I need to browse even more of your api...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants