Bare socket implementation & proxy support

[rmckay]

This isn't really a 'bug' per-say, but it would be really nice to have this library implemented on top of the regular java Socket interface rather than the java nio one because the nio stuff doesn't work with proxy servers http://blog.tsunanet.net/2010/08/java-nio-and-overengineering.html

[Davidiusdadi]

When you talk about the socket interface do you mean the server or the client side?

[rmckay]

On Thu, 3 May 2012 09:22:17 -0700, David Rohmer wrote:

When you talk about the socket interface do you mean the server or
the client side?

I'm using it on the client side (actually I'm using
https://github.com/benkay/java-socket.io.client which uses WebSocket
under the hood).

Cheers,
Rob

[Davidiusdadi]

i never needed to authenticate to a proxy when using websockts and also did not considered it during development even though it is covered by the protocol specification http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-17#page-17

So sooner or later i will take care of it.

Regarding this blogpost: Someone discovered that nonblocking io consumes more memory than blocking one...by use of his smart mind and fancy inspection tools. He probability also thinks of wappers as an antipattern because of performance loss duo avoidable function calls.

In case of my humble english could not stop you from despising nio you will probbably want your non nio websocket :)

Probably someone will have to implement such a non nio client then. It would be great if that someone would be you.

But currently there is not so much you can do because the IO related stuff is not yet 100% separated from the rest.
But because we want wss we need that separation anyway.
After pull 83 is through i will make sure that one can build his server or client as he likes.

EDIT: I highly likely can not lock horns with the author of that post in such topics also its fun to act as if :)
Someday....

[TooTallNate]

When I originally wrote this lib, I started off with the regular Socket class. I quickly bumped into walls when it came to trying to read and write data at the same time. At least when I implemented draft 75, non-blocking IO was a requirement because you have to both read and write data at the same time, and using two threads for that would be bad.

Anyways, I'm not speaking for David here, but I doubt this lib will ever have a version that doesn't use non-blocking IO.

[rmckay]

On Sat, 5 May 2012 23:07:03 -0700, Daniel Wu wrote:

NIO works with proxies, it just does not support the global VM proxy
properties (e.g. -Dhttp.proxyHost) which is not flexible (it's kind
of
having everything proxied or nothing proxied) and I never use them.
If
you want nio with proxies you just need a couple of lines of code:

Proxy proxy = new Proxy(Proxy.Type.HTTP, proxyUrl);
URL url = new URL("http://java.sun.com/");
URConnection conn = url.openConnection(proxy);

Although bio is slightly faster in a low concurrent environment, but
it does not scale. bio creates too many threads which put too much
pressure on the CPU context switching, and the JVM will also be
slowed
down by the big heap which bio model requires. Most importantly, you
have a hard limit of native threads on most operating systems. So
many
projects, especially server side projects, are going to drop bio
support totally. AFAIK, jetty 9 will totally remove bio support. Some
fancy languages I have been using like erlang do not provide a bio
model at all, its I/O was written in C with non-blocking model, it
uses a few shared native threads to simulate lightweight process
which
can scale up to millions concurrent processes on a commodity server.
Although Java does not have this programming model at language level
like erlang but you should be able to achieve the same scalability
with the nio model.

Thanks for all the replies..

For my purposes I only want one or possibly two WebSockets so the
overhead is almost completely irrelevant.. it's merely a question of
getting something that works with proxies. I'm not sure I understand
your example lines above -- surely that is for use with regular Java
Sockets, not the SocketChannel nio stuff used by WebSocket.. there
doesn't appear to be any Proxy support in the nio SocketChannel classes
which is basically my problem. If you want proxies you have to implement
your own proxy support on top of the SocketChannel. I might even resort
to some non-java solution - there are libraries you can preload that
take over the operating system socket io functions and add proxy support
(for example http://tsocks.sourceforge.net/).. but I'll play with java
solutions for a bit longer.

I originally started trying to write a replacement SocketChannel class
based on the original Socket class (using different threads for
reading/writing).. and was going to try and hack up a classloader to
poke it in place of the nio one but that turned out to be a pain
(loading my version in place of the 'real one').. still might be worth
doing for other purposes but probably not this.

I then started to hack some code into the WebSocket library to support
an HTTP proxy -- should be fairly simple, just make it spit out

CONNECT :80 HTTP/1.1
Host:

and then eat up the

HTTP/1.1 Connection Established.

message and then resume processing as per usual. With SOCKS it is
pretty similar except a more compact protocol.

http://en.wikipedia.org/wiki/SOCKS#Protocol

it's still just a case of sending one write and expecting one read and
from then on it's just a regular TCP connection.

I haven't quite got this working yet though.. basically I am having
problems handling the "HTTP/1.1 Connection Established." reply which
gets processed by the usual handleRead() stuff which then ends up
closing the connection due to it being unexpected.. I should be able to
get this working though.. I'll post whatever I come up with here
although I'm afraid it may end up being more of a complete hackjob than
something anyone else would want to use ;)

Cheers,
Rob

[DanielYWoo]

Sorry, Rob, I realized I replied too fast and deleted my comment but you still saw it:-). My fault, SocketChannel does not support proxy and we have to tunnel it. Hopefully this can help you a little bit:
http://www.koders.com/java/fid6C544EFC7FC1729D0DD2E500C72B74FD88479F6A.aspx

[arunkjn]

Long time since the last reply. Any progress here?
I am using this in one of my projects.
I need to forward a websockets proxy request from a node.js proxy server to the socket.io server.
It connects directly to the client, but does not support connection through proxy.
Is there any workaround or hack?

[Davidiusdadi]

We don't have proxies yet but there is progress :)

It can be easily implemented the same way ssl has been implemented:

You derive a new class from WrappedByteChannel ( just like SSLSocketChannel2 does) which performs the proxy stuff.
I have not yet looked at how to "talk" to a proxy but it seems to be simple:

CONNECT example.com:80 HTTP/1.1
Host: example.com
Proxy-authorization: Basic ZWRuYW1vZGU6bm9jYXBlcyE=

I think i will have implemented it within the next month.

[arunkjn]

Thanks for the update David.

Will look into this and try to implement proxy support.

I am also considering using an embedded Jetty server in my application to serve websocket content.
Jetty seems to support proxy.

Will let you know what I eventually come up with.

Thanks & regards,
Arun Kumar Jain | +91 9379820103 | www.mu-sigma.comhttp://www.mu-sigma.com |

From: David Rohmer [mailto:notifications@github.com]
Sent: Friday, September 28, 2012 2:02 AM
To: TooTallNate/Java-WebSocket
Cc: Arun Kumar Jain
Subject: Re: [Java-WebSocket] Bare socket implementation & proxy support (#88)

We don't have proxies yet but there is progress :)

It can be easily implemented the same way ssl has been implemented:

You derive a new class from WrappedByteChannel ( just like SSLSocketChannel2 does) which performs the proxy stuff.
I have not yet looked at how to "talk" to a proxy but it seems to be simple:

CONNECT example.com:80 HTTP/1.1
Host: example.com
Proxy-authorization: Basic ZWRuYW1vZGU6bm9jYXBlcyE=

I think i will have implemented it within the next month.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/88#issuecomment-8952936.

---

This email message may contain proprietary, private and confidential information. The information transmitted is intended only for the person(s) or entities to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited and may be illegal. If you received this in error, please contact the sender and delete the message from your system.

Mu Sigma takes all reasonable steps to ensure that its electronic communications are free from viruses. However, given Internet accessibility, the Company cannot accept liability for any virus introduced by this e-mail or any attachment and you are advised to use up-to-date virus checking software.

[leizhu]

@Davidiusdadi ,I have also used Java-WebSocket in my project, and need to use proxy. Does the client support proxy now?

[Davidiusdadi]

Until now there has been no proxy support. But i just introduced basic proxy support in 76d1206.
The default proxy-"handshake" uses this scheme:

CONNECT example.com:80 HTTP/1.1
Host: example.com

I just copied the scheme it from http://tools.ietf.org/html/rfc6455#section-4.1 and did not even tried out if the code works since i currently do not have a proxy at hand.

In order to get better proxy support i would like you to take a look at src/main/example/ProxyClientExample.java and implement what is still missing.

Lets stay in contact to get proxy support working as soon as possible.

[leizhu]

compile:
[javac] Compiling 19 source files to /Users/leizhu/workspace/Cetas/Java-WebSocket/build/classes
[javac] /Users/leizhu/workspace/Cetas/Java-WebSocket/src/main/java/org/java_websocket/client/WebSocketClient.java:71: cannot find symbol
[javac] symbol : class DefaultWebSocketClientFactory
[javac] location: class org.java_websocket.client.WebSocketClient
[javac] private WebSocketClientFactory wsfactory = new DefaultWebSocketClientFactory( this );
[javac] ^
[javac] Note: /Users/leizhu/workspace/Cetas/Java-WebSocket/src/main/java/org/java_websocket/client/DefaultSSLWebSocketClientFactory.java uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 1 error

@Davidiusdadi ,seems that the class DefaultWebSocketClientFactory is missed...Can not compile with the latest codes

[Davidiusdadi]

ops, seems i forgot to commit that file. i will push it after work around 8 pm GMT+1

[netdragonboberb]

Nice! You rock! I'm going to start playing with this immediately. I'll get back to you on results.

Edit: I'll be verbose in describing what I'm seeing because I may not solve this myself and want to share everything I've done, just in case I hit a dead-end. I'll consolidate this later if need be.

After some initial testing ON ANDROID, I think your stuff is working fine, but either some proxy programs like fiddler and charles are having problems with websocket or the Android proxy layer is inserting a rogue colon into the GET line. I can't figure out which yet, but I will find the cause. I single-stepped through your code, but I couldn't find the cause in your code, and it looks like you are creating the HTTP Headers fine. So I suspect the issue is outside your code.

Charles 3.6.5 doesn't seem to support SSL websocket or websocket in general. I tried using both version 8 and 13 (DRAFT_17). For SSL websocket, and then went to Proxy > Proxy Settings > SSL and check "Enable SSL proxying" then add the correct websocket SERVER and port under Locations. I used "*" for port. And I couldn't get websocket to proxy properly through Charles even off of Chrome and Firefox using Javascript, so Charles is obviously busted.

Fiddler 4.4.4.0 is doing something strange and seems to think there is an extra colon after the GET line (but Charles isn't seeing that, so I think it's a Fiddler bug). I am pretty sure Java-Websocket isn't adding this since I'm single-stepping through. So I'm still looking into this and will probably fire up Wireshark on both the Android and computer and try to figure out what's going on. I suspect it's Android's proxy layer. I tried both websocket protocol 13 and 8.

This is what Fiddler is showing, coming from Java-Websocket on Android to Fiddler proxy on my desktop (notice the colon at the end of the Get line - I ran through websocket code line by line and couldn't see this in the buffers or ascii arrays, so I don't suspect this is java-websocket doing it):

VERSION 8:

CONNECT SERVER_NAME_OMMITTED:81 HTTP/1.1
Host: SERVER_NAME_OMMITTED:81
GET /94528e54-fd69-4572-939f-622cc8c6be5b HTTP/1.1:
Connection: Upgrade
Host: SERVER_NAME_OMMITTED:81
Sec-WebSocket-Key: Bh4wcYkU2ZsZ5lMBerXvqA==
Sec-WebSocket-Version: 8
Upgrade: websocket

VERSION 13:

CONNECT SERVER_NAME_OMMITTED:81 HTTP/1.1
Host: SERVER_NAME_OMMITTED:81
GET /94528e54-fd69-4572-939f-622cc8c6be5b HTTP/1.1:
Connection: Upgrade
Host: SERVER_NAME_OMMITTED:81
Sec-WebSocket-Key: Nc3B94eJzdmaO2HzalzkrA==
Sec-WebSocket-Version: 13
Upgrade: websocket

Edit: I found out through wireshark later that Fiddler is adding the port on the Host header.

Update:
Sadly, it COULD be Java-Websocket causing the issue, since I just used Websocket's Javascript implementation in Chrome on the Android, and Fiddler handled it properly. But it could still be Android causing it. So it's time for Wireshark on both ends and see what's being sent out, and received by the proxy, before Fiddler gets to it. If that doesn't tell me what's going on, I'll see if I can find the final point in Java-Websocket before things go to nio and figure out if it looks good there. I'm not confident I've dissected that far enough yet.

For comparison, these are the headers being sent by Chrome:
CONNECT SERVER_NAME_OMMITTED:81 HTTP/1.1
Host: SERVER_NAME_OMMITTED:81
Connection: keep-alive
GET /94528e54-fd69-4572-939f-622cc8c6be5b HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: SERVER_NAME_OMMITTED:81
Origin: SERVER_NAME_OMMITTED
Sec-WebSocket-Key: /4c6cZ/iq7C2inyfQxNlxw==
Sec-WebSocket-Version: 13

Update: So there definitely is no colon in what Wireshark is receiving on the proxy server, so I would suspect Fiddler doesn't like the ordering of HTTP headers, or the content of the headers, or something like that. Here's what Wireshark shows for the proxy packets. One thing that really stands out is the positioning of Upgrade:Websocket not matching the RFC or the example above http://www.rfc-editor.org/rfc/rfc6455.txt In both cases, it looks like Upgrade: websocket should be called before Connection: Upgrade. Now, I'm looking at the draft functions like Draft_75::postProcessHandshakeResponseAsServer and it looks like at least in some cases, it's pushing it in the right order, but maybe they are ending up getting reordered later on, or maybe running through different logic. I'm debugging it now.

Update 2: Upgrade:websocket is definately out of order due to a Map being used in postProcessHandshakeResponseAsServer but that isn't the cause of the issue. Albeit something that should be fixed.

Update 3: I noticed in Wireshark that the Host header in the CONNECT request is missing /r/n/r/n at the end, like Chrome sends. Fiddler was also complaining that the Host header doesn't match the CONNECT hostname. I think there could be two causes: the lack of port on the Host or the /r/n/r/n. However, I believe the port is optional in HTTP (would need to double-check) and even the RFC doesn't show a port on the Host: header, so I suspect it's the lack of /r/n/r/n. Wireshark data below (hostname ommitted)
0000 c0 cb 38 2c 86 f4 10 bf 48 ca 15 8b 08 00 45 60 ..8,.... H.....E`
0010 00 92 be ee 40 00 40 06 c7 3c ac 11 2d ed ac 11 ....@.@. .<..-...
0020 2d cb a9 7a 22 b8 59 f1 c8 d5 ec d2 38 0b 80 18 -..z".Y. ....8...
0030 00 e5 61 a3 00 00 01 01 08 0a 01 b1 76 3b 07 d5 ..a..... ....v;..
0040 49 30 43 4f 4e 4e 45 43 54 20 XX XX XX XX XX XX I0CONNEC T xxxxxx
0050 XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX xxxxxxxx xxxxxxxx
0060 XX XX XX XX XX XX XX 2e 63 6f 6d 3a 38 31 20 48 xxxxxxx. com:81 H
0070 54 54 50 2f 31 2e 31 0a 48 6f 73 74 3a 20 XX XX TTP/1.1. Host: xx
0080 XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX xxxxxxxx xxxxxxxx
0090 XX XX XX XX XX XX XX XX XX XX XX 2e 63 6f 6d 0a xxxxxxxx xxx.com.

Update: Alright, so I solved part of the issue, and now Fiddler is no longer inserting the colon, however I'm still not getting it to send/receive messages through websocket when proxied (yet Chrome has no issue through Fiddler, and I have no issue when not proxied). Anyway, the updated code for WebSocketClient.java . Fiddler didn't like the port not being on the end of Host, and since it's optional, it doesn't hurt to use it. The other important thing was \r\n\r\n needed at the end (I also changed \n to \r\n after HTTP/1.1 to have a proper end of line)

I'll check it in when I've solved the other issues.

WebSocketClient.java line : 414

    public String buildHandShake() {
        StringBuilder b = new StringBuilder();
        String host = uri.getHost();
        b.append( "CONNECT " );
        b.append( host );
        b.append( ":" );
        b.append( getPort() );
        b.append( " HTTP/1.1\r\n" );
        b.append( "Host: " );
        b.append( host );
        b.append( ":" );
        b.append( getPort() );
        b.append( "\r\n\r\n" );

Edit:

I figured out what else was going wrong with the proxy. When websocket sends the proxy CONNECT, the server responds with Connection Established, so if the library receives a Connection Established response, it needs to ignore it and wait for the next response. But acceptHandshakeAsClient is treating it like it should be "Switching Protocols", which will be the next response.

[netdragonboberb]

Update:
Alright, I updated the code to fix the issues, and also changed TreeMap to LinkedHashMap in handshakeDataImpl1 to preserve ordering of header fields so that ordering matches RFC. I forked and I'm going to now check in the changes and send a pull request.

As a reminder, Fiddler proxy will work now but Charles Proxy doesn't seem to handle websocket for any clients.

[agad66]

Hi, i have recently started to use this library for websocket connection and i like it. I would like to continue using it but i have stumbled into a problem. My websocketclient is behind a proxy that needs user authentication and i didn't find any documentation or examples on how to set user and password for proxy. Where can i find some guidance if it is posible to setup user and posible and how. Thank you.

[binwiederhier]

Same here. Bump.

[privatejava]

I have a proxy server which have user and password but I am not sure how to add them in ExampleClient as it only accepts InetSocketAddress in setProxy. Please help

[BrushfireDigitalServices]

I'm going to close this due to changes in the codebase since 2012 and general inactivity in this thread. Feel free to open new issues about specific aspects of proxy support and mention this issue if relevant.

[daluu]

Is it sufficient to say that from https://github.com/TooTallNate/Java-WebSocket/wiki/Using-the-WebSocket-through-a-http-proxy, proxy support is there in this library with exception to (HTTP) proxy with (user + password) authentication?

Although RE: @shuckc's comment, anyone know if it works for Android or just in (desktop) Java? For Java, there seems to be a platform limitation that was fixed a while back (in Java 8?). But not aware of anything similar for Android, there's no mention of the issue or whether there is or isn't HTTP proxy support for sockets in Android. Just asking as it seems we've encountered the mentioned Java limitation error using this library with HTTP proxy specified on Android. I guess for Android, one has to refer to #672.