Skip to content

not found subproto extension: binary, base64 #28

Closed
yazgoo opened this Issue Jul 25, 2013 · 19 comments

4 participants

@yazgoo
yazgoo commented Jul 25, 2013

Hi,

While trying to connect to mqtt using this library:
http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.javascript.git/

I get the following log

./src/lighttpd -D -f my_config -m src/.libs/                                                    
2013-07-25 09:04:13: (log.c.166) server started 
2013-07-25 09:04:22: (mod_websocket.c.520) found extension: /mqtt 
2013-07-25 09:04:22: (mod_websocket_handshake.c.407) Sec-WebSocket-Version: 13 
2013-07-25 09:04:22: (mod_websocket_handshake.c.329) not found subproto extension: binary, base64 
2013-07-25 09:04:22: (mod_websocket.c.549) disconnect client fd: 5 

And Here is my config:

server.document-root = "/tmp/"
server.port = 8080
server.modules += ( "mod_websocket" )
websocket.server = ( "/mqtt" =>
        (
         "host" => "127.0.0.1",
         "port" => "1883"
        )
        )
websocket.debug = 4

On the client side, here is the error I get

"NetworkError: 404 Not Found - http://192.168.0.11:8080/mqtt"
mqtt
Object { errorCode=7, errorMessage="AMQJS0007E Socket error:undefined."}
light.js (ligne 150)
Firefox can't establish a connection to the server at ws://192.168.0.11:8080/mqtt.
this.socket = new WebSocket(wsurl, ['binary', 'base64']);

Thanks

@nori0428
Owner

Hello.

I'm not familiar with MQTT.
But if the destination server is a WebSocket server, you should not need the mod_websocket.
Do only like this.
var ws = new WebSocket('ws://ipaddr_of_MQTT_server:1883/mqtt');

On the other hand, if the destination is a TCP server, you should not need the subprotocol.
this.socket = new WebSocket(wsurl);

Regards.

@ralight
ralight commented Jul 26, 2013

Try this as your config instead:

server.document-root = "/tmp/"
server.port = 8080
server.modules += ( "mod_websocket" )
websocket.server = ( "/mqtt" =>
        (
         "host" => "127.0.0.1",
         "port" => "1883",
         "subproto" => "mqtt",
         "type" => "bin"
        )
        )
websocket.debug = 4
@yazgoo
yazgoo commented Jul 26, 2013

Thank you very much to you both for your help, here is how I got it to run:

lighttpd configuration:

server.document-root = "/tmp/"
server.port = 8080
server.modules += ( "mod_websocket" )
websocket.server = ( "/mqtt" =>
        (
         "host" => "127.0.0.1",
         "port" => "1883",
         "type" => "bin"
        )
        )
websocket.debug = 4

mqttws31.js:

commented the subprotos:

this.socket = new WebSocket(wsurl);//, ['binary', 'base64']);
@nori0428
Owner
nori0428 commented Dec 3, 2013

no more handle subproto on mod_websocket ver 2.9. And no need to handle subproto for TCP mode.

@nori0428 nori0428 closed this Dec 3, 2013
@nori0428 nori0428 reopened this Feb 10, 2014
@nori0428
Owner

Hello, finson.

At the first, I can't understand why you are so angry.This issue closed 2 months ago and no one commented for a long time.
(And I didn't know of its existence at the Stack Overflow until you pointed out.)

If you have some opinions like this time, please add a comment here and notify your opinions to me.

And the second, there's a difficult problem dealing additional header of WebSocket like as Sec-WebSocket-Protocol on the WebSocket - TCP proxy.
Certainly, WebSocket header and protocol have Sec-WebSocket-Protocol header, but TCP header does not have such a field.It means that the mod_websocket can't pass some informations like as Sec-WebSocket-Protocol to the backend.

So I decided not to handle subprotocol field(etc.) at WebSocket-TCP proxy mode.If you want to use subprotocol correctly, you must use WebSocket library with your application.(WebSocket proxy is able to pass Sec-WebSocket-Protocol to the backend.)

Or...If you have another nice solution, please teach me.

Regards.

@finson
finson commented Feb 10, 2014

Nori:

I'm very sorry my note came across as angry. I'm not, and I didn't mean to express that at all. In fact, I think you've provided a very nice tool that I was planning to use. I went into so much detail in my posts just so that you and others more expert than I could perhaps see a simple mistake that I was making.

Perhaps my use case is a little different (or not well thought through!). Let me describe it.

There are existing TCP-based tools that I want to use as message brokers for the MQTT protocol. Specifically, I have installed Mosquitto. It doesn't know anything about websockets, but it works very well as a TCP accessible program. I can't rewrite Mosquitto to use websockets, it's an existing mature product. Also, it's only one of many programs like this.

I also want to use a browser to access the output of Mosquitto. However, websockets are the only way I can create a socket in the browser. So I need an adapter, like mod_websocket.

Note that I don't need to tell Mosquitto what protocol to use, it only talks one language (MQTT). However, in the browser client I do need to know that the server I am talking to plans to talk the same language as the client. Again, I'm using a pre-written library on the client side, and I don't have any choice about what it does when sending a websocket upgrade request. As it happens, it specifies a sub-protocol in the request, which is what I want.

At this point, the upgrade request gets sent to mod_websocket. I agree that you can't send the subproto specifier to the backend with TCP, but I don't think you need to. It's not the server that really needs to know what the client wants, it's the client that needs to know that the server will do the right thing.

mod_websocket has been configured and it is providing access to a specific service. So it would seem to me that if the admin has set a subproto value in the config file, then that value should get sent back to the client in the handshake response. Then the client will be happy. If it doesn't like the returned value, then it can close the connection, just as expected. If the returned subproto value is acceptable but not actually correct (the backend actually is talking some other protocol) then that is the fault of the admin, not of mod_websocket.

So my expectation was that you would just echo back to the client whatever subproto had been configured, and would not send it to the backend at all. It's part of the websocket negotiation, not the TCP connection to the backend.

I hope that is clearer (and not angry!). I know that I tend to go on and on when I write things like this, but I'm just trying to provide enough detail, not express hostility or anything like that.

Doug Johnson

@nori0428
Owner

Thank you for your reply, Johnson.
I'm not good at English, so I am relieved to find that you are not angry:)

I know that mosquitto's js lib use subprotocols and has less meaning it now.
But in the future, the subprotocol field may be that it has an important meaning.
I'm afraid it and so I decided to not handle subprotocol at mod_websocket.
And mod_websocket does not reply Sec-WebSocket-Protocol header as you describe.

The absence of such a field is equivalent to the null value (meaning that if the server does not wish to agree to one of the suggested subprotocols, it MUST NOT send back a |Sec-WebSocket-Protocol| header field in its response

But, I understand that you(and some MQTT users) want to use without changing mosquitto's js lib.So I'll add some codes that is able to work with mosquitto.
Please wait for a while.

Regards.

@nori0428 nori0428 added the Normal label Feb 10, 2014
@nori0428 nori0428 self-assigned this Feb 10, 2014
@nori0428
Owner

Oh, I didn't read correctly.

So it would seem to me that if the admin has set a subproto value in the config file, then that value should get sent back to the client in the handshake response.

It's a nice idea.Thanks. I'll implement with this plan.

@finson
finson commented Feb 10, 2014

Great, thank you!

@nori0428 nori0428 added a commit that referenced this issue Feb 10, 2014
@nori0428 fix issue #28
 - enable to work with mosquitto.
b27c97c
@nori0428
Owner

I comitted.(But not tested yet... I'm busy now, sorry.)
Plz check like below. And If work well, please notify me.
(I don't have any mosquitto environment.)

$ git clone https://github.com/nori0428/mod_websocket
$ git checkout develop
$ ./configure --with-lighttpd=/path/to/lighttpd
$ make install
$ cd /path/to/lighttpd
$ ./autogen.sh
$ ./configure --with-websocket --with-pcre
$ make
$ ./lighttpd -f /path/to/lighttpd.conf

A configuration sample for mosquitto is described at sample/etc/conf.d/websocket.conf.
Regards.

@finson
finson commented Feb 10, 2014

@nori0428 I will check it, but I'm on the road all day today and in a meeting tonight. Soon, though, soon! Thanks.

@finson
finson commented Feb 11, 2014

@nori0428 I didn't get very far.

$ git clone https://github.com/nori0428/mod_websocket
$ cd mod_websocket
$ git checkout develop
$ ./configure --with-lighttpd=../lighttpd1.4

-->-bash: ./configure: No such file or directory.

$ git clone --recursive git://github.com/nori0428/mod_websocket.git
$ cd mod_websocket 
$ git checkout develop
$ autoconf

--> configure.ac:5: error: possibly undefined macro: AM_INIT_AUTOMAKE
If this token and others are legitimate, please use m4_pattern_allow.

$ git clone --recursive git://github.com/nori0428/mod_websocket.git
$ cd mod_websocket 
$ git checkout develop
$ aclocal 
$ autoconf
$ ./configure --with-lighttpd=contrib/lighttpd1.4/

--> configure: error: cannot find install-sh, install.sh, or shtool in m4 "."/m4

Any thoughts on how I can make this work? Presumably it's a configuration difference between our systems or something silly that I am doing, but I'm not a make/configure expert at all and so I'm kind of stuck at this point.

Doug

@nori0428
Owner

Hi Doug, Thank you for your trying to make.
And I'm sorry to forget writing some commands.

$ git clone --recursive git://github.com/nori0428/mod_websocket.git
$ cd mod_websocket
$ git checkout develop
$ ./bootstrap
$ ./configure --with-lighttpd=contrib/lighttpd1.4/

bootstrap is unziping gtest, doing checkint out lighttpd, aclocal, autoconf and automake.
Regards.

@finson
finson commented Feb 12, 2014

Norio:

Progress! I built and now can run the develop branch lighttpd/mod_websocket. There is a failure during the connection handshake, but it's not the same as it was before, and it may be the MQTT connection handshake that has the problem, not the websocket part. I'll do some Wireshark packet captures and be able to tell you more details later.

In the meantime, here is the output from lighttpd/mod_websocket during the browser connect attempt.

2014-02-12 07:33:59: (response.c.553) Path         : /srv/www/mqtt
2014-02-12 07:33:59: (mod_websocket_handshake.c.98) allowed origins are not specified
2014-02-12 07:33:59: (mod_websocket.c.503) works as WebSocket-TCP Proxy
2014-02-12 07:33:59: (mod_websocket.c.510) WebSocket Version = 13
2014-02-12 07:33:59: (mod_websocket.c.531) will recv text data from backend
2014-02-12 07:33:59: (mod_websocket.c.152) try to connect backend -> 127.0.0.1 : 1883
2014-02-12 07:33:59: (mod_websocket.c.180) connected 10.170.9.18 : 10996 ( fd = 6 ) -> 127.0.0.1 : 1883 ( fd = 7 )
2014-02-12 07:33:59: (mod_websocket_handshake.c.416) send handshake response
2014-02-12 07:33:59: (mod_websocket_frame.c.299) recv data from client ( fd = 6 ), size = 0xb6f27258
2014-02-12 07:33:59: (mod_websocket_frame.c.321) type = binary
2014-02-12 07:33:59: (mod_websocket_frame.c.372) specified payload size = 0x06
2014-02-12 07:33:59: (mod_websocket_frame.c.419) read payload, size = 0x06
2014-02-12 07:33:59: (mod_websocket_frame.c.426) rest of frame size = 0x00
2014-02-12 07:33:59: (mod_websocket.c.650) send data to backend ( fd = 7 ), size = 0x02c6dc
2014-02-12 07:33:59: (mod_websocket.c.234) recv data from backend ( fd = 7 ), size = 0x04
2014-02-12 07:33:59: (mod_websocket_frame.c.228) type = text
2014-02-12 07:33:59: (mod_websocket_frame.c.250) payload size = 0x04
2014-02-12 07:33:59: (mod_websocket_frame.c.277) frame size = 0x06
2014-02-12 07:33:59: (mod_websocket.c.696) send data to client ( fd = 6 ), size = 0xb6e1f1f0
2014-02-12 07:33:59: (mod_websocket_frame.c.299) recv data from client ( fd = 6 ), size = 0x00
2014-02-12 07:33:59: (mod_websocket_frame.c.334) type = close
2014-02-12 07:33:59: (mod_websocket.c.637) disconnected from client ( fd = 6 )
2014-02-12 07:33:59: (mod_websocket.c.639) send close response to client ( fd = 6 )
2014-02-12 07:33:59: (mod_websocket_frame.c.245) type = close
2014-02-12 07:33:59: (mod_websocket_frame.c.250) payload size = 0x04
2014-02-12 07:33:59: (mod_websocket_frame.c.277) frame size = 0x06
2014-02-12 07:33:59: (mod_websocket.c.192) disconnect backend ( fd = 7 )
~
@finson
finson commented Feb 13, 2014

mqtt-connack-as-ws-txt
Norio:

I think I found the current problem. (See attached packet capture image.)

After the websocket upgrade has completed, the client sends a message to the backend program. In my case, this is an MQTT connect message. The client sends it as a websocket binary message. In the example, this is packet number 7. mod_websocket passes that on to the backend, and gets back a response in packet 10. The response is a 4 byte binary packet, an MQTT CONNACK. The bytes are 20 02 00 00, indicating that the backend accepts the connection. This response is echoed out to the client by mod_websocket (packet 12), but the websocket packet is marked as Text, not Binary, and so the client rejects it as a malformed packet and the connection is closed.

The mod_websocket config file has these lines in it:

websocket.server = ("/mqtt" =>
                          (
                                "host" => "127.0.0.1",
                                "port" => "1883",
                                "subproto" => "mqttv3.1",
                                "type" => "bin"
                           ),
)

I'm not sure how mod_websocket decides between Text and Binary tagging, but I think my config should force all websocket traffic going to the client to be marked binary. Is that right?

Doug

@nori0428
Owner

Hi, Doug.

I'm sorry.I had a mistake.
It will work well only writing "type" => "binary".

websocket.server = ("/mqtt" =>
                          (
                                "host" => "127.0.0.1",
                                "port" => "1883",
                                "subproto" => "mqttv3.1",
                                "type" => "binary"
                           ),
)

and configuration samples are shown at 'sample/etc/conf.d/websocket.conf.sample'

Regards.

@finson
finson commented Feb 13, 2014

Connected! Thank you for the help and fast updates. Doug

connected-to-mqtt-broker

@nori0428
Owner

Thank you for your trying, Doug.
I'll add test codes, merge into master branch and add tags ...(etc.) soon.

Regards.

@nori0428
Owner

merged into master branch.

@nori0428 nori0428 closed this Feb 14, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.