Describe Raw transport

[oberstet]

WAMP can run over Raw transports instead of WebSocket.

Each message is prefixed with a uint32 (big endian) that provides the (serialized) length of the following WAMP message.

This works for both JSON and MsgPack serialization.

[beatgammit]

I'm not particularly satisfied with this for a couple reasons:

• For many applications, most messages will be small (<256 bytes), so there's 3 unnecessary bytes
• No way to negotiate an encoding

For the first, instead of using 4 bytes, WAMP could use a chaining approach (if the first byte is 0xff, there's one more byte following). This results in a variable-length big-endian number. The max number of bytes can be limited for now, but extended later. Or the WebSocket framing could be used (page 13), which has 7-bit length specifier (if == 126, the next 16 bits are used, if == 127, the next 64 are used). I'm sure there are other chaining strategies that could be considered.

I'm more worried about negotiating an encoding. Once of the stated benefits of RawSocket is that it can be run behind a firewall (e.g. "misused" 443). This means the server must choose a single encoding for all clients.

One possible solution is that clients can send a JSON HELLO message with the encoding specified in the Details argument. If the client wants to operate over an encoding other than JSON, it could assume the rest of the conversation is encoding that way. If the server doesn't support the encoding, the server would fallback to JSON.

I'm not sure if the complexity is currently worth it, but it seems like some kind of protocol negotiation is a better long-term strategy. From what I've seen, WAMP is designed for long-lived connections, so having a sizable connection overhead for negotiation is not a big deal.

I'm not as worried about the message length overhead as I am about extensibility later.

[oberstet]

@beatgammit this reinvent half of WebSocket.

• variable length encoding of integers require a state machine. you cannot just read N octets from wire where N is fixed
• serialization format negotiation: that is transport specific. again require a preflight message exchange with a state machine.

If one wants the features above, there is WebSocket.

So we won't change RawSocket. It's already deployed on the street.

[beatgammit]

If one wants the features above, there is WebSocket.

The problem with websockets is that implementing it requires implementing HTTP, websocket handshaking and message bloat per packet due to the framing. The two features I mentioned only require a small subset of what WebSocket provides, and at least the second only requires a single message before the rest of the WAMP conversation.

I see the benefit of the fixed 4-byte size (though I think 2 bytes is more than sufficient), but I still think a simple encoding negotiation is a good idea.

So we won't change RawSocket. It's already deployed on the street.

I thought that since the Advanced Profile is still in alpha it can still be changed.

Perhaps a middleground is a way to change the encoding in an existing session? This way you don't have to listen on multiple ports, but you can still support multiple encodings. I think the common use-case for this is negotiate over JSON (easier debugging) and later switch to MsgPack for normal operation.

I'm mostly worried about allowing clients that can't support MsgPack communicate on the same port as those that can.

If you're set on not changing the RawSocket spec, I can bring this idea up on the mailing list for discussion.

[oberstet]

Sure, I excaggerated with "half of WS". Regarding "alpha" status of AP: yes, that right in general (for most of AP), but there are some features marked "stable". RawSocket isn't marked - which is a bug.

Rgd: "mostly worried": I can see this point. Definitely. How about:
Rgd RawSocket: there is one change I'd be willing to think more about / poll on the list:

The client initially sends a 4 octet magic that signals the encoding it wants to speak. If the router supports that encoding it'll go on and MUST speak that encoding. If the router does not support the encoding, it hard drops the TCP.

Of course you could also propose a whole new transport for WAMP that does as you like. And gather feedback on the list. No problem. Given enough support, we might add it to the AP. What I can say is that I won't implement it in Autobahn or Crossbar.io. I don't need it.

[beatgammit]

I've put up a discussion on the google group: https://groups.google.com/forum/#!topic/wampws/bSWpvAYbFDY

[oberstet]

Ah, great! I think an informal "open community" / rough consensus + working code approach is a good thing.

Rgd the concrete concern "running on 1 port": are you also concerned about running WebSocket and RawSocket on 1 port?

Might be doable also, since WebSocket starts with GET (4 octets). In principle. At least for AutobahnPython, implementation would be non-trivial.

[beatgammit]

I'm honestly more concerned about the requirement to have a port per encoding than running multiple "transports" on the same port. For example, I may have 80, 443 and 8080 at my disposal, and if I have 80 and 443 tied up by their appropriate uses, that leaves me only one port to run on. I can run ws and wss on 80 and 443 respectively, but I can't (currently) run both JSON and MsgPack RawSockets over 8080.

I think trying to run two "transports" on the same port is overly complicated. Just having a way of specifying the encoding at startup would satisfy me. I just think if this gets put in, there should be room for extensibility later should another connection-time feature be requested (e.g. encryption). WebSockets has this, but I don't want to be using WebSockets for a browserless client (i.e. arduino, desktop application). Each client has it's own limitations, so I think the spec should allow for some level of configuration.

Honestly, a magic 4-octet header would be nice, but it also might be nice for there to be a required 4 octet response to allow for extensibility later. Just like the rest of WAMP, this will allow different implementations to support different features, but at least the basic format must be specified before RawSockets is "stable".

[oberstet]

@beatgammit

running on port 8080 will limit the usefulness of RawSocket on restricted networks
because of being blocked. why not run 8080 (rawsocket-json) and 8081 (rawsocket-msgpack) then? running rawsocket on 80 also will not work in restricted networks, because intermediaries will inspect and see no HTTP. running on 443 using TLS will work in most cases - but then the question is not only how to support both json and msgpack, but also where to run WebSocket when 443 is already occupied by rawsocket.

encryption: TLS. no need for WAMP to do anything special here.

what should a 4 octet response magic signal or negotiate? this is still at WAMP transport level preflight .. no WAMP level features can be negotiated here.

[beatgammit]

running on port 8080 will limit the usefulness of RawSocket on restricted networks
because of being blocked

Sure, on some networks, but some restricted networks just block the port numbers without inspecting traffic. Obviously networks will differ, but I don't think that's particularly relevant to my main concern: future extensibility.

From what I've seen, the rest of WAMP is very flexible, except for RawSocket. I don't think that should be the case.

what should a 4 octet response magic signal or negotiate? this is still at WAMP transport level preflight .. no WAMP level features can be negotiated here.

Mostly transport-level features like serialization format, compression, encryption, etc. None of this is relevant to WAMP. I'm imagining some sort of bit-field where the client would request certain features (e.g. MsgPack encrypted with AES), and the server would respond with the subset of requested features that is supported (i.e. MsgPack, but not encrypted). If the server can't honor the request, the socket will be closed immediately (though responding with a 4-byte response is encouraged).

Enough of the 4-byte request/response should be left open for possible extension later.

This style of negotiation is very minimal, extensible, and uses feature-detection to connect to the server. Clients can be upgraded to support newer WAMP features independently of the server.

With the current specification, if my server was to stop supporting JSON and only support MsgPack, the only way to signal this would be to reject connections on a dedicated JSON port. I'd also have to open a new port for every combination of features that could be supported (e.g. an AES-encrypted MsgPack port, non-encrypted MsgPack port, etc). This gets very tedious very fast and can be easily solved via a simple negotiation.

I'm not saying that the meaning of the 4-byte packet needs to be specified immediately, just that putting that in place now will allow for extension later without having to introduce a new Transport.

[oberstet]

Flexbility comes at a price: complexity. RawSocket should be dead simple.

Regarding the concrete future extensibility features you have in mind:

• transport level encryption: I wouldn't support anything but TLS for this. And for TLS, it's a non-issue (since all the encryption related negotiation happens before the first user level octets are sent/received anyway)
• compression: a proper negotiation for compression likely will involve more than setting a bit "compression = true". Have a look a permessage-deflate. It is more involved. There is window context take over yes/no and compression window bits. Each one in both directions. And not only that: with deflate, there is no 1:1 relation between app level messages and transport level messages anymore.
• signaling of unsupported serialization: say the client only supports JSON over RawSocket and the server only MsgPack. Instead of replying with a selected serializer, the server would need to signal the error. Which immediately raised the question on how to encode errors in the proposed 4 octet reply from the server.

It all gets more complicated quite quickly.

Regarding networks compatibility: for me, this is one of the most important concerns. Many mobile and enterprise networks are very restricted.

In summary, what I'd like to see is this: being able to run WebSocket and RawSocket and Long-Poll with all possible serializers on 1 TLS server port 443.

This is a goal I can subscribe to. And I am ready for the simplest change that allows above.

The simplest change would require the client to send a 4 octet magic (just an unstructured bunch of random octets fixed in the spec) depending on the serializer it wants. Those 4 octets must not be equal to GET. When the server speaks the serializer, it continues. If not, it drops the connection. A client may then retry using a different serializer (and optionally remember the one that worked for a subsequent connection).

The concerns I have with the former are:

• the client does not get an explicit "positive" feedback from the server ("yes, I really do speak RawSocket with the serializer you requested").
• the server might drop the connection for other reasons, like max. number of connections reached.

In other words: if we specify a reply by server, I'd like to see the ability to signal "positive OK" as well as specific error reasons for rejections rather than negotiation. Those could be magic values also. No need to have structure inside the 4 octet request/reply.

Should we have an (unstructured) 4 octet magic reply, those magic values must be different from HTTP.

In a way, using magic values already provides some kind of extensiblity: e.g. new magic values for new serializers. New magic values for new error signals.

[beatgammit]

Apparently there hasn't been any discussion on the google group yet, so I'll describe my proposal here.

In summary, what I'd like to see is this: being able to run WebSocket and RawSocket and Long-Poll with all possible serializers on 1 TLS server port 443.

If this is the direction it will go, then it's best to completely avoid anything that might be mistaken for HTTP or TLS. After reviewing the relevant specs, it looks like HTTP's first byte is ASCII minus control characters (look at token), and TLS uses 0x14-0x18.

So, I propose that we do something like this:

• byte1 - magic octet > 0x7f to specify WAMP (unambiguous since neither TLS nor HTTP can use this)
• byte2 - encoding formats the client supports (e.g. json, msgpack)
• byte3 - compression formats the cilent supports (e.g. gzip, lzma)
• byte4 - encryption formats supported (e.g. TLS, twofish, etc)

The response could look something like this:

• byte1 - magic octet > 0x7f to confirm WAMP support
• byte2 - encoding format that will be used (if 0, error)
• byte3 - compression format that will be used (if 0, no compression)
• byte4 - encryption format that will be used (if 0, no encryption)

Only one bit on each of bytes 2, 3 and 4 would be set on the response. If an encryption format is accepted, the specification would dictate how that is to be set-up. Once encryption is set up, the socket is ready for WAMP.

In the case of byte2 being an error, bytes 3 and 4 can be used for signifying what the error is. For future-proofing this, the MSB of byte2 will be reserved for future use (set to 0 always until the meaning of setting to 1 is defined).

I think this gets us both what we want:

• you can use TLS first to initiate the socket (in which case byte4 would be 0)
  • if TLS is broken, you can use another encryption format
• if you want a WAMP-layer compression instead of a transport-layer compression, set byte3 to 0
  • if more information about windowing is necessary, that can be defined per compression format; set the appropriate bit to signal that more information is coming
  • to mark RawSocket as "stable", no compression formats need to be supported (though AFAIK most compression options are part of the compression format, and the server can decide what those values are)
• unhandled serialization is easy, just set byte2 to 0 (or define the MSB on byte2 to be an error code, and the server would reply with what it can support)

Would this style of "negotiation" work for what you'd like to use it for?

Note: I'm mostly just concerned about serialization formats, but since there's available space, I felt these are reasonable things that could be specified at the transport level.

[oberstet]

@beatgammit we're getting closer;)

rgd. "it's best to completely avoid anything that might be mistaken for HTTP": good idea! the first octet should be fixed to avoid that altogether.

rgd. TLS:

• probably we are talking a little bit past each other here (is that correct english? anyway): from WAMP transport point of view, TLS is invisible. In fact, if you use TLS via e.g. OpenSSL, the first octet received or sent by the application is not anything TLS. The TLS handshaking already happened before, and the app receives/sends the clear-text stream.
• as mentioned, I think it is a very bad idea to invent a new transport-level encryption scheme. TLS has issues, but anything new we would invent would be very likely even worse.

In summary, you convinced me of the following - and it's my understanding that we two reached consensus on this:

1. It should be possible to run RawSocket with JSON, MsgPack and potentially others on one port
2. There should be a 4 octet value sent by client to router first
3. The 4 octets sent should indicate the serializer the client wants to speak
4. The router should reply with a 4 octet value

plus

1. It should be possible to run RawSocket, WebSocket and Long-poll on one port

In particular the latter is a real value-added proposition. I think this is important, since implementers that already have RawSocket will have to change their implementations.

Where we don't yet have consensus is the following.

Encryption

• You: different transport level encryption schemes should be possible
• Me: if transport level encryption then TLS

I am fixed on this. I don't think there is an argument that would convince me (see above).

Compression

Prenote: doing compression at the underyling transport level: I guess you are talking about TLS compression here? If so: please not. This can be a severe security issue (side channel attacks). If you can somehow trick the peer into sending/receiving known plaintext, you can measure the compressed length from outside and draw conclusions about the session key in use. If we are talking about TCP: there is no compression avail at that level.

Then, from my experience implementing permessage-deflate, this is trickier than it seems on first sight.

On the other hand, having compression on RawSocket: that would be an additional value proposition. If it is worth the complexity added is another question. But on this (and only this) point I'd be ready to continue a discussion.

Here is my KISS proposal:

• Client-to-Router (wants to speak JSON): 0x 7F E1 03 1A
• Client-to-Router (wants to speak MsgPack): 0x 7F 4D 94 87

and replies

• Router-to-Client (OK with speaking requested serializer): 0x 7F AF 41 6E
• Router-to-Client (Serializer not supported): 0x 7F F1 CA 2D
• Router-to-Client (Connection limit reached): 0x 7F 75 CC 57
• possibly more

That is all unstructured magic values (I just used some random stuff), but first octet fixed.

It is extensible in two dimensions:

• new serializers
• new errors

It does not allow compression. It does not provide extensibility for yet unknown dimensions. However: YAGNI! ;)

[beatgammit]

• You: different transport level encryption schemes should be possible
• Me: if transport level encryption then TLS

I am fixed on this. I don't think there is an argument that would convince me (see above).

Here are a couple reasons why you may not want TLS but still need encryption (I'm sure there are others):

• TLS implementation bugs (Heartbleed, goto fail)
  • disable TLS while the patch is applied; clients fallback to another encryption
  • this can be faster than patching the server in some situations (e.g. game server where downtime is worse than security implications)
• TLS broken - this has happened with other algorithms, so it's not unthinkable that it could happen again

I understand that TLS is basically another transport since it's basically a black-box (data goes in, same data comes out), but there have been very real problems with popular TLS implementations. When it comes to security, there should always be multiple options in case one is compromised.

Prenote: doing compression at the underyling transport level: I guess you are talking about TLS compression here? If so: please not... If we are talking about TCP: there is no compression avail at that level.

This is exactly what I'm talking about and why I think it should be part of the RawTransport spec. If a transport meets the preconditions (message-based, bidirectional, etc), then it can be used for RawTransport and features like encryption and compression can be done on top of that, just like TLS does.

It is extensible in two dimensions:

...

It does not allow compression. It does not provide extensibility for yet unknown dimensions. However: YAGNI! ;)

And mine is extensible in 4:

• new serializers (up to 7 total, could be modified to allow for more if this is a potential issue)
• new errors
• new encryption (requires additional negotiation phase, just like TLS if done after the 4-octet packet)
• compression (won't require negotiation phase for most compression algorithms)

It's also KISS since an implementation that only does MsgPack and JSON only needs to check two "magic" values as well.

Just reserving the last two bytes on request and response (in non-error case) is enough to allow the above to be added later.

If you're really adamant about your proposal, I'm okay with that and I consider it a net win from the current spec, and exactly what I originally hoped to achieve.

[oberstet]

@beatgammit To make compression effective, it needs to cross WAMP message boundaries. Compressing each WAMP message individually will be of very, very limited gain. I have measured this. If you turn off "compression context takeover" on permessage-deflate, compression ratios drop off sharply up to the point where it gets worse than not compressing at all. Compression (as transport encryption) should be below the message based transport that WAMP assumes.

Compression does need negotiation in practice, since there is a tradeoff between compression effectivness and memory consumption (among other costs like CPU cycles of course). With permessage-deflate and the maximum compression window size, there is several 100KB additional memory footprint per connection on a server. Please see https://github.com/crossbario/crossbar/wiki/WebSocket%20Compression#production-settings and the link to "expert advice" there.

TLS itself or TLS implementation being broken: sure. The solution is to update a system regularily. The solution is not to reinvent TLS or write a transport level encryption from scratch. You likely will get it wrong.

"adamant": I look at that last proposal not as "being mine", but as the culmination of a fruitful discussion, that has elements brought in by both of us, and which seems to solve the original itch you felt. It's just that I need to be able to defend it publicly .. means I need to be convinced of it .. since implementers might (rightfully) object that we are adding and changing stuff all the time;)

I honestly see it as the simplest change that fits the bill. With a clear value prop: able to run all transports mentioned on 1 port.

If you agree, could you take the proposal to the WAMP list? Shouting for a consensus call with implementers. Maybe some guys wake up .. =)

Another note: if you care about TLS-independent encryption and/or end-to-end encryption, there is #81 . I'd highly welcome if you would take part in that (should you care) - you've been taking part and helping improving WAMP for so long ..

[beatgammit]

if you care about TLS-independent encryption and/or end-to-end encryption, there is #81

I think this may alleviate my main concerns. The question of compression (and batching) seems like it should be a separate thing altogether since there's no gain according to your tests.

I still think it's wise to leave the last two bytes untouched just in case a valid use for them is thought up later (e.g. automatic connection re-establishment, message batching, max message length, etc).

That being said, I'm satisfied for now with your proposal and I'll try the mailing list again to see if anyone else has any other input.

[oberstet]

The "batched modes" of WAMP transports: I came up with this as a conclusion from this: http://tavendo.com/blog/post/dissecting-websocket-overhead/

I don't have a blog post on permessage-deflate, but with WAMP JSON payloads and "context takeover" (which means, the compression dictionary used is reused across WAMP messages) and a "window size" of >12 bits (which means a sliding window of 2^12 octets is compressed) the compression ratios I saw was 2-15. If you turn off "context takeover" (which means each WAMP message is compressed on its own .. no compression dictionary reuse), the ratio completely collapses. If you ZIP a single 30 octet message like [7, "com.mypapp.foo", [1, 2]] (not a real WAMP message, but you know what I mean), there isn't much "repeating" and hence no compression. If the same message, or slightly differing messages are compressed together, e.g. repeating URIs will get crunched.

[oberstet]

@beatgammit One more thing: besides the "one port" value prop. you mention "microcontrollers" on the mailing list. This is interesting. Can we bring WAMP to a 8KB RAM Arduino?

RawSocket is much easier to implement than WebSocket, and can be implemented efficiently (length prefix, no fragmentation). There is one concern: memory. A hard length limit on the maximum serialized message length. That would allow use of a statically allocated memory buffer.

I'd be interested in continuing a discussion on this. There are 3 concerns:

1. max. serialized message length (to allow static allocation of sending/receiving buffer)
2. a serialization format that works "in-place" (in-memory without actually serializing), but nevertheless is "self-describing" (no static typed, no IDL compiled whatever) - needed since WAMP is dynamically typed
3. WAMP can have arbitrary number of RPCs in flights (and similar for other WAMP requests)

My current view:

1. should be handled at transport level. if so, we should weave that into RawSocket somehow now.
2. we can come up with something later. RawSocket will support pluggable serializers ...
3. ??? - if handled/negotiated, should probably be done in WAMP session opening handshake

What do you think?

[oberstet]

rgd 2), here: https://news.ycombinator.com/item?id=7901991

and: https://kentonv.github.io/capnproto/news/2014-06-17-capnproto-flatbuffers-sbe.html

[beatgammit]

Can we bring WAMP to a 8KB RAM Arduino?

I don't see why not. At one time I had a project on Arduino that used WebSocket framing and JSON to control a servo. It didn't do HTTP and the full WebSocket negotiation because I didn't want that overhead. It wasn't real-time or anything, but I think WAMP is light enough that it would work quite well, especially with a faster serializer.

I'm sure a C++ WAMP client could be easily adapted for the purpose.

My current view:
...
2. we can come up with something later. RawSocket will support pluggable serializers ...

I think this is the way to go. Keep RawSocket a general transport and make a more custom transport if necessary. I think RawSocket will be more than sufficient for my projects and I can use the same implementation for desktop applications and embedded.

[jcelliott]

I think this is a good improvement to the RawSocket proposal. I would argue for reserving the last two bytes until we have a concrete use for them though. One byte should be enough to specify the serialization, and if we add anything else to the four byte message later this will keep things simpler (by not having to create magic 3-byte numbers for every combination of features we want to specify)

There are two possible formats for this:

• A single magic number representing each serialization, with 255 possible. Clients would specify a serialization and the server would either accept the connection with that serialization, or reject the connection.
• An 8-bit field, with each bit specifying a possible serialization. This would allow the client to specify all the serializations it would accept in a single connection attempt rather than try and fail several times until the server accepted one. The server would respond with the serialization to be used.

Both methods are simple to implement, but the second one would have less overhead for establishing a connection (where RawSocket is meant for low overhead). I think either method would be acceptable though.

[oberstet]

@beatgammit rgd AutobahnCpp: that will be very hard to bring to an Arduino. one reason is the use of C++11. I'd probably rather write a AutobahnC thing that uses pure C89 for maximum reach to MCUs.

@jcelliott 8 bit field for serializations: yes, the client can signals all serializers it supports, but it can't prioritize which one to speak. how would the server select if it supports all of the client one's? WAMP-WebSocket has full priorization/negotiation support on this - but we should not start reinvent WebSocket .. as that would render RawSocket pointless: we should keep it really simple IMO

Ok. You both seem to lean towards having "structure inside the 4 octets". How about this:

Client-to-Router request (first 4 octets sent):

| 0x7F | SS | LL | 0x00 |

• The first octet is the magic 0x7F
• SS is the serializer requested: 1 = JSON, 2 = MsgPack, etc. 0 is NOT allowed.
• LL signals the maximum serialized message length 2^L the client is willing to receive. A value of 0 signals: no limit.
• the trailing octet is reserved and must be set to 0x00 for now.

Router-to-Client response (first 4 octets sent):

| 0x7F | SS | LL or EE | 0x00 |

• The first octet is the magic 0x7F
• SS is the serializer accepted: 1 = JSON, 2 = MsgPack, etc. 0 signals denial of connection, and then the 3rd octet EE provides the error
• LL signals the maximum serialized message length 2^L the router is willing to receive. A value of 0 signals: no limit.
• EE signals an error (connection denied)
• the trailing octet is reserved and must be set to 0x00 for now.

Example:

• Client-to-Router: 0x 7F 02 08 00 - client wants to speak MsgPack and can receive max. 256 length messages.
• Router-to-Client: 0x 7F 02 00 00 - router is OK speaking MsgPack and can receive messages of arbitrary length
• Router-to-Client: 0x 7F 02 10 00 - router is OK speaking MsgPack and can receive messages of max. length 65536
• Router-to-Client: 0x 7F 00 01 00 - router denies connection: unsupported serializer
• Router-to-Client: 0x 7F 00 02 00 - router denies connection: max connection count reached

[oberstet]

Besides above change (whatever version we reach consensus on), I'd like to discuss also the option of adding ping/pong to RawSocket. Related to #90

In a way, ping/pong for connection loss detection and keep alive is really a transport level feature. I somehow feel uneasy adding this at the WAMP level.

Here is one option to add ping/pong for RawSocket:

Now: A message is serialized, and prefixed with the length of the serialized message as an unsigned integer (4 octets).

Proposal: The upper (MSB) 2 bits of the unsigned integer are taken out as an opcode:

• 0: regular WAMP message (as today)
• 1: PING request
• 2: PONG reply
• 3: reserved

In case of PING or PONG, the length (after taking out the opcode) provides the length of the subsequently following byte string which is to be echo'ed back exactly.

With the proposal, a serialized WAMP message can have at most 2^30 = 1GB length. I still think this is total overkill in practice - more than enough.

[beatgammit]

rgd AutobahnCpp: that will be very hard to bring to an Arduino. one reason is the use of C++11. I'd probably rather write a AutobahnC thing that uses pure C89 for maximum reach to MCUs.

Yeah, that could cause problems. And I think Arduino's C++ is stripped-down anyway. I honestly much prefer C to C++ anyway, especially since it's easier to write bindings to in other languages.

Ok. You both seem to lean towards having "structure inside the 4 octets". How about this:

That sounds great!

The more I think about it, the less I like the bitfield idea. A client can try with the preferred encoding, and if it fails, try the next. For short-lived clients (e.g. micro-controller that can't sustain connections), the accepted encoding can be stored in memory for the next connection attempt.

Proposal: The upper (MSB) 2 bits of the unsigned integer are taken out as an opcode:

I guess this depends on how much WebSocket functionality RawSocket should have. Do you think there's any sense in doing interleaving partial messages? If so, the whole first byte could be reserved. This would still allow for 16M messages (just so happens to be max size for MongoDB/BSON documents), which should be more than sufficient for WAMP. Large messages could be broken up into smaller chunks just like in WebSocket transport.

I don't personally have a need for documents > 2^16, but if larger documents were allowed at the transport level without blocking other messages, perhaps data transfer could be done over WAMP as well.

Perhaps this is getting too complicated though, since RawTransport is meant to be simple.

[Matthias247]

Hi,

I just stumbled over this while trying on collecting some more info on WAMP.
Here's my opinion:

Serialization negotiation
I think that's an absolutely necessary and good thing, because having Raw Socket limited to MessagePack seemed a little bit unfortunate. I also think it should work in the same way as with websockets: The client sends a list of supported serialization methods he understands and the server selects one - or none and quits. However I think using a single byte as bitfields with only 8 options seems not verfy far sighted. I would send a header-prefixed byte-list to the server, which gives up to 255 possible formats while still providing flexibility at the server.
E.g. 0x02 0x01 0x03 means that the clients wants to speak either serialization format 0x01 or 0x03.

Yes, that would make the initial handshake also variable length and probably 2 asynchronous reads. But I don't think that's a big problem. As the connection is anyways long-running and connection establishment is only a small part of it I don't see the need why these packets must be limited to 4bytes.
Either you totally leave the handshake out (which reduces the transport implementation significantly) or you can also go for a longer length.

Actual message header
The header should really stay at fixed 32bit. That causes by far the least headaches. Anything significantly smaller is not practial and the overhead is neglible (you already have lots of overhead through TCP, the inner serialization, etc.).

Pings/Pongs
On the hand I think it's useful to be able to handle this at connection level and that using 2bits from the header for it still leaves a more than sensible 1GB message size. But on the other hand it makes the implementation of the raw transport much more complicated.

Transmitting the maximal message length
If the handshake packets can be longer than 4 byte (see above) you can also send the maximum length directly in bytes.
But anyways, if you define such a value you would also need to the define how the clients and servers use it. The only sensible default that comes to my mind is that any participant automatically closes the connection if it determines that the remote can't handle the payload. Sending messages below the max. size and dropping those bigger than the max. size will only cause problems (e.g. you would receive some pub/sub events and some other ones not, which would get you an imcomplete picture). But that would not add much benefit versus letting the remote closing the connection if encounters a message header that gives a size that it cannot process.
So you avoid transmission of a (partial) message, but buy this with the additional complexity of transmitting and handling all max. message sizes.
If you leave the decision what to do with too-big-messages to the receiver it could be still up to the receiver what to do with them (e.g. skipping them without closing the connection for a memory-constrained microcontroller application). But you can only do this when you are exactly aware of the application that is running on top of WAMP and how that behaves regarding dropped messages.
Hmm, if you really want that you could also exchange an additional field besides the max. size that tells the remote what to do with the message if it exceeds the max. size (drop message, close connection, sth. else). But that's putting application level knowledge in the transport layer and further complicates the options.

[oberstet]

@beatgammit

Do you think there's any sense in doing interleaving partial messages?

No. This would make RawSocket more complicated than WebSocket. For WebSocket, multiplexing or priorization are not part of RFC6455. There are multiple, unfinished proposals to add that as a WebSocket extension. E.g. here is mine: https://github.com/oberstet/permessage-priority/blob/master/draft-oberstein-hybi-permessage-priority.txt

Anything in this direction will likely make it impossible to run RawSocket from statically allocated memory - which is what I am after on MCUs. I'd like to be able to do the following in C on a MCU:

#define MAXLEN = 256;
char buffer_in[MAXLEN];
char buffer_out[MAXLEN];

and use buffer_in and buffer_out for all WAMP communication. Ideally, I'd like to use a serializer that does not even serialize, but uses those buffers "in place". But that's another issue ..

@Matthias247

I think that's an absolutely necessary and good thing, because having Raw Socket limited to MessagePack seemed a little bit unfortunate.

You can run RawSocket with different serializers today. It's just not possible to run all serializers on 1 server port. Which is the itch that we want to address.

I also think it should work in the same way as with websockets

Adding full negotiation makes it too complex. Which is the whole point of RawSocket: make it dead simple. If we drop that goal, it's pointless altogether, since then we can simply use WebSocket.

Ping/Pongs ..But on the other hand it makes the implementation of the raw transport much more complicated.

Why? Mask out the top 2 bits to get the length. If the opcode == 0, process as today. If opcode == 1, it's a ping - echo back. If opcode == 2, it's a pong.

If the handshake packets can be longer than 4 byte

No. RawSocket is so much simpler than WebSocket because you always exactly know how many octets you must receive before you can go on processing.

Transmitting the maximal message length

The maximum length a peer is able to receive is communicated in the handshake. It's just a power of 2, instead of an arbitrary integer value. If the peer send a longer message, the receiver will hard drop the connection. That is a transport level violation. Same goes when serialized value is invalid. Hard drop. A non-conforming peer is either a buggy implementation or an attacker. In both cases, the sane way is to hard drop, not to handle it (which might introduce even more problems).

[Matthias247]

You can run RawSocket with different serializers today. It's just not possible to run all serializers on 1 server port. Which is the itch that we want to address.

Ok, I didn't recognized that, thought RawSocket was MsgPack only (because of AutobahnCpp). So the problem is that currently you need to know on client-side which serialization the remote uses? That's no big problem. And binding to 2 ports on server-side isn't a big problem either.

Why? Mask out the top 2 bits to get the length. If the opcode == 0, process as today. If opcode == 1, it's a ping - echo back. If opcode == 2, it's a pong.

Yes - extracting the opcode is easy. But to handle it properly you need a statemachine in the Transport. If you received a ping you might not immediatly need to send a PONG when you already have one queued. And when you are currently sending something else you can't immediatly send the reponse. In my experience from implementing websockets such things also introduce lot's of coupling between sending and receiving, which is complicated if you do that in multiple threads.

No. RawSocket is so much simpler than WebSocket because you always exactly know how many octets you must receive before you can go on processing.

I would still know that. There could be a fixed length part of the header which indicates how long the optional variable-length part is. You could even make it completly the same as the remaining message by making the handshake also a [4byte-lenth handshake-content] messsage just like all other raw-transport messages.

The maximum length a peer is able to receive is communicated in the handshake. It's just a power of 2, instead of an arbitrary integer value. If the peer send a longer message, the receiver will hard drop the connection. That is a transport level violation. Same goes when serialized value is invalid. Hard drop. A non-conforming peer is either a buggy implementation or an attacker. In both cases, the sane way is to hard drop, not to handle it (which might introduce even more problems).

My question was that if the receiver will hard-drop the connection on too-big-messages what is the advantage of telling the length to the transmitter? The receiver can do this also without exchanging the maximum length.

[oberstet]

So the problem is that currently you need to know on client-side which serialization the remote uses?

Yes, exactly. Both need to have implicit knowledge about which serializer to use. AutobahnCpp only implements MsgPack (currently) mainly because I hadn't time to do JSON as well.

If you received a ping you might not immediatly need to send a PONG when you already have one queued.

That is how WebSocket does it (in fact, WebSocket doesn't require to respond at all. The spec says: "respond as soon as practical" - which might be never). We should do better and simpler with RawSocket: each and every ping must be replied with exactly one pong, and unsolicited pongs are forbidden. Then you don't need a state machine.

do that in multiple threads

Don't use threads;) Ok, sometimes it's required, e.g. AutobahnAndroid uses background reader and writer threads. But it's a non-issue there, since it employs queues: the receiving leg that gets pinged can just enqueue the pong onto the sending leg.

With RawSocket, my main two use cases are:

• as a transport for components running over local transports (like loopback TCP and Unix domain sockets)
• as a transport for tiny devices (MCUs) where RAM is as low as 8KB

how long the optional variable-length part is

Why would we need a variable length header?

what is the advantage of telling the length to the transmitter

The transmitter e.g. a Router could things like, e.g. for PubSub

• drop the event (not forwarding to client) and track dropped events
• prohibit publishing to the topic already
• remove the payload, and send an event with extra information ("payload_limit_exceeded = true")

The receiver can do this also without exchanging the maximum length.

Yes, that's possible today. E.g. for production use, Crossbar can be configured to hard-drop WebSocket upon exceeding a configured message (or WebSocket fragment) length. DoS protection.

[beatgammit]

Do you think there's any sense in doing interleaving partial messages?

No. This would make RawSocket more complicated than WebSocket.

That makes sense. I don't think I'll need it (main use-case was streaming large binary assets without blocking other messages) because I can just use multiple connections, just wanted to know your thoughts about it.

I'm on board with the current proposal then.

each and every ping must be replied with exactly one pong, and unsolicited pongs are forbidden

👍

The client sends a list of supported serialization methods he understands and the server selects one - or none and quits.

You can get the same effect by reconnecting with different requested encodings until one is accepted. You get the same effect without requiring additional complexity, and you only sacrifice a few reconnects (once; the accepted encoding can be cached if reconnects are expensive).

My question was that if the receiver will hard-drop the connection on too-big-messages what is the advantage of telling the length to the transmitter? The receiver can do this also without exchanging the maximum length.

The transmitter e.g. a Router could things like, e.g. for PubSub
...

Also, some non-essential fields could be dropped for clients that can't receive the whole payload. This information isn't currently available, but I can see that being specified later.

[oberstet]

I have written up the proposal on this branch: https://github.com/tavendo/WAMP/blob/rawsocket/spec/advanced.md#rawsocket-transport

This is very close to what we have discussed above, but allowing for more extensibility. 2 octets in both the handshake request and reply are reserved. The opcode in messages goes into the upper octet of the message prefix. That leaves 16M for a message maximum (which I agree with @beatgammit is already more than enough/sane in practice anyway). The max msg. length now MUST be announced by both peers (can be between 512 - 16M).

I announce that on the list. To all interested: please review if you have time. I'd like to merge to trunk, mark the feature as stable, and then update AutobahnPython and AutobahnCpp.

[Matthias247]

You are missing 1byte in both the request and response for the handshake.
I'm not a big fan of reconnecting until it works because it actually has a much bigger overhead than sending one time list of requested protocols.

Regarding ping/pong: I just saw in the advanced specification that also heartbeat messages are supported. Wouldn't that make the a ping/pong superfluos (or the other way around)? As long as something is regularly sent the TCP stack itself will care about the connection beeing open or report errors when data (or an ACK) could not be transmitted.

[oberstet]

@Matthias247 missing reserved octet: ah, thanks! fixed.

rgd reconnecting for finding a serializer the router speaks: in practice, most routers might simply speak all serializers. whereas a client might only speak one often. and even if the former doesn't hold: a client can remember what serializer worked. chaning this would require a handshake request of variable size (if priorization should be taken into account).

rgd "hearbeat": yes. in fact, heartbeat was mixing 2 features, one of them being ping/pong like. we will remove heartbeat completely (for now). see my mail on the WAMP list.

[beatgammit]

I'm not a big fan of reconnecting until it works because it actually has a much bigger overhead than sending one time list of requested protocols.

a client can remember what serializer worked. chaning this would require a handshake request of variable size

And if that's not possible (e.g. microcontroller w/o writable persistent state), msgpack is in the basic profile so it most be supported by a WAMP-compliant router, so it's probably the one that'd be used. A microcontroller like this would probably only be able to support one serialization anyway, so it doesn't really matter.

Other clients would probably be able to remember some kind of state, so this is only matters for the initial connection.

@Matthias247 - Can you think of a use-case where this would not be acceptable?

[oberstet]

this is in the spec for some time now

[konsultaner]

@oberstet I just saw this on Stackoverflow.

By having the browser generate a random mask for each frame, the hostile client code cannot choose the byte patterns that appear on the wire and use that to attack vulnerable network infrastructure.

Shouldn't have RawSockets a mask as well?

[oberstet]

no, because a) browser cannot talk rawsocket, and b) everything should run over TLS anyways