ActionCable protocol negotiation

[danielrhodes]

A first stab at negotiating an acceptable protocol as was mentioned in #23976 @maclover7 @jeremy @javan

Going forward, if the ActionCable protocol changes, there ought to be some way for the client/server to negotiate this change and gracefully stop communicating if no common protocol can be established.

Using the Sec-Websocket-Protocol from the WebSocket spec, the client and server negotiate a protocol from a list. Most of the work is being done by the WebSocket driver and the web browser WebSocket client, so all this really does is make sure the client does not get into a bad state if the server has been upgraded and the client has not.

The browser websocket client automatically chooses to disconnect if the server does not respond with an acceptable protocol (this is not configurable), so instead of trying to reconnect (as this would not be useful), the client shuts down the monitor.

[rails-bot]

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @arthurnn (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[javan]

What browsers have you tested this in? I'm curious if protocols and onclose event codes are broadly supported. Unclear from documentation I've found online: https://developer.mozilla.org/en-US/docs/Web/API/WebSocket#Browser_compatibility and https://developer.mozilla.org/en-US/docs/Web/API/CloseEvent#Browser_compatibility

[javan]

return is implicit in CoffeeScript, you can cut it.

[javan]

More CoffeeScripty:

if event.code is 1006
  ActionCable.log("WebSocket received abnormal close error (#{event.code}):", event.reason)

[javan]

Also, why an additional log for 1006 only? Might as well log event.code, event.reason, and event.wasClean with every close.

[javan]

In CoffeeScript you can do unless event.wasClean (or if not event.wasClean).

[jeremy]

Cool! TIL that Sec-Websocket-Protocol is for subprotocol negotiation.

[jeremy]

Can we bubble this up to the app in some way so it can react in the UI, e.g. show a prompt to reload the page or auto-reload it?

[danielrhodes]

Yeah we need a way of telling the app why there was a hard disconnect. To be discussed, as there are other cases where we need this as well (e.g. authentication failure).

[danielrhodes]

@javan Ok I tested it on Chrome, Safari, Firefox, and Mobile Safari. I don't have IE, but it does have the CloseEvent. Mobile Safari was the only one which behaved differently from the others (it did not automatically disconnect when the negotiation failed). Therefore, I changed the code to check the protocol negotiated to see if it is in the list the client knows it can accept and if not, to disconnect. I likewise took out that close event code stuff and checked the protocols as a way to decide if the monitor should be stopped. I think that should resolve your comments.

[jeremy]

How about e.g. actioncable-v17-json? Tack on the transport encoding (which is always JSON now) and use a big chunky version that looks nothing like the AC version number since the only purpose is to break major compat (there are no "minor" changes).

Then the server can negotiate by parsing out the requested version and transfer encoding and picking the matching protocol handler for the conn.

[matthewd]

Do we only want to version the ACa transport version, or also provide for an application-level value? I'm thinking of something equivalent to config.assets.version...

[danielrhodes]

@jeremy This is a good idea.

[danielrhodes]

@matthewd This kind of gets to a bigger question of how much backwards-compatibility we build into this.

If somebody wants to make their ActionCable server a publicly available resource, then backwards compatibility becomes important. Likewise, I made a Swift client for ActionCable and a changing protocol version becomes a real pain for the developer because the longer upgrade cycle becomes a chicken-and-egg problem.

On the other hand, if this is really only used so a person can communicate with their own backend in the browser, there doesn't need to be a lot of backwards compatibility built-in. Only enough so that the client can gracefully degrade until the next reload.

In either case, there is no benefit for the developer in being able to pin a version because if the server is able to support a version, it should because it's an internal protocol.

I'm interested to see what feedback we get about this.

[jeremy]

App-level protocol versioning is interesting. It could be helpful, at minimum, to provide a big hammer so apps can wiggle out of a stale-client compatibility mess by bumping the protocol.

For other cases, like changing channel behavior, I'd lean to leaving it up to apps to make changes in a compatible way. But, heck, leaning on bumping the app proto version could be simpler in many of these cases too.

[jeremy]

Good one to pass as a keyword arg, no need to keep @options around

[jeremy]

def initialize(env, options, event_target, event_loop, protocols: ActionCable::PROTOCOLS)

[jeremy]

Or glob the keyword args as websocket options:

def initialize(env, options, event_target, event_loop, **websocket_options)

@driver = ::WebSocket::Driver.rack(self, @websocket_options)

[danielrhodes]

Would be happy to do this as it is cleaner. I haven't seen much use of the Ruby keyword arguments in the Rails codebase, but I guess it's ok now?

UPDATE: Seems Ruby 2+ is now required for Rails 5, so I'll do the keyword args.

[jeremy]

@driver = ::WebSocket::Driver.rack(self, protocols: protocols)

[danielrhodes]

In the comments @jeremy pointed out that it would be helpful if there was some way the client could communicate to the app a reason for the disconnect. There isn't currently a global way of doing this, so any opinions on this? We can pass some sort of reason object through to the disconnect method like WebSocket does.

[jeremy]

Ditto

[jeremy]

Do we need the || nil? Would only kick in if protocols was oddly false

[jeremy]

Ditto

[javan]

This will prevent the connection monitor from infinitely attempting to reconnect (as noted in #23976 (comment)), but leaves you disconnected until you reload your browser. Think we should notify subscriptions of this state so they can manage / instruct reloading.

[danielrhodes]

Yes, @jeremy wants something along these lines. I guess just a simple object with code and a reason passed into the disconnect method?

[javan]

Yeah, I think that'd work fine. Would be cool to hot-reload the client-side js, but that's probably more trouble than it's worth for what will be an infrequent event.

[danielrhodes]

I guess if you could plug into Sprockets somehow, you could pull a new version of the classes (perhaps namespaced differently for each asset build) and then hot swap in the ActionCable classes and reconnect. It would be super cool, but yes a lot of trouble.

As a side note: Facebook was doing something similar on Android for awhile, before Google made them stop by modifying the TOS for the Google Play Store.

[javan]

How about adding :protocols to INTERNAL instead? It's a general purpose data bag.

[javan]

I'd extract this to a new method since it's duplicated

[maclover7]

Do we want this to have the same version number as the gem? Would prevent having two different version numbers for Action Cable...

[jeremy]

The protocol version is distinct from the library version and will be stable over longer periods. Using separate versions is intentional.

[jeremy]

Missed this line? options gone now

[danielrhodes]

Oops :-)

[javan]

For debugging, it would be useful to log the client and server protocol versions rather than say they're not common.

[danielrhodes]

The server does not tell the client which subprotocols it supports. It simply chooses one or none, and then responds with that protocol in the header.

[javan]

I mean add @webSocket.protocol and ActionCable.PROTOCOLS to the log message

[danielrhodes]

What's the status of this? Are we good to merge? I think after the reviews, the footprint of the change in terms of behavior/code is now quite low.

[maclover7]

@danielrhodes I think this is coming along nicely -- can you try to squash down some of your commits?

[danielrhodes]

Yes will do that now. Github actually just added a feature to do this automatically on merge -- something I'm sure you will appreciate!

[javan]

Apologies for asking again since I think you explained this already, but it's not clear to me what the purpose of actioncable-unsupported protocol is. Perhaps it could be defined as a separate constant that gets included in this array so it can be documented separately.

Additionally, now that we have a v1 protocol in place, what kinds of changes warrant bumping the version? (I know the answer, but think it's worth documenting).

[maclover7]

Sorry if I've already asked this -- is there any downside to having the protocol version match the current Action Cable version? Any breaking changes with Action Cable's websockets communication infrastructure should theoretically also be a regular-Action Cable breaking change. Having two version numbers seems a bit much...

[danielrhodes]

@javan Yeah the reason for this is that the browser side WebSocket client sticks with the bizarre recommended implementation where the client does not tell us why it disconnected, when it could either be a bad disconnect or a protocol mismatch. So the actioncable-unspported protocol is the way to get around this: it will always be the last to be chosen, and if it is, we know for sure there is a mismatch. The alternatives to this are really hacky.

@maclover7 I initially asked myself the same thing. However, the protocol version is distinct from the Rails version, and ideally it would be changed quite a bit less often than the Rails version. Also since these protocol changes are almost always breaking changes, semantic versioning is not quite appropriate here.

[javan]

protocols (Optional)
Either a single protocol string or an array of protocol strings. These strings are used to indicate sub-protocols, so that a single server can implement multiple WebSocket sub-protocols (for example, you might want one server to be able to handle different types of interactions depending on the specified protocol). If you don't specify a protocol string, an empty string is assumed.
-- https://developer.mozilla.org/en-US/docs/Web/API/WebSocket

Guess I'm still not clear why the client needs know about actioncable-unsupported. What would happen if we only pass the supported protocol and let the server disconnect if that protocol isn't specified?

[javan]

Sorry, that's not right, I know we're not disconnecting server side on protocol mismatch. But can the server set the protocol to actioncable-unsupported if the client doesn't open the WebSocket with a supported protocol?

For example, client connects with:

@webSocket = new WebSocket(@consumer.url, "actioncable-v1-json")

And if the server only knows about actioncable-v2-json then it responds by setting the protocol to actioncable-unsupported.

[danielrhodes]

The short answer is no. The server must choose one or none of the protocols the client presents to it. We provide a list of protocols to the Faye-Websocket driver, and we likewise give a list of supported protocols to the browser websocket client. If the server doesn't support any of the protocols, the "correct" way to handle this is to return nothing -- except we get penalized for doing this on the client side because of the implementation recommendations... hence this unsupported stuff.

The longer answer is that yes we could fork the Faye-Websocket driver and do this... but my experience with the browser websocket client has been that it is extremely stubborn, so it would probably not like this and throw an error if it received a protocol it did not send.

[javan]

Gotcha. Thanks for explaining!

[javan]

Move away from dispatching an event here. It's a plain object now.

[danielrhodes]

Are you saying name it something different? Any suggestions?

[javan]

Yeah, maybe document it as disconnected: ({ willAttemptReconnect: boolean }) ->?

[javan]

I just don't want to suggest that it's an Event instance.

[danielrhodes]

Yeah makes sense. I just changed it to that.

[javan]

Could do @webSocket?.close() if @isActive() and drop the isClosed method entirely. Side note: we intentionally named isActive to not match a connection state and isClosed doesn't follow that pattern.

[danielrhodes]

Yep don't see why not. Changed.

[javan]

Tracking @disconnected was originally in place because to prevent calling the disconnect method twice since both the error and close handlers called it. Now that disconnect has been removed, I think we can rely on the connection state alone and remove @disconnected.

[danielrhodes]

I think that should stay for now. It still seems to prevent some conditions where onClose is called by the websocket even though the websocket is already disconnected, which I think would be confusing to the developer. Personally I like those callbacks to be very definitive: if disconnected is called, it means it really just disconnected. So we shouldn't be making lots of callbacks while we are trying to reestablish a connection in the background.

[javan]

👍 if that's the case. Strange that close would be dispatched twice though. Can you reproduce?

[danielrhodes]

Oh yeah for sure. On every unsuccessful connection this is how the browser says a connection did not happen. I guess it would have made more sense for them to add an event to onError, but what can you do?

[javan]

How about splitting the protocols here into usefully named vars:

[supportedProtocols..., unsupportedProtocol] = protocols

[javan]

Using the suggested changes above, this would change to:

@getProtocol() in supportedProtocols

[danielrhodes]

I had a back and forth with @kaspth about this... he wanted to minimize the heavy separation I made in the code between the supported and unsupported protocols. However, I don't think your suggestion conflicts with what he was looking for, so I made the change.

[javan]

Is there a scenario where messages are received after calling webSocket.close()?

[danielrhodes]

Yes the server fires out the welcome message pretty fast after connecting. The server doesn't decide to disconnect on a protocol mismatch, the client does.

[javan]

Can we prevent the server from sending messages on protocol mismatch?

[danielrhodes]

Yes we could. I ran through some scenarios where we could just tell the server to stop communicating in these cases, but it gets kind of murky and the websocket protocol seems to indicate the onus is on the client to disconnect when it doesn't like the protocol not the server. Also if somebody writes a client in something other than in the browser (that would be me :-P), they might not want that behavior. Specifically, the problem is that when a client is deployed in a mobile app, you want the client to be pretty flexible if the protocol needs to change -- so the server going dark is like the worst thing that could happen!

[javan]

Cool. Mostly asking out of curiosity. Adding this guard makes sense.

[javan]

Isn't checking @getProtocol() in supportedProtocols enough? If the protocol is in that array then it's not unsupportedProtocol.

[danielrhodes]

Sure. It's just extra safety I guess.

[javan]

Should we set disconnected = false only if the protocol is supported and remove the disconnected check in the onMessage handler?

[danielrhodes]

I tried this before, but that welcome message was coming in and making the connection monitor think it was still online. It doesn't happen when code is as it is.

[javan]

open: ->
  ActionCable.log("WebSocket onopen event, protocol is '#{@getProtocol()}'")
  if @isProtocolSupported()
    @disconnected = false
  else
    ActionCable.log("WebSocket opened with an unsupported protocol. Stopping monitor and disconnecting.")
    @close(allowReconnect: false)

[javan]

Guess that ^ doesn't remove the need to check disconnected in the message handler, but I find it clearer since it consolidates the logging and removes the early return.

[danielrhodes]

Yeah I was fooling around with this and was going back and forth, but then I settled on a successful connection means actual server connection and a protocol is chosen. That work we did with the welcome message was actually quite useful in telling the client something was wrong during authentication failure, and we would lose this in the case where a successful connection means just connecting to the server.

[javan]

We've hit sporadic missing subscriptions in Basecamp. Hoping this is the cause! ❤️

[javan]

I settled on a successful connection means actual server connection and a protocol is chosen.

👍 to that, but you're setting disconnected = false before checking the protocol.

[javan]

Reviewing the open handler and where it sets @disconnected is the only thing left to address before this can be merged, IMO.

[danielrhodes]

Alright I changed the message handler to check @isActive instead. It works the same. I don't quite understand why it worked with @disconnected even though as you pointed out there could be a race condition -- whatever underlying threading is going on obviously prioritized the close before the message is all I can surmise.

[jeremy]

Re. subscription requests before welcome: we buffer incoming messages and process the buffer after welcome, so sending early subscription requests should work as expected.

[javan]

Looking great! Any issues deploying the server-side changes while clients still have the previous .js assets running?

[danielrhodes]

@javan There shouldn't be. If the client doesn't send any protocols it wants to use, the server won't either.

[javan]

Sorry to nitpick, but add a newline after this comment please. Indicates that all methods below are "private", not just the isPorotocolSupported method.

[danielrhodes]

👍

[danielrhodes]

Squashed the commits for squashlover7, err I mean @maclover7 :-P

P.S. I've actually started to like squashing now too, even if it's a conspiracy to keep everybody's commit numbers low to suppress rankings.

[javan]

This isn't right. @isActive is a function, not a property.

I think we want to check that the WebSocket is using a supported protocol here so messages aren't received during the brief window after successfully connecting using an unsupported protocol.

message: (event) ->
  return unless @isSupportedProtocol()
  ...

[danielrhodes]

👍

[javan]

Not loving how this method reads now. The early return only guards against logging. I suggest:

open: ->
  ActionCable.log("WebSocket onopen event, using '#{@getProtocol()}' subprotocol")
  @disconnected = false
  unless @isProtocolSupported()
    ActionCable.log("Protocol is unsupported. Stopping monitor and disconnecting.")
    @close(allowReconnect: false)

[danielrhodes]

👍

[javan]

OK to merge as far as I'm concerned. Thanks for sticking with this one @danielrhodes!

[jeremy]

Fantastic work @danielrhodes!

[kaspth]

Well done! ❤️

[cdurante]

is this typo?

I see the the function isProtocolSupported, but not isSupportedProtocol

[Error] TypeError: this.isSupportedProtocol is not a function. (In 'this.isSupportedProtocol()', 'this.isSupportedProtocol' is undefined) message (connection.self-7605b1bcff260663a68e0847ec3cfece34a28e2cb3c61fb59bcac2dfab6b6ad7.js:131) (anonymous function)

[jeremy]

Sure is. Thanks @cdurante!