Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broadcasting options #2

Open
palkan opened this issue Dec 4, 2016 · 17 comments
Open

Broadcasting options #2

palkan opened this issue Dec 4, 2016 · 17 comments
Labels

Comments

@palkan
Copy link
Member

@palkan palkan commented Dec 4, 2016

Currently, AnyCable requires Redis as a dependency for the only one thing: sending messages from the application to the WebSocket server for broadcastings (ActionCable.server.broadcast ...).

The idea is to provide more options for the users and thus not to require Redis for everyone.

TODO adapters:

Other possible options:

  • UDP ?
  • ...
@palkan palkan added the discussion label Dec 4, 2016
@kobaltz
Copy link

@kobaltz kobaltz commented Dec 21, 2016

Personally, I like having Redis as a dependency as most of my current applications use two Redis instances; persistent store and volatile. However, I'd always be open to something different if the maintainability/performance was the same or better. I would caution against PostgreSQL as a dependency as that would be more limiting than having a Redis dependency. While I do use PostgreSQL in some applications, we have to use other databases in other cases.

@maurcarvalho
Copy link

@maurcarvalho maurcarvalho commented Dec 21, 2016

HTTP REST seems like an "out of external dependency" way to go.

@palkan can you say a little more about how do you imagine it?
will it be something like a http handler between WS and ActionCable.server.broadcast ...

@palkan
Copy link
Member Author

@palkan palkan commented Dec 21, 2016

@kobaltz The idea is to avoid additional dependency if it's possible. Most Rails applications already use Redis, so they are likely to be OK with it (and that's why for now Redis is the only option, btw).

But there are applications which don't use it. And that's the main reason why we're looking for alternatives.

There can also be one more reason: performance and scalability. For i-dunno-how-much-large apps, it would be better to use RabbitMQ or even Kafka.

We do not have to provide all possible options but an abstraction to build them (as separate gems).
On the other hand, to develop a good abstraction it would be useful to know these alternatives beforehand.

P.S. As you probably remember, Action Cable initially also depended on Redis, the community didn't like it.

@palkan
Copy link
Member Author

@palkan palkan commented Dec 21, 2016

@maurcarvalho Sure.

Agree, that's the easiest option (from the development point of view).

Assume that our WebSocket server implements HTTP endpoint to receive broadcast messages. Then we can write our #broadcast method like this:

def broadcast(channel, payload)
  Net::HTTP.post_form(AnyCable.config.http_endpoint, { "channel" => channel, "payload" => payload })
end

Of course, it's a very straightforward solution. It would be great to re-use connection ('keep-alive') of even have a connection pool, send requests asynchronously and deal with errors.
Maybe, we should put it to ActiveJob out-of-box.

Not so easy)

@AnthonySuper
Copy link

@AnthonySuper AnthonySuper commented Dec 24, 2016

If you're already using GRPC, why not also use it to deliver messages to the server?

It can be something like:

service AnycableServer {
    rpc Broadcast (BroadcastingMesssage) returns (google.protobuf.Empty) {}
}

message Message {
    string channel = 1;
    string payload = 2;
}

Now, this would make a server a bit harder to implement, which is something we need to consider.

@palkan
Copy link
Member Author

@palkan palkan commented Dec 26, 2016

If you're already using GRPC, why not also use it to deliver messages to the server?

That's the reason:

... this would make a server a bit harder to implement

We have to run GRPC server as a part of our WebSocket server, which is not available yet for all languages (e.g. for Erlang).

And another problem: consider clustered configuration – several WebSocket servers, several application servers. Then we need a GRPC balancer – yet another problem.

@AnthonySuper
Copy link

@AnthonySuper AnthonySuper commented Dec 26, 2016

Okay, gotcha. I didn't even consider the load balancing part, which is a pretty big issue now that I think about it.

An HTTP solution seems like it will likely be the best for now, in that case.

@palkan palkan mentioned this issue Apr 18, 2017
@pbrumm
Copy link

@pbrumm pbrumm commented Apr 18, 2017

I think having nsq would be very interesting for the message passing.
http://nsq.io/components/nsqd.html

it enables the actioncable component to just put the message into a queue, and then each instance of the websocket server could register a channel on that queue with a name from a config file. This would allow multiple websocket servers to receive messages without concern for maintaining an active connection with the rails app.

the rails app could also be modified to just be reading messages from a queue with a single channel.

nsq has client libraries in many languages and is written in go.

@slayer
Copy link

@slayer slayer commented Jun 10, 2017

What about MQTT ?

@palkan
Copy link
Member Author

@palkan palkan commented Jun 10, 2017

@slayer

What about MQTT ?

Although, MQTT-based broadcasting is possible, I think, it would too complicated for our use case. The main (IMO) feature of MQTT – quality of service – doesn't make sense in our case: if a WebSocket server is down and doesn't receive broadcast messages (through HTTP/Redis/queue), it's likely not to handle client connections too.

Broadcast messages a ephemeral from the WS server point of view. At least for now.

@jtoy
Copy link

@jtoy jtoy commented Jul 13, 2018

love the idea of postgres pub/sub

@khalilovcmd
Copy link

@khalilovcmd khalilovcmd commented Jan 23, 2019

Re:

HTTP REST – great for development/testing,

Nice! 👍

small projects with the only one websocket server instance

Just trying to understand more, if understand correctly, the main advantage of AnyCable is: to write a high-performant real-time application using Ruby as a language for implementing a business-logic.

Wouldn't depending on a HTTP transport medium null the performance benefits that is gained by AnyCable specifically for small projects?

I think it is also subjective of how small is a small project, but wondering to get more of your thoughts about it.

@palkan
Copy link
Member Author

@palkan palkan commented Jan 23, 2019

Wouldn't depending on a HTTP transport medium null the performance benefits that is gained by AnyCable specifically for small projects?

Yep, from the performance point of view there is no a huge benefit.
But you can still have benefits from the resources usage point of view.

how small is a small project

Yeah, that's a good question.
We should think about the number of simultaneous connections (peak and average) and the message rate/payload size.
I think, the threshold to start thinking about AnyCable (instead of just Action Cable) is somewhere between 500 and 1000 connections on average or 5k-10k during peak hours.

@jtoy
Copy link

@jtoy jtoy commented Jan 23, 2019

@palkan
Copy link
Member Author

@palkan palkan commented Jan 23, 2019

What’s the largest known anycable project?

The largest project I'm aware of (and I'm not aware of the most of them, unfortunately) had ~10k concurrent connections (that's equipe.com, one year ago); they handled this with 4 1x dynos on Heroku (before switching to AnyCable they had 20 2x dynos for ActionCable).

We are testing now if we can support 60k concurrent connections.

Could you please share some information on your use case?

@khalilovcmd
Copy link

@khalilovcmd khalilovcmd commented Jan 24, 2019

@palkan Thanks for the context! – I see you abstracted the broadcasting adapters already, I will give it a try with adding one of the broadcasting adapter options you listed above, I will tag you in a PR ☺️

@palkan
Copy link
Member Author

@palkan palkan commented Jan 26, 2019

@jtoy
Could you please share some information on your use case? I’ve added an issue to collect the feedback:
#68

@palkan palkan mentioned this issue May 8, 2020
5 of 5 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.