Rolling Restarts with Rails ActionController::Live

[Geokoumpa]

Hello,
we have a live controller in our rails app, with an action streaming new alerts to users. We have implemented the heartbeat solution as described here rails/rails#10989

We can't however do a rolling restart when at least one open session exists. As soon as the user closes the client and the heartbeat kicks in, the thread is killed and the restart goes on, but until then Puma remains down. This is n't a situation we can handle on production of course, so I thought I'd ask how anyone else has handled it. Otherwise live controllers are pretty much useless.

[Leooo]

Having the same issue, puma_worker_killer doesn't restart a worker until after a user disconnects. Did you find a way to solve this properly? Could interrupt the SSEs from the server, but don't want to interrupt all the SSEs, would like to shut only the connections created by the worker which is shutting down..

[Geokoumpa]

I decided using a separate server for SSEs. I used this gem http://mikeatlas.github.io/realtime-rails/ first, but the node server proposed here uses websockets, which didn't work for me due to infrastructure restrictions. So I made a similar express.js server with SSEs and kept using the same gem server side for authentication. Let me know if you need to use the SSE version.

[Leooo]

Tried to keep the ActionController::Live streaming inside my main app, and indeed I have the same issue: I tried to shut down all the SSEs for the worker I want to restart by fetching the index of the worker in the SSE loop:

ObjectSpace.each_object(Puma::Cluster::Worker).map { |obj| obj }.first.try(:index)

But this is not enough: I have to close ALL SSEs connections for ALL workers in my app before being able to restart a particular worker, even the connections on other workers. Seems like a bug to me (from Puma cluster, not from ActionController::Live?) and that makes ActionController::Live with Puma not scalable.

=> Feature request: be able to restart a worker even if some (SSEs) processes on other workers are running?

[tetherit]

Any solution for this?

[evanphx]

You'll be able to use force_shutdown_after to tell puma to kill any Live running threads in 3.4.0.