Description
When Heroku decides to restart our app (e.g. for an upgrade, or just automatically once a day), it sends SIGTERM and then waits 30 seconds for it to exit.
Right now, we don't have any SIGTERM handling at all, so
This creates some race conditions: for example, suppose that a PR is merged into this repo by a new contributor. This kicks off two processes:
- snekomatic sees that it's a new contributor's first PR, and tries to invite them and post a comment
- heroku sees that there's a new commit in snekomatic, and tries to reboot snekomatic to pick up the changes
In practice we kinda get away with it, because it takes heroku a few seconds to notice the new commit and then build a new container image, so snekomatic wins the race. But if snekomatic gets fancier and starts doing more, this will become more of a problem.
It's easy to have a task listen for SIGTERM. But after it gets it, what should it do? I guess we want to do a graceful shutdown of hypercorn (i.e. finish processing current requests, but stop accepting new ones). I don't think hypercorn has any support for this? And Trio doesn't make it particularly easy yet either (cf python-trio/trio#147).
Oh, no, I'm wrong: hypercorn has a config option shutdown_event
that it uses for doing graceful shutdown when running in multi-process mode. Maybe we can use this when from API mode too? Is it just as simple as passing shutdown_event=some_trio_event_object
as part of our config? CC @pgjones