Skip to content

Graceful shutdown / SIGTERM #6

Open
@njsmith

Description

@njsmith

When Heroku decides to restart our app (e.g. for an upgrade, or just automatically once a day), it sends SIGTERM and then waits 30 seconds for it to exit.

Right now, we don't have any SIGTERM handling at all, so

This creates some race conditions: for example, suppose that a PR is merged into this repo by a new contributor. This kicks off two processes:

  • snekomatic sees that it's a new contributor's first PR, and tries to invite them and post a comment
  • heroku sees that there's a new commit in snekomatic, and tries to reboot snekomatic to pick up the changes

In practice we kinda get away with it, because it takes heroku a few seconds to notice the new commit and then build a new container image, so snekomatic wins the race. But if snekomatic gets fancier and starts doing more, this will become more of a problem.

It's easy to have a task listen for SIGTERM. But after it gets it, what should it do? I guess we want to do a graceful shutdown of hypercorn (i.e. finish processing current requests, but stop accepting new ones). I don't think hypercorn has any support for this? And Trio doesn't make it particularly easy yet either (cf python-trio/trio#147).

Oh, no, I'm wrong: hypercorn has a config option shutdown_event that it uses for doing graceful shutdown when running in multi-process mode. Maybe we can use this when from API mode too? Is it just as simple as passing shutdown_event=some_trio_event_object as part of our config? CC @pgjones

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions