Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add the ability to wait for a period of time after SIGTERM #3298

Merged
merged 4 commits into from
Oct 26, 2022
Merged

Add the ability to wait for a period of time after SIGTERM #3298

merged 4 commits into from
Oct 26, 2022

Commits on Oct 26, 2022

  1. Add the ability to wait for a period of time after SIGTERM 

    Adds the ability to supply a "shutdown delay" to Mimir components such
that they will disable HTTP keep-alives and mark themselves as not
ready when receiving SIGTERM or SIGINT (via HTTP /ready or gRPC) but
wait a configurable amount of time before actually stopping.

Fixes an issue during rollouts on Kubernetes where the Grafana Cloud
Gateway holds on to connections to query-frontends even when they
shutdown resulting in user-facing read errors. By closing connections
during shutdown, marking the component as "not ready", and still
continuing to serve requests we ensure that:

* Users don't see any disruption
* Connections to the stopping component are not pooled
* Kubernetes service endpoints are removed before the pod

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
    @56quarters
    56quarters committed Oct 26, 2022
    Configuration menu
    Copy the full SHA
    558123f View commit details
    Browse the repository at this point in the history

  2. Code review changes. 

    Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
    @56quarters
    56quarters committed Oct 26, 2022
    Configuration menu
    Copy the full SHA
    2691502 View commit details
    Browse the repository at this point in the history

  3. Handle shutdown inline in Run goroutine. 

    Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
    @56quarters
    56quarters committed Oct 26, 2022
    Configuration menu
    Copy the full SHA
    a8a84a4 View commit details
    Browse the repository at this point in the history

  4. Phrasing. 

    Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
    @56quarters
    56quarters committed Oct 26, 2022
    Configuration menu
    Copy the full SHA
    1fb3f42 View commit details
    Browse the repository at this point in the history