Make OOM not be a SIGKILL

[grosser]

Atm apps that go over the memory limit are hard killed 'OOMKilled', which is bad (losing state / not running cleanup code etc)

Is there a way to get SIGTERM instead (with a grace period or 100m before reaching the limit) ?

[smarterclayton]

@kubernetes/sig-node-feature-requests

[vishh]

It is not possible to change OOM behavior currently. Kubernetes (or runtime) could provide your container a signal whenever your container is close to its memory limit. This will be on a best effort basis though because memory spikes might not be handled on time.

[grosser]

FYI using this crutch atm https://github.com/grosser/preoomkiller

any idea what would need to change to make OOM behavior configureable ?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[yujuhong]

/remove-lifecycle stale
/cc @dashpole

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[tonglil]

Was this meant to be closed?

It seems like @yujuhong meant to say /remove-lifecycle rotten?

[dashpole]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[Draiken]

When the node is reaching OOM levels I guess I understand some SIGKILLs happening but when a pod is reaching it's manually set resource limit it also gets a SIGKILL. As the initial post mentions, this can cause a lot of harm.

As a workaround we're going to try and make the pod unhealthy before it reaches the memory limit to get a graceful shutdown.
Kubernetes sending this signal beforehand would solve the issue.

If I want this feature created, how should I go about it? Should I provide a PR with code changes or ping someone to make a proposal?

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[gajus]

This remains an active issue.

There appears to be no way to gracefully handle OOMKilled at the moment.

@dashpole Can this be re-opened?

[dashpole]

/reopen
/remove-lifecycle rotten

[andy-v-h]

What makes this really interesting is sometimes you want the kernel to do the OOMKill when the node is under memory pressure, other times it woul dbe nice if kubernetes would preemptively SIGTERM as a OOMKill because the container is approaching a set limit

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[george-angel]

/remove-lifecycle stale

[s2maki]

Even better would be that OOM causes the call to brk to fail (aka malloc returning NULL), allowing the application to gracefully handle running out of memory the normal way instead of killing the process when it goes over.

[lonre]

It should provide opportunity to gracefully shutdown

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[george-angel]

/remove-lifecycle stale

[ffromani]

for reasons outlined in #40157 (comment) we can't just change the signal delivered. OTOH we can integrate with some OOM daemons, but this would require a separate discussion and KEP.

[ffromani]

Kubernetes does not use issues on this repo for support requests. If you have a question on how to use Kubernetes or to debug a specific issue, please visit our forums.

/remove-kind feature
/kind support
/close

[k8s-ci-robot]

@fromanirh: Closing this issue.

In response to this:

Kubernetes does not use issues on this repo for support requests. If you have a question on how to use Kubernetes or to debug a specific issue, please visit our forums.

/remove-kind feature
/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tsuna]

@fromanirh this is not a support request, it's a legit feature request.

[T3rm1]

@fromanirh Can you reopen this, please? It's clearly a feature request, not a support case.

[aojea]

for reasons outlined in #40157 (comment) we can't just change the signal delivered. OTOH we can integrate with some OOM daemons, but this would require a separate discussion and KEP.

it is a feature request for the Kernel not for Kubernetes , the kernel generates the SIGKILL

[johnnyshields]

@aojea there non-kernel solutions here, such as triggering graceful shutdowns at a threshold memory usage (e.g. 95%) before the hard limit is reached.

[aojea]

@aojea there non-kernel solutions here, such as triggering graceful shutdowns at a threshold memory usage (e.g. 95%) before the hard limit is reached.

oh, that is not clear from the title and from the comments, sorry

for reasons outlined in #40157 (comment) we can't just change the signal delivered. OTOH we can integrate with some OOM daemons, but this would require a separate discussion and KEP.

so it should be retitled or open a new issue with the clear request ... and for sure that will need a KEP

[johnnyshields]

Sure, how about titling it: "Add graceful memory usage-based SIGTERM before hard OOM kill happens"

[mike-chen-samsung]

Is there another Issue that's been recreated for this? I can't find one in the Issues list. If not, I can create a new feature request issue.