Bug 1807140: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert #676

lilic · 2020-02-25T16:59:26Z

cc @openshift/openshift-team-monitoring

/hold
Lets see if we need this adjusted first or if its fine like this. So far in 4.4 there were no alerts firing for this, but then we only have CI and nightlies and the OOM bug was fixed.

openshift-ci-robot · 2020-02-25T17:04:23Z

@lilic: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-02-25T17:04:34Z

@lilic: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

paulfantom · 2020-02-26T09:56:48Z

jsonnet/rules.jsonnet

+              message: 'Multiple containers were out of memory killed within the past 15 minutes.',
+            },
+            labels: {
+              severity: 'info',


Why info? OOM means we don't have enough memory in a cluster and it should be scaled, which in many cases is a manual action requiring human intervention and possibly a page. Can we raise it to warning?

No, because this alert is not exact we do not want to do warning level, decided in the PR that was merged and offline conversations with the team. They are adding a proper metric in crio directly in 4.5 that will actually have the info we need on how many containers are being OOM and we will remove this alert and make it warning then. For now as this is good enough alert and its mainly used for us to get a ballpark on how often we see OOMs in the wild.

See #668 and #642 (comment)

It's mostly because this is a heuristic, we assume that if a container restarted and it's last termination reason was an OOM, then all of those restarts were OOMs. If we truly knew that x amount of OOMs happened then I would agree to making this warning but with this heuristic I would be more careful.

openshift-ci-robot · 2020-02-26T10:41:28Z

@lilic: This pull request references Bugzilla bug 1807139, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "4.5.0" instead
expected Bugzilla bug 1807139 to depend on a bug in one of the following states: MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1807139: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lilic · 2020-02-26T17:28:32Z

note: This lowers the limit, compared to master, to 3 containers OOMing. I tested and this is the rough guesstimate of my testing, we can always adjust later.

Should be ready to be merged. @openshift/openshift-team-monitoring PTAL

/hold cancel

brancz · 2020-02-26T17:31:12Z

jsonnet/rules.jsonnet

@@ -196,6 +196,17 @@
              severity: 'warning',
            },
          },
+          {
+            expr: 'sum(max by(namespace, container, pod) (increase(kube_pod_container_status_restarts_total[12m])) and max by(namespace, container, pod) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}) == 1) > 5',


iirc 12min comes from some kube behavior right? Do we have some docs about this to link to for future Frederic to understand?

ha :D yes this is the way kube restarts containers, no docs just a lot of testing with various numbers of timespans, so it eventually normalised the graph.

ok, that works for me .. no need for comment then

lilic · 2020-02-26T17:31:32Z

/bugzilla refresh

openshift-ci-robot · 2020-02-26T17:31:34Z

@lilic: This pull request references Bugzilla bug 1807139, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "4.5.0" instead
expected Bugzilla bug 1807139 to depend on a bug in one of the following states: MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

metalmatze · 2020-02-26T18:25:42Z

Looking at this without extra context is there a reason this isn't added to the kubernetes-mixin project first and then in here?
Is it the urgency of the Bugzilla? Doesn't seem to Openshift specific. 🤔

lilic · 2020-02-26T18:51:09Z

@metalmatze we will be adding the proper alert in the mixin project when the crio metric is available, see the metric in the PR linked in the previous PR that was merged into master, this PR is just backporting it :D This is a guess alert and not exact at all, we want to fully test it and just want to get an indication of how often these alerts fire, see discussion in this PR a bit above :) #676 (comment) and its also detailed in the bugzilla linked and on slack. ;) I have created a jira ticket for that so we don't forget to remove this once the proper metric that tracks the number of OOM killed containers is added.

brancz · 2020-02-26T18:56:40Z

/lgtm
/retest

openshift-bot · 2020-02-26T19:09:10Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-26T19:15:05Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-26T20:46:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-28T20:26:25Z

/retest

Please review the full test history for this PR and help us cut down flakes.

s-urbaniak · 2020-03-02T11:46:18Z

@lilic this needs a rebase now

openshift-ci-robot · 2020-03-02T12:24:57Z

@lilic: This pull request references Bugzilla bug 1807140, which is invalid:

expected dependent Bugzilla bug 1807139 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is NEW instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1807140: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lilic · 2020-03-02T12:26:02Z

@smarterclayton to have this merged in 4.4 we need to have the dependant bugzilla in verified or modified state, you mentioned you will get it verified?

smarterclayton · 2020-03-03T16:45:05Z

#668 was at 10, you need change master to 5 and then associate it with 1807139 and then once it merges this will merge (you don't need a verified PR in master, you just need a merged PR in master)

lilic · 2020-03-04T10:47:59Z

/bugzilla refresh

openshift-ci-robot · 2020-03-04T10:48:03Z

@lilic: This pull request references Bugzilla bug 1807140, which is invalid:

expected dependent Bugzilla bug 1807139 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is POST instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lilic · 2020-03-04T10:48:27Z

PR opened #690

lilic · 2020-03-05T12:05:36Z

/bugzilla refresh

openshift-ci-robot · 2020-03-05T12:05:43Z

@lilic: This pull request references Bugzilla bug 1807140, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "4.5.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This triggers when multiple containers were killed within a certain range.This alert is not 100% accurate, but it is the best we can currently do, with the metrics we have.

lilic · 2020-03-05T12:25:37Z

Had to rebase so lgtm was removed @openshift/openshift-team-monitoring PTAL

lilic · 2020-03-05T13:11:36Z

/bugzilla refresh

openshift-ci-robot · 2020-03-05T13:11:43Z

@lilic: This pull request references Bugzilla bug 1807140, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.4.0) matches configured target release for branch (4.4.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1807139 is in the state MODIFIED, which is one of the valid states (MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED (ERRATA))
dependent Bugzilla bug 1807139 targets the "4.5.0" release, matching the expected (4.5.0) release
bug has dependents

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

s-urbaniak · 2020-03-05T14:57:34Z

/lgtm

openshift-ci-robot · 2020-03-05T14:58:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brancz, lilic, s-urbaniak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [brancz,lilic,s-urbaniak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-03-05T15:00:48Z

@lilic: All pull requests linked via external trackers have merged. Bugzilla bug 1807140 has been moved to the MODIFIED state.

In response to this:

Bug 1807140: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rphillips · 2020-03-05T21:47:25Z

/cherry-pick release-4.3

openshift-cherrypick-robot · 2020-03-05T21:47:40Z

@rphillips: #676 failed to apply on top of branch "release-4.3":

Applying: assets,pkg: Regenerate files
error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	assets/prometheus-k8s/rules.yaml
M	pkg/manifests/bindata.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/manifests/bindata.go
CONFLICT (content): Merge conflict in pkg/manifests/bindata.go
Auto-merging assets/prometheus-k8s/rules.yaml
Patch failed at 0002 assets,pkg: Regenerate files

In response to this:

/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 25, 2020

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 25, 2020

openshift-ci-robot requested review from paulfantom and pgier February 25, 2020 17:17

paulfantom reviewed Feb 26, 2020

View reviewed changes

lilic changed the title ~~jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert~~ Bug 1807139: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert Feb 26, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Feb 26, 2020

lilic force-pushed the bring-alert-oom branch from fbcbf40 to ee14eb5 Compare February 26, 2020 17:25

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 26, 2020

brancz reviewed Feb 26, 2020

View reviewed changes

openshift-ci-robot assigned brancz Feb 26, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 26, 2020

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 28, 2020

lilic force-pushed the bring-alert-oom branch from ee14eb5 to 4b17e1f Compare March 2, 2020 12:17

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Mar 2, 2020

lilic changed the title ~~Bug 1807139: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert~~ Bug 1807140: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert Mar 2, 2020

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 5, 2020

lilic added 2 commits March 5, 2020 13:21

jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert

74518bd

This triggers when multiple containers were killed within a certain range.This alert is not 100% accurate, but it is the best we can currently do, with the metrics we have.

assets,pkg: Regenerate files

28d09e0

lilic force-pushed the bring-alert-oom branch from 4b17e1f to 28d09e0 Compare March 5, 2020 12:24

openshift-ci-robot removed lgtm Indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Mar 5, 2020

lilic changed the title ~~Bug 1807140: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert~~ Bug 1807140: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert Mar 5, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Mar 5, 2020

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Mar 5, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 5, 2020

openshift-merge-robot merged commit 1e1fcac into openshift:release-4.4 Mar 5, 2020

lilic deleted the bring-alert-oom branch March 5, 2020 15:01

rphillips mentioned this pull request Mar 5, 2020

[release-4.3] Bug 1807141: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert #692

Merged

Bug 1807140: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert #676

Bug 1807140: jsonnet/rules.jsonnet: Add MultipleContainersOOMKilled alert #676

Conversation

lilic commented Feb 25, 2020

openshift-ci-robot commented Feb 25, 2020

openshift-ci-robot commented Feb 25, 2020

paulfantom Feb 26, 2020

Choose a reason for hiding this comment

lilic Feb 26, 2020

Choose a reason for hiding this comment

brancz Feb 26, 2020

Choose a reason for hiding this comment

openshift-ci-robot commented Feb 26, 2020

lilic commented Feb 26, 2020

brancz Feb 26, 2020

Choose a reason for hiding this comment

lilic Feb 26, 2020 • edited

Choose a reason for hiding this comment

brancz Feb 26, 2020

Choose a reason for hiding this comment

lilic commented Feb 26, 2020

openshift-ci-robot commented Feb 26, 2020

metalmatze commented Feb 26, 2020

lilic commented Feb 26, 2020

brancz commented Feb 26, 2020

openshift-bot commented Feb 26, 2020

openshift-bot commented Feb 26, 2020

openshift-bot commented Feb 26, 2020

openshift-bot commented Feb 28, 2020

s-urbaniak commented Mar 2, 2020

openshift-ci-robot commented Mar 2, 2020

lilic commented Mar 2, 2020

smarterclayton commented Mar 3, 2020

lilic commented Mar 4, 2020

openshift-ci-robot commented Mar 4, 2020

lilic commented Mar 4, 2020

lilic commented Mar 5, 2020

openshift-ci-robot commented Mar 5, 2020

lilic commented Mar 5, 2020

lilic commented Mar 5, 2020

openshift-ci-robot commented Mar 5, 2020

s-urbaniak commented Mar 5, 2020

openshift-ci-robot commented Mar 5, 2020

openshift-ci-robot commented Mar 5, 2020

rphillips commented Mar 5, 2020

openshift-cherrypick-robot commented Mar 5, 2020

lilic Feb 26, 2020 •

edited