New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mon: add ability to mute health alerts #29422

Merged

liewegas merged 28 commits into ceph:master from liewegas:wip-mute

Aug 15, 2019

Member

liewegas commented Jul 31, 2019 •

edited by smithfarm

https://tracker.ceph.com/issues/40420

TODO

add qa/standalone test(s)
documentation mute/unmute commands, behavior

batrick added mon needs-review labels

Member Author

liewegas commented Aug 1, 2019

retest this please

liewegas requested review from jecluis and neha-ojha

August 1, 2019 02:43

liewegas added core feature labels

sebastian-philipp requested a review from LenzGr

August 1, 2019 09:24

Contributor

sebastian-philipp commented Aug 1, 2019

@LenzGr do we need a dashboard integration?

Member Author

liewegas commented Aug 1, 2019

@LenzGr do we need a dashboard integration?

Definitely!

liewegas force-pushed the wip-mute branch from 385eeb9 to dac0943 Compare

August 1, 2019 20:03

liewegas added the wip-sage3-testing label

liewegas changed the title ~~mon: add ability to mute health aleeeerts~~ mon: add ability to mute health alerts

dzafman mentioned this pull request

feature: Health warnings on long network ping times, add "dump_osd_network" to get a report #28755

Merged

3 tasks

liewegas force-pushed the wip-mute branch from dac0943 to 65caa8a Compare

August 5, 2019 20:01

liewegas requested a review from dzafman

August 6, 2019 22:08

dzafman reviewed

View reviewed changes

Contributor

dzafman left a comment

Comment for 0ef1b8d should say "unmute" not "unmount"

qa/standalone/mon/health-mute.sh Show resolved Hide resolved

qa/standalone/mon/health-mute.sh Outdated Show resolved Hide resolved

src/mon/HealthMonitor.cc Show resolved Hide resolved

liewegas force-pushed the wip-mute branch from ec357ca to 6804913 Compare

August 7, 2019 22:26

liewegas added wip-sage2-testing and removed wip-sage3-testing labels

liewegas requested a review from a team as a code owner

August 8, 2019 16:22

liewegas force-pushed the wip-mute branch 2 times, most recently from 1490f4c to fa511c0 Compare

August 8, 2019 20:27

liewegas added wip-sage-testing and removed wip-sage2-testing labels

neha-ojha approved these changes

View reviewed changes

Member

neha-ojha left a comment

needs a rebase and left a few nits, other looks great! (sorry for the delay in reviewing this)

doc/rados/operations/health-checks.rst Outdated

+              MGR_DOWN
+              ________
+              All manager` daemons are currently down.  The cluster should normally

Member

neha-ojha Aug 10, 2019

nit: extra "`"

doc/rados/operations/health-checks.rst Outdated

+              All manager` daemons are currently down.  The cluster should normally
+              have at least one running manager (``ceph-mgr``) daemon.  If no
+              manager daemon is running, the clusters ability to monitor itself will

Member

neha-ojha Aug 10, 2019

nit: cluster's

doc/rados/operations/health-checks.rst Outdated

+              ``mon_osd_snap_trim_queue_warn_on`` option (default: 32768).
+              This warning may trigger if OSDs are under excessive load and unable
+              to keep up with their background work, or if the OSDs internal

Member

neha-ojha Aug 10, 2019

nit: OSD's or OSDs'

doc/rados/operations/health-checks.rst Outdated

+              ``mon_data_size_warn`` (default: 15 GiB).
+              A large database is unusual, but may not necessarily indicate a
+              problem.  Monitor databases may grow in size of there are placement

Member

neha-ojha Aug 10, 2019

s/of/when/

doc/rados/operations/monitoring.rst Outdated

+                ceph health unmute OSD_DOWN
+              A health check mute may optionally have a TTL (time to live)
+              associated with it, sucht that the mute will automatically expire

Member

neha-ojha Aug 10, 2019

s/sucht/such

src/mon/MonCommands.h Outdated

@@ @@ -211,6 +211,11 @@ COMMAND_WITH_FLAG("injectargs " \ @@
               COMMAND("status", "show cluster status", "mon", "r")
               COMMAND("health name=detail,type=CephChoices,strings=detail,req=false", \
               	"show cluster health", "mon", "r")
+              COMMAND("health mute name=code,type=CephString name=ttl,type=CephString,req=false " \

Member

neha-ojha Aug 10, 2019

nit: having each name in a fresh line may be easier to read

liewegas added 4 commits

August 14, 2019 20:37


          mon/health_checks: drop dump_summary_compat

ee30f1b

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/health_checks: format 'health summary' with a colon

2a3f89f

Signed-off-by: Sage Weil <sage@redhat.com>


          mon: refactor/simplify health [detail]

062681a

Get rid of single caller helpers.  Instead, assimilate all the checks
together at once, and have two separate blocks, one for formatted, and
one for plaintext output.  Much easier to follow!

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: maintain list of mutes

e2c320d

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas added 23 commits

August 14, 2019 20:37


          mon: apply mutes to health [detail]

553f1c5

- de-escalate severity
- mark mutes in structured output
- note mutes in summary text output
- mark mutes in detail text output

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: expire mutes based on ttl

ffc98aa

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: add sticky flag to mutes

2dff777

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: add gather_all_health_checks helper

1600ff6

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: automatically clear non-sticky mutes when alert cl…

70db3f0

…ears

If the alert goes away, drop the mute.

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: move get_health_status()

29d7430

This operates exclusively on HealthMonitor members.  Make public member
private again.

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: persist summary for non-sticky mutes

c09dc4e

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/PGMap: fix summary form for POOL_APP_NOT_ENABLED

1ae154a

Count goes first.

Signed-off-by: Sage Weil <sage@redhat.com>


          doc/rados/operations/health-alerts: document BLUESTORE_NO_COMPRESSION

1b6745e

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/PGMap: fix summary form for bluestore health alerts

e4784af

Count goes first.

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: allow muted alert counts to decrease but not increase

0acff80

If the summary starts with a digit, parse a count.

If the count goes up, clear the mute.

If the count goes down, update the mute so that we ratchet the threshold
down.

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/PGMap: use nice timespan for PG stuck warnings

164de7d

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthMonitor: simplify health alert dump

7834cd8

Use dump() member instead of duplicating!  The only reason we had this
before was because the detail portion was optinoal

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/health_checks: associate a count with health_alert_t

d0eb22f

0 means this is a singleton.  Otherwise, we can sum this up, either
via merge() or get_or_add().

We always structure this so the count goes toward zero (more healthy), so
if a value is too low, then we count how much too low it is.

Signed-off-by: Sage Weil <sage@redhat.com>


          mon/HealthCheck: check mutes based on count, not parsing the summary …

7385e91

…string

This is more explicit and robust, and works with the PG warnings, which
don't conform to the "%d ..." form that the other messages do.

Signed-off-by: Sage Weil <sage@redhat.com>


          doc/rados/operations/health-checks: document MGR_DOWN

078ef21

Signed-off-by: Sage Weil <sage@redhat.com>


          doc/reados/operations/health-checks: document PG_SLOW_SNAP_TRIMMING

7e9ba0a

The mitigation steps are weak, but it's not clear concrete guidance to
provide.

Signed-off-by: Sage Weil <sage@redhat.com>


          doc/rados/operations/health-checks: document AUTH_BAD_CAPS

0eba993

Signed-off-by: Sage Weil <sage@redhat.com>


          doc/rados/operations/health-checks: document OSD_NO_DOWN_OUT_INTERVAL

dd5e985

Signed-off-by: Sage Weil <sage@redhat.com>


          doc/rados/operations/health-checks: document MON_DISK_{LOW,CRIT,BIG}

95b8e9f

Signed-off-by: Sage Weil <sage@redhat.com>


          qa/standalone/mon/health-mutes: add tests

710fef9

Make sure mute and unmute work.  Make sure stick is sticky. Mkae sure
counts can go down bupt if they go upt hte mute clears.

Signed-off-by: Sage Weil <sage@redhat.com>


          doc/rados/operations/monitoring: document muting health alerts

2a1b58b

I think someday the docs for how health alerts work (here) and the
enumeration of all actual alerts should be restructured.  For now this
si the simplest placde to fit this!

Signed-off-by: Sage Weil <sage@redhat.com>t


          qa/tasks/mgr/dashboard/test_health: update schema

403f119

Also fix the 'checks' field, which is a list of objects, not strings.  (The
test doesn't notice because it's empty.)

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas force-pushed the wip-mute branch from fa511c0 to 403f119 Compare

August 15, 2019 01:40

liewegas removed the needs-review label

liewegas merged commit 403f119 into ceph:master

liewegas added a commit that referenced this pull request


          Merge PR #29422 into master

09beeb0

* refs/pull/29422/head:
	qa/tasks/mgr/dashboard/test_health: update schema
	doc/rados/operations/monitoring: document muting health alerts
	qa/standalone/mon/health-mutes: add tests
	doc/rados/operations/health-checks: document MON_DISK_{LOW,CRIT,BIG}
	doc/rados/operations/health-checks: document OSD_NO_DOWN_OUT_INTERVAL
	doc/rados/operations/health-checks: document AUTH_BAD_CAPS
	doc/reados/operations/health-checks: document PG_SLOW_SNAP_TRIMMING
	doc/rados/operations/health-checks: document MGR_DOWN
	mon/HealthCheck: check mutes based on count, not parsing the summary string
	mon/health_checks: associate a count with health_alert_t
	mon/HealthMonitor: simplify health alert dump
	mon/PGMap: use nice timespan for PG stuck warnings
	mon/HealthMonitor: allow muted alert counts to decrease but not increase
	mon/PGMap: fix summary form for bluestore health alerts
	doc/rados/operations/health-alerts: document BLUESTORE_NO_COMPRESSION
	mon/PGMap: fix summary form for POOL_APP_NOT_ENABLED
	mon/HealthMonitor: persist summary for non-sticky mutes
	mon/HealthMonitor: move get_health_status()
	mon/HealthMonitor: automatically clear non-sticky mutes when alert clears
	mon/HealthMonitor: add gather_all_health_checks helper
	mon/HealthMonitor: add sticky flag to mutes
	mon/HealthMonitor: expire mutes based on ttl
	mon: apply mutes to health [detail]
	mon/HealthMonitor: implement mute and unmount commands
	mon/HealthMonitor: maintain list of mutes
	mon: refactor/simplify health [detail]
	mon/health_checks: format 'health summary' with a colon
	mon/health_checks: drop dump_summary_compat

Reviewed-by: Neha Ojha <nojha@redhat.com>

sebastian-philipp mentioned this pull request

mgr/k8sevents: Add mgr module for kubernetes event integration #29520

Merged

Contributor

smithfarm commented Sep 17, 2019 •

edited

@liewegas (echoing https://tracker.ceph.com/issues/40420#note-2) Since https://tracker.ceph.com/issues/40420 references this PR as its fix, I added that URL to the description.

Should this feature be backported to nautilus?

neha-ojha mentioned this pull request

nautilus: mon: Warn when too many reads are repaired on an OSD #35686

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment