mgr/crash: raise warning about recent crashes and other improvements #29034

liewegas · 2019-07-14T16:33:28Z

raise health warning about recent crashes
'ls-new' as well as 'ls' command
'acrhive' and 'archive-all' commands
keep crashes in mgr memory
automatic pruning

liewegas · 2019-07-14T16:35:27Z

@alfredodeza I had some battles with python 3 here, mainly with all of the comprehensions from dicts. IIUC, I have to do for key, value in mydict.items(), whereas in python2 the .items() wasn't needed. is that right?

dmick · 2019-07-15T22:16:29Z

I believe in Python2 "for x in d" would give you the keys, so the same as "for x in d.keys()". You always needed items() or iteritems() to get (k, v) pairs. iteritems() is the default in P3, so is now just items().

Signed-off-by: Sage Weil <sage@redhat.com>

alfredodeza · 2019-07-17T14:16:00Z

This is compatible with both Python versions (which Dan explains):

for key, value in dictionary.items()

doc/rados/operations/health-checks.rst

src/pybind/mgr/crash/module.py

src/pybind/mgr/pg_autoscaler/module.py

src/pybind/mgr/crash/module.py

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas · 2019-07-19T18:46:29Z

@noahdesu i removed the telemetry test since the crash module is doing a better validity check on the crash dump.. that ok?

Signed-off-by: Sage Weil <sage@redhat.com>

dotnwat · 2019-07-22T20:25:58Z

Yup

…

On Fri, Jul 19, 2019 at 11:47 AM Sage Weil ***@***.***> wrote: @noahdesu i removed the telemetry test since the crash module is doing a better validity check on the crash dump.. that ok? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Signed-off-by: Sage Weil <sage@redhat.com>

all comments addressed

liewegas · 2019-07-24T14:31:18Z

(finally) passing tests! http://pulpito.ceph.com/sage-2019-07-23_18:50:19-rados-wip-sage-testing-2019-07-23-0925-distro-basic-smithi/

neha-ojha · 2019-07-24T17:08:26Z

doc/rados/operations/health-checks.rst

+
+The time period for what "recent" means is controlled by the option
+``mgr/crash/warn_recent_interval`` (default: two weeks).
+


should we also add retain_interval here?

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas · 2019-07-24T17:57:57Z

I pushed another commit that adds it to the mgr/crash.rst file. I don't think it belongs in the health report section since it's unrelated to the report (or mitigating it).

neha-ojha · 2019-07-24T20:10:46Z

I pushed another commit that adds it to the mgr/crash.rst file. I don't think it belongs in the health report section since it's unrelated to the report (or mitigating it).

looks good

* refs/pull/29034/head: doc/mgr/crash: document missing commands, options qa/suites/rados/singleton/all/test-crash: whitelist RECENT_CRASH qa/suites/rados/mgr/tasks/insights: whitelist RECENT_CRASH qa/tasks/mgr/test_insights: crash module now rejects bad crash reports mgr/telemetry: fix remote into crash do_ls() mgr/crash: don't make these methods static mgr/BaseMgrModule: handle unicode health detail strings mgr/crash: verify timestamp is valid qa/suites/mgr: whitelist RECENT_CRASH mgr/crash: remove unused var mgr/crash: remove unused import 'six' qa/workunits/rados/test_crash: health check mgr/crash: improve validation on post mgr/crash: automatically prune old crashes after a year mgr/crash: raise RECENT_CRASH warning for recent (new) crashes mgr/crash: add 'crash ls-new' mgr/crash: add option and serve infra mgr/crash: keep copy of crashes in memory mgr/pg_autoscaler: adjust style to match built-in tables mgr/crash: make 'crash ls' a nice table with a NEW column mgr/crash: nicely format 'crash info' output mgr/crash: add 'crash archive <id>', 'crash archive-all' commands Reviewed-by: Neha Ojha <nojha@redhat.com>

liewegas requested review from dmick and neha-ojha July 14, 2019 16:33

liewegas added core feature labels Jul 14, 2019

liewegas added the wip-sage-testing label Jul 15, 2019

liewegas added 2 commits July 16, 2019 09:22

mgr/crash: add 'crash archive <id>', 'crash archive-all' commands

31f650c

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: nicely format 'crash info' output

35f15f8

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas force-pushed the wip-crash-health branch 2 times, most recently from e7773d4 to cbedfbc Compare July 16, 2019 14:26

alfredodeza previously requested changes Jul 17, 2019

View reviewed changes

liewegas force-pushed the wip-crash-health branch 6 times, most recently from b28067b to 0619d26 Compare July 18, 2019 14:38

liewegas requested a review from alfredodeza July 18, 2019 23:56

alfredodeza reviewed Jul 19, 2019

View reviewed changes

src/pybind/mgr/crash/module.py Outdated Show resolved Hide resolved

src/pybind/mgr/crash/module.py Outdated Show resolved Hide resolved

src/pybind/mgr/pg_autoscaler/module.py Show resolved Hide resolved

src/pybind/mgr/crash/module.py Show resolved Hide resolved

liewegas added 8 commits July 19, 2019 09:43

mgr/crash: make 'crash ls' a nice table with a NEW column

c691f2f

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/pg_autoscaler: adjust style to match built-in tables

44a99f2

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: keep copy of crashes in memory

6ae4c42

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: add option and serve infra

7cbaa5a

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: add 'crash ls-new'

bebb1c3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: raise RECENT_CRASH warning for recent (new) crashes

c885ee7

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: automatically prune old crashes after a year

b2e2085

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: improve validation on post

422871d

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas force-pushed the wip-crash-health branch from 0619d26 to 08be693 Compare July 19, 2019 14:44

liewegas requested a review from alfredodeza July 19, 2019 18:46

liewegas added 9 commits July 19, 2019 14:45

qa/workunits/rados/test_crash: health check

f6e0894

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: remove unused import 'six'

5b2cbb3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: remove unused var

37a8f2d

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/mgr: whitelist RECENT_CRASH

e2ff58b

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: verify timestamp is valid

277d974

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/BaseMgrModule: handle unicode health detail strings

52875c6

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: don't make these methods static

32a4b51

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/telemetry: fix remote into crash do_ls()

01d1d23

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_insights: crash module now rejects bad crash reports

3bddcc5

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas force-pushed the wip-crash-health branch from 08be693 to 9941e28 Compare July 19, 2019 21:21

liewegas added wip-sage2-testing and removed wip-sage-testing labels Jul 20, 2019

qa/suites/rados/mgr/tasks/insights: whitelist RECENT_CRASH

209ce4e

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas force-pushed the wip-crash-health branch from 9941e28 to 209ce4e Compare July 22, 2019 18:42

qa/suites/rados/singleton/all/test-crash: whitelist RECENT_CRASH

9257175

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas added wip-sage-testing and removed wip-sage2-testing labels Jul 22, 2019

neha-ojha approved these changes Jul 24, 2019

View reviewed changes

doc/mgr/crash: document missing commands, options

1ea7570

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas merged commit 1ea7570 into ceph:master Jul 24, 2019

liewegas deleted the wip-crash-health branch July 24, 2019 23:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mgr/crash: raise warning about recent crashes and other improvements #29034

mgr/crash: raise warning about recent crashes and other improvements #29034

liewegas commented Jul 14, 2019

liewegas commented Jul 14, 2019

dmick commented Jul 15, 2019

alfredodeza commented Jul 17, 2019

liewegas commented Jul 19, 2019

dotnwat commented Jul 22, 2019 via email

liewegas commented Jul 24, 2019

neha-ojha Jul 24, 2019

liewegas commented Jul 24, 2019 via email

neha-ojha commented Jul 24, 2019


		The time period for what "recent" means is controlled by the option
		``mgr/crash/warn_recent_interval`` (default: two weeks).

mgr/crash: raise warning about recent crashes and other improvements #29034

mgr/crash: raise warning about recent crashes and other improvements #29034

Conversation

liewegas commented Jul 14, 2019

liewegas commented Jul 14, 2019

dmick commented Jul 15, 2019

alfredodeza commented Jul 17, 2019

liewegas commented Jul 19, 2019

dotnwat commented Jul 22, 2019 via email

liewegas commented Jul 24, 2019

neha-ojha Jul 24, 2019

Choose a reason for hiding this comment

liewegas commented Jul 24, 2019 via email

neha-ojha commented Jul 24, 2019