[ceph] Ceph commands output enhancement #1726

zerodayz · 2019-07-10T06:46:00Z

Add json-pretty output for most of the useful commands
for easier consumption by Automation tools

Signed-off-by: Robin Cernin rcerninr@redhat.com

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

Is the commit message split over multiple lines and hard-wrapped at 72 characters?
Is the subject and message clear and concise?
Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
Does the commit contain a Signed-off-by: First Lastname email@example.com?

zerodayz · 2019-07-10T06:55:51Z

I have added two self.add_cmd_output just to distinguish the json-pretty format vs human readable if you have any other idea let me know and we could split it or put altogether.

The reason behind this change is that automation tools could parse json output natively and much easier.

pmoravec · 2019-07-10T07:07:04Z

Why the first commit referencing to #1 ? :)

For the commands you take twice (once in human-readable format, once in json): isnt json format sufficient for humans as well? Or how much additional time these extra commands execution could take? (I won't like much duplicated info collected that takes too long time).

zerodayz · 2019-07-10T07:33:42Z

Why the first commit referencing to #1 ? :)

Seems it took the merge commit of my fork.

For the commands you take twice (once in human-readable format, once in json): isnt json format sufficient for humans as well? Or how much additional time these extra commands execution could take? (I won't like much duplicated info collected that takes too long time).

I am getting ceph cluster and will give you the rough estimates. One min

zerodayz · 2019-07-10T07:46:16Z

@pmoravec

[root@mon1 ~]# cat test.sh 
echo "Executing ceph df detail --format json-pretty"
{ time ceph df detail --format json-pretty >/dev/null ; } 2>&1  | grep real
echo "Executing ceph health detail --format json-pretty"
{ time ceph health detail --format json-pretty >/dev/null ; } 2>&1 | grep real
echo "Executing ceph fs dump --format json-pretty"
{ time ceph fs dump --format json-pretty >/dev/null ; } 2>&1 | grep real
echo "Executing ceph osd df --format json-pretty"
{ time ceph osd df --format json-pretty >/dev/null ; } 2>&1 | grep real
echo "Executing ceph osd df tree --format json-pretty"
{ time ceph osd df tree --format json-pretty >/dev/null ; } 2>&1 | grep real
echo "Executing ceph osd dump --format json-pretty"
{ time ceph osd dump --format json-pretty >/dev/null ; } 2>&1 | grep real
echo "Executing ceph osd tree --format json-pretty"
{ time ceph osd tree --format json-pretty >/dev/null ; } 2>&1 | grep real
echo "Executing ceph -s --format json-pretty"
{ time ceph -s --format json-pretty >/dev/null ; } 2>&1 | grep real
[root@mon1 ~]# time ./test.sh 
Executing ceph df detail --format json-pretty
real	0m0.327s
Executing ceph health detail --format json-pretty
real	0m0.333s
Executing ceph fs dump --format json-pretty
real	0m0.327s
Executing ceph osd df --format json-pretty
real	0m0.355s
Executing ceph osd df tree --format json-pretty
real	0m0.362s
Executing ceph osd dump --format json-pretty
real	0m0.325s
Executing ceph osd tree --format json-pretty
real	0m0.353s
Executing ceph -s --format json-pretty
real	0m0.330s

real	0m2.739s
user	0m2.026s
sys	0m0.484s

zerodayz · 2019-07-10T07:49:36Z

To me seems the JSON are faster then regular ones:

[root@mon1 ~]# time ./test2.sh 
Executing ceph df detail 
real	0m0.369s
Executing ceph health detail 
real	0m0.374s
Executing ceph fs dump 
real	0m0.457s
Executing ceph osd df 
real	0m0.350s
Executing ceph osd df tree 
real	0m0.352s
Executing ceph osd dump 
real	0m0.401s
Executing ceph osd tree 
real	0m0.393s
Executing ceph -s 
real	0m0.344s

real	0m3.068s
user	0m2.168s
sys	0m0.627s

zerodayz · 2019-07-10T07:51:04Z

I will update the MR and replace the old ones with JSON, its faster, better usage with automation tools as well it's still readable.

zerodayz · 2019-07-10T08:08:24Z

@pmoravec,

Last commit c4ac964 has the updated commands.

zerodayz · 2019-07-10T09:05:47Z

We are adding 4 new commands to ceph plugin as well output will be in JSON format that allows everyone for easier integration with automation tool.

The execution time raise roughly by 1s

[root@mon1 ~]# time ./test3.sh 

real	0m6.554s
user	0m4.708s
sys	0m1.265s

[root@mon1 ~]# time ./test4.sh 

real	0m5.590s
user	0m3.982s
sys	0m1.140s

CHANGE FROM

            "ceph status",
            "ceph health detail",
            "ceph osd tree",
            "ceph osd stat",
            "ceph osd dump",
            "ceph mon stat",
            "ceph mon dump",
            "ceph df",
            "ceph report",
            "ceph osd df tree",
            "ceph fs dump --format json-pretty",
            "ceph fs ls",
            "ceph pg dump",
            "ceph health detail --format json-pretty",
            "ceph osd crush show-tunables",
            "ceph-disk list"

TO

            "ceph status --format json-pretty",
            "ceph health detail --format json-pretty",
            "ceph osd tree --format json-pretty",
            "ceph osd df tree --format json-pretty",
            "ceph osd stat",
            "ceph osd dump --format json-pretty",
            "ceph osd df --format json-pretty",
            "ceph mon stat",
            "ceph mon dump --format json-pretty",
            "ceph df --format json-pretty",
            "ceph df detail --format json-pretty",
            "ceph report",
            "ceph fs ls --format json-pretty",
            "ceph fs dump --format json-pretty",
            "ceph pg dump --format json-pretty",
            "ceph osd crush show-tunables",
            "ceph-disk list",
            "ceph osd erasure-code-profile ls",
            "ceph -v",
            "ceph -s --format json-pretty"

pmoravec · 2019-07-10T13:28:00Z

ACK to the PR.

ashishkumsingh · 2019-07-11T10:20:51Z

hey @zerodayz, this PR would be helpful from Insights perspective. However, from a support engineer point of view having these command outputs in 'json-pretty' format will make troubleshooting very difficult.

Please close this PR or keep the original commands and append the new commands with 'json-pretty' format.

pmoravec · 2019-07-11T10:28:45Z

How difficult is to read the json format for humans? (I guess it is very subjective)

Am I right that keeping also the non-json commands will increase plugin execution time by approx. 6 seconds?

(I am trying to assess pros and cons - how much it will hurt customers and support if the plugin will always take extra 6s and how much will we gain by the extra-human-convenient format collected)

TurboTurtle · 2019-07-11T14:15:57Z

General ack from me on the PR, but it needs to drop the leading merge commit. We don't allow merge commits in PRs to keep the tree clean.

In regards to @ashishkumsingh's request, I tend to agree about json-formatting being slightly more difficult for human eyes to parse, and I don't think an extras 6 seconds of execution time is killer for us. @zerodayz I'd recommend keeping the current collection and then making a secondary pass through all those commands with logic that appends the --format json-pretty option. To keep the directory clean, I'd say stick the json output under a new subdir, e.g.:

self.add_cmd_output([
    "%s --format json-pretty" % s for s in ceph_cmds
], subdir="json_output")

zerodayz · 2019-07-11T23:49:05Z

BEFORE CHANGE:

[root@mon1 ceph]# ls
ceph_df                            ceph_fs_ls                               ceph_mon_dump                 ceph_osd_df_tree  ceph_osd_tree  ceph_status
ceph-disk_list                     ceph_health_detail                       ceph_mon_stat                 ceph_osd_dump     ceph_pg_dump
ceph_fs_dump_--format_json-pretty  ceph_health_detail_--format_json-pretty  ceph_osd_crush_show-tunables  ceph_osd_stat     ceph_report

AFTER CHANGE:

[root@mon1 ceph]# ls -la
total 88
drwx------.  3 root root 4096 Jul 12 09:43 .
drwxr-xr-x. 62 root root 4096 Jul 12 09:46 ..
-rw-r--r--.  1 root root  343 Jul 12 09:43 ceph_df
-rw-r--r--.  1 root root  595 Jul 12 09:43 ceph_df_detail
-rw-r--r--.  1 root root  161 Jul 12 09:43 ceph-disk_list
-rw-r--r--.  1 root root 1033 Jul 12 09:43 ceph_fs_dump
-rw-r--r--.  1 root root   73 Jul 12 09:43 ceph_fs_ls
-rw-r--r--.  1 root root  511 Jul 12 09:43 ceph_health_detail
-rw-r--r--.  1 root root  245 Jul 12 09:43 ceph_mon_dump
-rw-r--r--.  1 root root  157 Jul 12 09:43 ceph_mon_stat
-rw-r--r--.  1 root root  738 Jul 12 09:43 ceph_osd_df
-rw-r--r--.  1 root root 1204 Jul 12 09:43 ceph_osd_df_tree
-rw-r--r--.  1 root root 2812 Jul 12 09:43 ceph_osd_dump
-rw-r--r--.  1 root root    8 Jul 12 09:43 ceph_osd_erasure-code-profile_ls
-rw-r--r--.  1 root root   19 Jul 12 09:43 ceph_osd_stat
-rw-r--r--.  1 root root  784 Jul 12 09:43 ceph_osd_tree
-rw-r--r--.  1 root root 5748 Jul 12 09:43 ceph_pg_dump
-rw-r--r--.  1 root root  562 Jul 12 09:43 ceph_-s
-rw-r--r--.  1 root root  562 Jul 12 09:43 ceph_status
-rw-r--r--.  1 root root   91 Jul 12 09:43 ceph_-v
drwx------.  2 root root 4096 Jul 12 09:44 json_output
[root@mon1 ceph]# ls -la json_output/
total 216
drwx------. 2 root root  4096 Jul 12 09:44 .
drwx------. 3 root root  4096 Jul 12 09:43 ..
-rw-r--r--. 1 root root  1284 Jul 12 09:43 ceph_df_detail_--format_json-pretty
-rw-r--r--. 1 root root   748 Jul 12 09:43 ceph_df_--format_json-pretty
-rw-r--r--. 1 root root  3207 Jul 12 09:43 ceph_fs_dump_--format_json-pretty
-rw-r--r--. 1 root root   230 Jul 12 09:43 ceph_fs_ls_--format_json-pretty
-rw-r--r--. 1 root root  1677 Jul 12 09:43 ceph_health_detail_--format_json-pretty
-rw-r--r--. 1 root root   884 Jul 12 09:43 ceph_mon_dump_--format_json-pretty
-rw-r--r--. 1 root root   606 Jul 12 09:44 ceph_osd_crush_show-tunables
-rw-r--r--. 1 root root  4494 Jul 12 09:43 ceph_osd_df_--format_json-pretty
-rw-r--r--. 1 root root  6354 Jul 12 09:43 ceph_osd_df_tree_--format_json-pretty
-rw-r--r--. 1 root root 13350 Jul 12 09:43 ceph_osd_dump_--format_json-pretty
-rw-r--r--. 1 root root   154 Jul 12 09:43 ceph_osd_stat_--format_json-pretty
-rw-r--r--. 1 root root  4458 Jul 12 09:43 ceph_osd_tree_--format_json-pretty
-rw-r--r--. 1 root root 68231 Jul 12 09:43 ceph_pg_dump_--format_json-pretty
-rw-r--r--. 1 root root 52081 Jul 12 09:44 ceph_report
-rw-r--r--. 1 root root  4776 Jul 12 09:43 ceph_-s_--format_json-pretty
-rw-r--r--. 1 root root  4776 Jul 12 09:43 ceph_status_--format_json-pretty

zerodayz · 2019-07-12T00:09:54Z

@ashishkumsingh Do you agree with the last PR? It keeps the old commands in place and adds the new ones in json_output directory.
@TurboTurtle done

sos/plugins/ceph.py

zerodayz · 2019-07-12T15:32:51Z

@ashishkumsingh What about now?

ashishkumsingh · 2019-07-15T05:13:41Z

@zerodayz LGTM

zerodayz · 2019-07-15T23:30:40Z

@TurboTurtle Seems fine now, can we merge the PR please?

TurboTurtle · 2019-07-16T13:06:59Z

It looks fine to me, however merges are done by @bmr-cymru who is on leave at the moment.

zerodayz · 2019-08-13T06:31:43Z

@TurboTurtle is @bmr-cymru still on leave? Thanks!

ashishkumsingh · 2019-08-29T07:14:09Z

@zerodayz can you rebase this PR ?

Prior to this patch, ovn_host was disabled on containerized setups due to the fact that ovn-controller package is not installed in the host. This patch fixes it by checking if the ovn-controller process is running. Resolves: sosreport#1767 Signed-off-by: Daniel Alvarez <dalvarez@redhat.com> Signed-off-by: Bryn M. Reeves <bmr@redhat.com>

lgtm-com · 2019-09-17T06:11:21Z

This pull request introduces 1 alert when merging 9962ee3 into a895bf4 - view on LGTM.com

new alerts:

1 for Implicit string concatenation in a list

lgtm-com · 2019-09-17T06:29:55Z

This pull request introduces 1 alert when merging eb52c26 into a895bf4 - view on LGTM.com

new alerts:

1 for Implicit string concatenation in a list

pmoravec · 2019-09-18T10:37:57Z

sos/plugins/ceph.py

            "ceph osd crush show-tunables",
            "ceph-disk list",
            "ceph versions",
            "ceph osd crush dump"
+            "ceph -v"


Add a coma to the end of previous line (that's the "implicit concatenation" failure of LGTM - currently a command "ceph osd crush dumpceph -v" would be executed).

pmoravec · 2019-09-18T10:41:04Z

sos/plugins/ceph.py

+            "fs ls",
+            "fs dump",
+            "pg dump",
+        ]


Why some ceph * commands are taken from this list while others are executed "directly" (cf. L58-62 and L82)? Just because the later ones are taken with json format as well? (maybe renaming the ceph_cmds to something more obvious will help code clarity)

Because those commands share both regular and json output

ceph_cmds = [ "status", "health detail", "osd tree", "osd stat", "osd df tree", "osd dump", "osd df", "mon dump", "df", "df detail", "fs ls", "fs dump", "pg dump", ]

Regular output

self.add_cmd_output([ "ceph %s" % s for s in ceph_cmds ])

As well the same command can be re-used and collect json-pretty

self.add_cmd_output([ "ceph %s --format json-pretty" % s for s in ceph_cmds ], subdir="json_output")

That output is using same commands from ceph_cmds, they are shared and just subdirectory is different, this is to avoid duplicated code.

pmoravec · 2019-09-18T10:41:27Z

Generally ACK up to the missing coma.

zerodayz · 2019-09-19T02:00:43Z

@pmoravec I am rebasing on master and always get this change in way: sos/plugins/ovn_host.py Not sure what I can do to drop sos/plugins/ovn_host.py off the change.

Add json-pretty output for most of the useful commands for easier consumption by Automation tools Signed-off-by: Robin Cernin rcerninr@redhat.com

pmoravec

I guess you forgot to update your master branch of your fork (that you forked before the a895bf4 commit with ovn_host)?

Anyway we can merge just your commit.

zerodayz force-pushed the ceph-enhance branch from 1d64259 to 20f287a Compare July 10, 2019 08:33

zerodayz force-pushed the ceph-enhance branch 2 times, most recently from 03b9048 to 7f700ea Compare July 12, 2019 00:08

ashishkumsingh suggested changes Jul 12, 2019

View reviewed changes

sos/plugins/ceph.py Outdated Show resolved Hide resolved

sos/plugins/ceph.py Outdated Show resolved Hide resolved

zerodayz force-pushed the ceph-enhance branch from 7f700ea to c60fac9 Compare July 12, 2019 15:31

zerodayz force-pushed the ceph-enhance branch 2 times, most recently from f800826 to 9962ee3 Compare September 17, 2019 06:03

zerodayz force-pushed the ceph-enhance branch from 9962ee3 to eb52c26 Compare September 17, 2019 06:21

pmoravec reviewed Sep 18, 2019

View reviewed changes

zerodayz force-pushed the ceph-enhance branch from eb52c26 to 31958c2 Compare September 19, 2019 02:19

[ceph] Ceph commands output enhancement

6877fe1

Add json-pretty output for most of the useful commands for easier consumption by Automation tools Signed-off-by: Robin Cernin rcerninr@redhat.com

zerodayz force-pushed the ceph-enhance branch from 31958c2 to 6877fe1 Compare September 19, 2019 02:21

pmoravec approved these changes Sep 19, 2019

View reviewed changes

bmr-cymru closed this in 66ff290 Oct 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ceph] Ceph commands output enhancement #1726

[ceph] Ceph commands output enhancement #1726

zerodayz commented Jul 10, 2019 •

edited by pmoravec

Loading

zerodayz commented Jul 10, 2019

pmoravec commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019 •

edited

Loading

pmoravec commented Jul 10, 2019

ashishkumsingh commented Jul 11, 2019

pmoravec commented Jul 11, 2019

TurboTurtle commented Jul 11, 2019

zerodayz commented Jul 11, 2019

zerodayz commented Jul 12, 2019

zerodayz commented Jul 12, 2019

ashishkumsingh commented Jul 15, 2019

zerodayz commented Jul 15, 2019

TurboTurtle commented Jul 16, 2019

zerodayz commented Aug 13, 2019

ashishkumsingh commented Aug 29, 2019

lgtm-com bot commented Sep 17, 2019

lgtm-com bot commented Sep 17, 2019

pmoravec Sep 18, 2019

pmoravec Sep 18, 2019

zerodayz Sep 19, 2019 •

edited

Loading

zerodayz Sep 19, 2019

pmoravec commented Sep 18, 2019

zerodayz commented Sep 19, 2019

pmoravec left a comment

[ceph] Ceph commands output enhancement #1726

[ceph] Ceph commands output enhancement #1726

Conversation

zerodayz commented Jul 10, 2019 • edited by pmoravec Loading

zerodayz commented Jul 10, 2019

pmoravec commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019

zerodayz commented Jul 10, 2019 • edited Loading

pmoravec commented Jul 10, 2019

ashishkumsingh commented Jul 11, 2019

pmoravec commented Jul 11, 2019

TurboTurtle commented Jul 11, 2019

zerodayz commented Jul 11, 2019

zerodayz commented Jul 12, 2019

zerodayz commented Jul 12, 2019

ashishkumsingh commented Jul 15, 2019

zerodayz commented Jul 15, 2019

TurboTurtle commented Jul 16, 2019

zerodayz commented Aug 13, 2019

ashishkumsingh commented Aug 29, 2019

lgtm-com bot commented Sep 17, 2019

lgtm-com bot commented Sep 17, 2019

pmoravec Sep 18, 2019

Choose a reason for hiding this comment

pmoravec Sep 18, 2019

Choose a reason for hiding this comment

zerodayz Sep 19, 2019 • edited Loading

Choose a reason for hiding this comment

zerodayz Sep 19, 2019

Choose a reason for hiding this comment

pmoravec commented Sep 18, 2019

zerodayz commented Sep 19, 2019

pmoravec left a comment

Choose a reason for hiding this comment

zerodayz commented Jul 10, 2019 •

edited by pmoravec

Loading

zerodayz commented Jul 10, 2019 •

edited

Loading

zerodayz Sep 19, 2019 •

edited

Loading