Special scrub handling of hinfo_key errors #20947

dzafman · 2018-03-16T22:37:15Z

No description provided.

dzafman · 2018-03-17T00:40:50Z

src/tools/rados/rados.cc

+      bufferlist bl;
+      bufferlist::iterator bliter = k->second.begin();
+      decode(hi, bliter);  // Can't be corrupted
+      f.dump_stream("hashinfo") << hi;


Instead of using operator<< should we output JSON for the HashInfo?

dzafman · 2018-03-17T00:41:19Z

src/tools/rados/rados.cc

+      bufferlist bl;
+      bufferlist::iterator bliter = k->second.begin();
+      decode(ss, bliter);  // Can't be corrupted
+      f.dump_stream("snapset") << ss;


Update this too from operator<< to JSON for SnapSet?

yeah, sounds like a good idea

dzafman · 2018-03-20T16:51:29Z

Make sure that non-authoritative corruption is also reflected in list-inconsistent-snapset.

dzafman · 2018-03-21T18:54:37Z

It would be a much more involved change to handle all shards when dealing with snapshot analysis. I filed tracker http://tracker.ceph.com/issues/23428 to at least indicate which authoritative shard was used to determine snapshot consistency.

dzafman · 2018-03-21T18:56:02Z

Make sure that non-authoritative corruption is also reflected in list-inconsistent-snapset.

This is a lot of work which I don't think is a priority right now.

jdurgin · 2018-03-28T01:06:33Z

This looks good, I agree json for the snapset and hashinfo would be nice as well

tchaikov · 2018-03-29T01:00:55Z

http://pulpito.ceph.com/kchai-2018-03-28_14:42:54-rados-wip-kefu-testing-2018-03-28-1948-distro-basic-smithi/

tchaikov · 2018-03-29T01:02:30Z

@dzafman it's good to merge if we want to improve the json representation of snapset and hashinfo in another PR.

dzafman · 2018-04-06T04:38:47Z

@jdurgin @tchaikov I rebased this branch. Changed the system xattrs to output json format. Added a fix for tracker 23428.

Concerns for object_info/hashinfo/snapset:

It is easier to visually compare the differences in the single line format. Just use that format?
Keep the strings using old name (e.g. "object_info") for compatibility and add "*_json" for the json?
I noticed that the single line format doesn't show all fields
In the json output I'd like to show the flags as an array of strings instead of an integer

dzafman · 2018-04-10T14:49:56Z

retest this please - run-tox-mgr-dashboard (Failed)

jdurgin · 2018-04-10T15:37:31Z

making the snapset/hash_info/object_info json means it's machine-readable, instead of having to parse it again. this is worth it imo, since the single-line format isn't legible to folks who don't know the code already.

Using flag names instead of an int is a good idea. I'm fine keeping the same field names, just add something to PendingReleaseNotes about it.

So basically, this looks good to me, it just needs a release note.

Signed-off-by: David Zafman <dzafman@redhat.com>

Fixes: http://tracker.ceph.com/issues/23364 Signed-off-by: David Zafman <dzafman@redhat.com>

…t_missing/snapset_corrupted Signed-off-by: David Zafman <dzafman@redhat.com>

oi_attr_missing -> info_missing oi_attr_corrupted -> info_corrupted Signed-off-by: David Zafman <dzafman@redhat.com>

data_digest_mismatch_oi -> data_digest_mismatch_info omap_digest_mismatch_oi -> omap_digest_mismatch_info size_mismatch_oi -> size_mismatch_info obj_size_oi_mismatch -> obj_size_info_mismatch Signed-off-by: David Zafman <dzafman@redhat.com>

System attributes shown as "object_info", "snapset" and "hashinfo" Only output user attributes as "attrs" Drop leading undescore "_" for user attribute keys Improve logic as to when to show user attributes or specific system attributes Signed-off-by: David Zafman <dzafman@redhat.com>

Signed-off-by: David Zafman <dzafman@redhat.com>

dzafman · 2018-04-10T20:40:19Z

@jdurgin I had to rebase because of conflicts with the PendingReleaseNotes

Add SnapSet bufferlist to inconsistent_snapset_t Partial fix for http://tracker.ceph.com/issues/23428 Signed-off-by: David Zafman <dzafman@redhat.com>

Signed-off-by: David Zafman <dzafman@redhat.com>

dzafman · 2018-04-10T22:39:52Z

retest this please

tchaikov · 2018-04-13T02:55:10Z

PendingReleaseNotes

+    now snapset_missing).  The error snapset_mismatch has been renamed to snapset_error
+    to better reflect what it means.
+
+  * The head snapset information is output in json format as "snapset."  This means that


dzafman added feature bug-fix tools labels Mar 16, 2018

dzafman force-pushed the wip-23364 branch from 91b2f12 to 128bb4f Compare March 17, 2018 00:27

dzafman commented Mar 17, 2018

View reviewed changes

dzafman requested review from tchaikov and jdurgin March 19, 2018 16:27

dzafman added the needs-qa label Mar 19, 2018

dzafman changed the title ~~Special scrub handling of hinfo_key errors~~ DNM: Special scrub handling of hinfo_key errors Mar 20, 2018

dzafman force-pushed the wip-23364 branch from 128bb4f to 668f7b3 Compare March 21, 2018 18:49

dzafman changed the title ~~DNM: Special scrub handling of hinfo_key errors~~ Special scrub handling of hinfo_key errors Mar 21, 2018

tchaikov approved these changes Mar 28, 2018

View reviewed changes

tchaikov added the wip-kefu-testing label Mar 28, 2018

tchaikov removed needs-qa wip-kefu-testing labels Mar 29, 2018

dzafman added the DNM label Apr 4, 2018

dzafman force-pushed the wip-23364 branch 2 times, most recently from d54e33a to 70e91b3 Compare April 6, 2018 04:28

dzafman removed the DNM label Apr 10, 2018

dzafman force-pushed the wip-23364 branch from 3d62cb0 to 96481ce Compare April 10, 2018 19:20

dzafman force-pushed the wip-23364 branch from 96481ce to 8f93ed8 Compare April 10, 2018 19:23

dzafman added 12 commits April 10, 2018 13:26

test: Remove check that masks differences (let diff fail)

be815f9

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Don't crash in get_hash_info() with a corrupt HashInfo

0d4acc5

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Improve scrub handling of HashInfo (hinfo_key xattr)

d713c7d

Fixes: http://tracker.ceph.com/issues/23364 Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Rename ss_attr_missing/ss_attr_corrupted to snapse…

bec67e3

…t_missing/snapset_corrupted Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Change "oi_attr" to "info" in scrub handling errors

273f621

oi_attr_missing -> info_missing oi_attr_corrupted -> info_corrupted Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Without snapdir we don't have a head mismatch error anymore

0a8ed0b

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Clean-up test and use local values for number of objects and osds

0c7ac9d

Signed-off-by: David Zafman <dzafman@redhat.com>

osd rados command: Change error name snapset_mismatch to snapset_error

007cb45

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add getjson variable to save output

1b1d45b

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: SnapSet::dump() handle snapid_t not in clone_overlap or clone_size

866a52e

Signed-off-by: David Zafman <dzafman@redhat.com>

dzafman force-pushed the wip-23364 branch from 6e6cbcd to 96bf558 Compare April 10, 2018 20:28

jdurgin approved these changes Apr 10, 2018

View reviewed changes

dzafman added 3 commits April 10, 2018 13:51

osd rados command: Show snapset in list-inconsistent-snapset

60ae2b8

Add SnapSet bufferlist to inconsistent_snapset_t Partial fix for http://tracker.ceph.com/issues/23428 Signed-off-by: David Zafman <dzafman@redhat.com>

test: Be smarter about when jsonschema can be used

9c5ef19

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Add release notes for the output format changes

da0f588

Signed-off-by: David Zafman <dzafman@redhat.com>

dzafman force-pushed the wip-23364 branch from 96bf558 to da0f588 Compare April 10, 2018 20:52

dzafman merged commit 80137ca into ceph:master Apr 10, 2018

dzafman deleted the wip-23364 branch April 10, 2018 23:32

This was referenced Apr 12, 2018

osd: Change shard digests to hex like object info digests #21362

Merged

luminous: Special scrub handling of hinfo_key errors #21397

Merged

tchaikov reviewed Apr 13, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Special scrub handling of hinfo_key errors #20947

Special scrub handling of hinfo_key errors #20947

dzafman commented Mar 16, 2018

dzafman Mar 17, 2018 •

edited

jdurgin Mar 28, 2018

dzafman Mar 17, 2018

jdurgin Mar 28, 2018

dzafman commented Mar 20, 2018 •

edited

dzafman commented Mar 21, 2018

dzafman commented Mar 21, 2018

jdurgin commented Mar 28, 2018

tchaikov commented Mar 29, 2018

tchaikov commented Mar 29, 2018 •

edited

dzafman commented Apr 6, 2018 •

edited

dzafman commented Apr 10, 2018

jdurgin commented Apr 10, 2018

dzafman commented Apr 10, 2018

dzafman commented Apr 10, 2018

tchaikov Apr 13, 2018

Special scrub handling of hinfo_key errors #20947

Special scrub handling of hinfo_key errors #20947

Conversation

dzafman commented Mar 16, 2018

dzafman Mar 17, 2018 • edited

Choose a reason for hiding this comment

jdurgin Mar 28, 2018

Choose a reason for hiding this comment

dzafman Mar 17, 2018

Choose a reason for hiding this comment

jdurgin Mar 28, 2018

Choose a reason for hiding this comment

dzafman commented Mar 20, 2018 • edited

dzafman commented Mar 21, 2018

dzafman commented Mar 21, 2018

jdurgin commented Mar 28, 2018

tchaikov commented Mar 29, 2018

tchaikov commented Mar 29, 2018 • edited

dzafman commented Apr 6, 2018 • edited

dzafman commented Apr 10, 2018

jdurgin commented Apr 10, 2018

dzafman commented Apr 10, 2018

dzafman commented Apr 10, 2018

tchaikov Apr 13, 2018

Choose a reason for hiding this comment

dzafman Mar 17, 2018 •

edited

dzafman commented Mar 20, 2018 •

edited

tchaikov commented Mar 29, 2018 •

edited

dzafman commented Apr 6, 2018 •

edited