add scrub persist/query API #6898

tchaikov · 2015-12-11T15:01:24Z

http://tracker.ceph.com/issues/13505 Fixes: #13505

test for this PR ceph/ceph-qa-suite#795

related blueprint/discussion can be found at

loic-bot · 2015-12-11T16:44:19Z

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/10090/

SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/10090/

tchaikov · 2015-12-14T09:21:48Z

src/osd/ReplicatedPG.cc

@@ -12233,7 +12292,7 @@ void ReplicatedPG::_scrub(
      bufferlist bv;
      bv.push_back(p->second.attrs[OI_ATTR]);
      try {
-	oi = object_info_t(); // Initialize optional<> before decode into it
+        oi = object_info_t(); // Initialize optional<> before decode into it


remove this unintended space only change.

athanatos · 2015-12-15T17:57:37Z

Some comments, getting close!

tchaikov · 2015-12-17T14:01:26Z

i am preparing a new pull request implementing the get_inconsistent_snapsets().

dzafman · 2015-12-22T19:28:59Z

@tchaikov It is unfortunate that an object_info_t here has the same name as the internal OSD struct. As the user visible one is really a complete identifier of an object maybe object_id_t or object_identifier_t would be better.

tchaikov · 2015-12-23T08:08:44Z

@dzafman thanks. will go with object_id_t.

tchaikov · 2016-01-13T13:10:36Z

use object_id_t instead of object_info_t
fixed the handling of missing object in ObjectError::encode().

tchaikov · 2016-01-13T13:16:36Z

the work-in-progress test suite is posted at ceph/ceph-qa-suite#795 for proof-reading

athanatos · 2016-01-19T15:47:38Z

src/osd/PG.cc

@@ -3994,6 +4009,16 @@ void PG::chunky_scrub(ThreadPool::TPHandle &handle)
 	  scrubber.reserved_peers.clear();
 	}

+	{
+	  hobject_t oid = Scrub::make_scrub_object(info.pgid.pgid);


There might be two shards on the same OSD for the same PG. The scrub object should probably be built using the spg_t rather than the pg_t.

will do. and as will not encode the pg into the object name, as temp coll is per pg.

tchaikov · 2016-02-19T14:36:44Z

changes

in PG::chunky_scrub(): cleanup before creating a new scrub object.

tchaikov · 2016-02-19T15:14:13Z

the test suite: ceph/ceph-qa-suite#795

athanatos · 2016-02-23T17:58:38Z

src/osd/ScrubStore.cc

+{
+  return ("SCRUB_OBJ_" +
+	  std::to_string(pool) + "." +
+	  oid.name + oid.nspace + std::to_string(oid.snap));


This is problematic because you could have two objects in the same pg:

name: foo namespace: bar

name: foob namespace: ar
We need a seperator. I think there are other places in the code where object names are used to generate keys, how about using hobject_t::to_str()?

athanatos · 2016-02-23T19:56:58Z

I just set up a vstart cluster to play with it a bit. One thing I noticed is that after a repair, the list-inconsistent-obj api still returns the old incorrect values. I think the simplest solution would be for us to not populate the scrub store during a repair?

tchaikov · 2016-02-24T12:22:19Z

I think the simplest solution would be for us to not populate the scrub store during a repair?

will do.

tchaikov · 2016-02-24T14:05:06Z

changelog

osd: use hobject_t::to_str() to build the keys of scrub results.
osd: do not populate scrub result when repairing
tools/rados: pretty print the snap field. i.e. print "head", "snapdir", or hex of snapid
librados: expose rados_inconsistent_pg_list as a C API.
pybind: expose Rados.get_inconsistent_pgs
rebased against master

Signed-off-by: Kefu Chai <kchai@redhat.com>

to list the inconsistent PGs of given pool, it's a wrapper around the "ceph pg ls" command. Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

to list inconsistent PGs of a given pool. this command exposes the underlying get_inconsistent_pgs() API to user. Signed-off-by: Kefu Chai <kchai@redhat.com>

which present the inconsistent objects found in scrub Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

persist inconsistent objects found when comparing the ScrubMaps collected from replica/shards. the discrepancies between the auth copy and the replica are identified as inconsistencies. and hence encoded into the omap of an object of the temp coll of the PG in question. scrub_types.{h,cpp} are introduced to hide the details of how we persist the scrub types from the librados client. Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Samuel Just <sjust@redhat.com>

for presenting the inconsistent snapsets found in scrub Signed-off-by: Kefu Chai <kchai@redhat.com>

the inconsistent snapsets are identified in ReplicatedPG::_scrub() after we compared the authorized objects with their replica/shards. these inconsistent information are stored in the omap of objects with prefix "SCRUB_SS_". Signed-off-by: Kefu Chai <kchai@redhat.com>

it is a new pg op which returns the encoded objects stored when scrubbing. Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

to list inconsistent objects of a given PG, this command exposes get_inconsistent_objects() rados API to user. Signed-off-by: Kefu Chai <kchai@redhat.com>

to list inconsistent snapsets of a given PG, this command exposes get_inconsistent_snapsets() rados API to user. Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

add scrub persist/query API Reviewed-by: Samuel Just <sjust@redhat.com>

tchaikov added feature core labels Dec 11, 2015

tchaikov reviewed Dec 14, 2015
View reviewed changes

tchaikov force-pushed the wip-13505 branch 2 times, most recently from ddd3d59 to fb37a0c Compare December 22, 2015 08:44

tchaikov mentioned this pull request Dec 28, 2015

Wip scrub optimization #7056

Closed

tchaikov force-pushed the wip-13505 branch 5 times, most recently from bfd27ff to e32d61d Compare January 13, 2016 13:10

athanatos reviewed Jan 19, 2016
View reviewed changes

athanatos reviewed Feb 23, 2016
View reviewed changes

tchaikov force-pushed the wip-13505 branch from aa1d48f to 1eefb9e Compare February 24, 2016 13:59

tchaikov force-pushed the wip-13505 branch 2 times, most recently from d36129e to e3e23ca Compare February 24, 2016 15:08

tchaikov and others added 15 commits February 25, 2016 12:40

tools/rados: support more --format options

50bbf7f

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add get_inconsistent_pgs() to librados

cb4efbd

to list the inconsistent PGs of given pool, it's a wrapper around the "ceph pg ls" command. Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: add Rados.get_inconsistent_pgs method

d0af316

Signed-off-by: Kefu Chai <kchai@redhat.com>

rados: add "list-inconsistent-pg" command

4c32706

to list inconsistent PGs of a given pool. this command exposes the underlying get_inconsistent_pgs() API to user. Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add inconsistent_obj_t types

b43d480

which present the inconsistent objects found in scrub Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: more constness to spg_t

f236b7d

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/: clear scrub store safetly

8ed6277

Signed-off-by: Samuel Just <sjust@redhat.com>

librados: add inconsistent_snapset_t type

c828c39

for presenting the inconsistent snapsets found in scrub Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: add CEPH_OSD_OP_SCRUBLS pg op

3dea4f1

it is a new pg op which returns the encoded objects stored when scrubbing. Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add get_inconsistent_objects() API

dfc2f48

Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add get_inconsistent_snapsets() API

c9b593d

Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

rados: add "list-inconsistent-obj" cmd

8018eab

to list inconsistent objects of a given PG, this command exposes get_inconsistent_objects() rados API to user. Signed-off-by: Kefu Chai <kchai@redhat.com>

rados: add "list-inconsistent-snapset" cmd

e2374c4

to list inconsistent snapsets of a given PG, this command exposes get_inconsistent_snapsets() rados API to user. Fixes: ceph#13505 Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov force-pushed the wip-13505 branch from e3e23ca to e2374c4 Compare February 25, 2016 04:42

tchaikov added a commit that referenced this pull request Feb 25, 2016

Merge pull request #6898 from tchaikov/wip-13505

ce1169d

add scrub persist/query API Reviewed-by: Samuel Just <sjust@redhat.com>

tchaikov merged commit ce1169d into ceph:master Feb 25, 2016

tchaikov deleted the wip-13505 branch February 25, 2016 04:48

tchaikov added this to the jewel milestone Mar 2, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add scrub persist/query API #6898

add scrub persist/query API #6898

tchaikov commented Dec 11, 2015

loic-bot commented Dec 11, 2015

tchaikov Dec 14, 2015

athanatos commented Dec 15, 2015

tchaikov commented Dec 17, 2015

dzafman commented Dec 22, 2015

tchaikov commented Dec 23, 2015

tchaikov commented Jan 13, 2016

tchaikov commented Jan 13, 2016

athanatos Jan 19, 2016

tchaikov Jan 21, 2016

tchaikov commented Feb 19, 2016

tchaikov commented Feb 19, 2016

athanatos Feb 23, 2016

tchaikov Feb 24, 2016

athanatos commented Feb 23, 2016

tchaikov commented Feb 24, 2016

tchaikov commented Feb 24, 2016

add scrub persist/query API #6898

add scrub persist/query API #6898

Conversation

tchaikov commented Dec 11, 2015

loic-bot commented Dec 11, 2015

tchaikov Dec 14, 2015

Choose a reason for hiding this comment

athanatos commented Dec 15, 2015

tchaikov commented Dec 17, 2015

dzafman commented Dec 22, 2015

tchaikov commented Dec 23, 2015

tchaikov commented Jan 13, 2016

tchaikov commented Jan 13, 2016

athanatos Jan 19, 2016

Choose a reason for hiding this comment

tchaikov Jan 21, 2016

Choose a reason for hiding this comment

tchaikov commented Feb 19, 2016

tchaikov commented Feb 19, 2016

athanatos Feb 23, 2016

Choose a reason for hiding this comment

tchaikov Feb 24, 2016

Choose a reason for hiding this comment

athanatos commented Feb 23, 2016

tchaikov commented Feb 24, 2016

tchaikov commented Feb 24, 2016