osd: fine-grained statistics of logical object space usage #15199

xiexingguo · 2017-05-22T12:35:52Z

To test this change, we create an image of 5GB and do rbd bench write of 100MB:
./bin/rbd create bar -s 5120 && ./bin/rbd bench --io-type write --io-size 32K --io-total 100M --io-pattern rand rbd/bar

Below is the test result.

Was（3191MB）:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27052M        3859M         12.49
POOLS:
    NAME                  ID     USED      %USED     MAX AVAIL     OBJECTS
    rbd                   0      3191M     26.36         8914M        1174
    cephfs_data_a         1          0         0         8914M           0
    cephfs_metadata_a     2       2246         0         8914M          21

Now(about 100MB):

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27050M        3861M         12.49
POOLS:
    NAME                  ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd                   0      101216k      1.10         8913M        1178
    cephfs_data_a         1            0         0         8913M           0
    cephfs_metadata_a     2          892         0         8913M          21

E.g., this change can make "osd pool set-quota max_bytes" work nicely.

Signed-off-by: xie xingguo xie.xingguo@zte.com.cn

xiexingguo · 2017-06-14T11:40:57Z

@liewegas Any comments?

liewegas · 2017-06-14T14:01:34Z

Interesting! This is surprisingly simple.

In order to rely on this we probably need to update (deep?) scrub to verify the accuracy of the extents field. Presumably by reading everything outside the range and asserting it is zeros...

liewegas · 2017-06-14T14:01:48Z

@jdurgin what do you think?

jdurgin · 2017-06-14T15:25:46Z

I like the idea, I'm afraid it will end up a little more complex though. e.g. what about rbd clusters that use the set_alloc_hint to fully allocate space for an object with the first write?

This would also need to handle other write ops, e.g. truncate, zero, rollback, delete, etc.

write_update_size_and_usage() is only called for write, writefull, and ops that are implemented in terms of those.

liewegas · 2017-09-01T14:24:18Z

Yeah, I like the idea. If we can capture the other ops with similarly concise updates I'm all for this.

We'd need to make the compat decode path assume the full range is allocated in order to handle upgrades, and make sure we only trigger the sub-extent updates only if require_osd_release >= MIMIC.

And scrub.

xiexingguo · 2017-09-24T13:22:22Z

This passed several round of QA tests:
http://pulpito.ceph.com/xxg-2017-09-24_03:06:16-rados-wip-object-logic-size-distro-basic-smithi/
http://pulpito.ceph.com/xxg-2017-09-24_11:00:41-rados-wip-object-logic-size-distro-basic-smithi/
http://pulpito.ceph.com/xxg-2017-09-24_05:09:32-rados-wip-object-logic-size-distro-basic-smithi/

@liewegas I believe this is ready for review!

xiexingguo · 2017-09-25T22:44:45Z

@liewegas ping

liewegas · 2017-09-25T22:51:24Z

src/osd/PrimaryLogPG.cc

+      // trunc up
+      interval_set<uint64_t> to_add;
+      to_add.insert(oi.size, truncate_size - oi.size);
+      oi.extents.union_of(to_add);


should a simple oi.extents.insert(oi.size, truncate_size - oi.size) be sufficient? no need for to_add and union_of

liewegas · 2017-09-25T22:52:29Z

src/osd/PrimaryLogPG.cc

+             rollback_to->obs.oi.extents.begin();
+             p != rollback_to->obs.oi.extents.end(); ++p) {
+          obs.oi.extents.insert(p.get_start(), p.get_len());
+        }


wouldn't obs.oi.extents = rollback_to->obs.oid.extents be sufficient?

liewegas · 2017-09-25T22:55:50Z

src/osd/osd_types.cc

@@ -5091,6 +5100,7 @@ void object_info_t::dump(Formatter *f) const
  f->dump_unsigned("expected_write_size", expected_write_size);
  f->dump_unsigned("alloc_hint_flags", alloc_hint_flags);
  f->dump_object("manifest", manifest);
+  f->dump_stream("logical_extents") << extents;


may as well make this structures. open_array_section("extents"), open_object_section("extent"), dump_unsigned("offset", ...), ...

liewegas · 2017-09-25T22:56:51Z

src/osdc/Objecter.h

+               copy_reply.extents.begin();
+               p != copy_reply.extents.end(); ++p) {
+            out_extents->insert(p.get_start(), p.get_len());
+          }


again, here, why not use operator=?

xiexingguo · 2017-09-26T10:49:44Z

retest this please

xiexingguo · 2017-09-27T10:48:19Z

@liewegas All comments addressed.

QA: http://pulpito.ceph.com/xxg-2017-09-27_07:46:11-rados-wip-object-logic-size-distro-basic-smithi/

xiexingguo · 2017-09-27T12:44:29Z

changeset:
fix a typo in comment

xiexingguo · 2017-09-28T11:12:03Z

There are suspicious scrub-errors, need more testing and investigations.

E.g.: subset_of([5~10,20~5], 0, 100) -> [5~10,20~5] subset_of([5~10,20~5], 5, 25) -> [5~10,20~5] subset_of([5~10,20~5], 1, 10) -> [5~5] subset_of([5~10,20~5], 8, 24) -> [8~7, 20~4] Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

To test this change, we create an image of 5GB and do rbd bench write of 1GB: ./bin/rbd create bar -s 5120 && ./bin/rbd bench --io-type write --io-size 32K --io-total 100M --io-pattern rand rbd/bar Below is the test result. Was: GLOBAL: SIZE AVAIL RAW USED %RAW USED 30911M 27052M 3859M 12.49 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 3191M 26.36 8914M 1174 cephfs_data_a 1 0 0 8914M 0 cephfs_metadata_a 2 2246 0 8914M 21 Now: GLOBAL: SIZE AVAIL RAW USED %RAW USED 30911M 27050M 3861M 12.49 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 101216k 1.10 8913M 1178 cephfs_data_a 1 0 0 8913M 0 cephfs_metadata_a 2 892 0 8913M 21 E.g., this change can make "osd pool set-quota max_bytes" work nicely. Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Normal reads support trimmed read length, and so shall checksums! This fixes occasionally failure of rados/thrash test scripts, e.g.: (1) create object using WriteOp with random generated length (2) normal writes might accompany with TruncOp of randomized chosen truncate_size (3) for ReadOp, pick a random 'length' to read, and do checksum simultaneously for the same range ([0, 'length']) to read too. Since the 'length' for reading is randomized chosen, it might exceed the current object size, and hence causing an EOVERFLOW error. Related issues: http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_01:52:47-rados-wip-object-logic-size-distro-basic-smithi/1657337 http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_14:14:19-rados-wip-object-logic-size-distro-basic-smithi/1658015 Fix the above problems by keeping pace with normal reads. Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

xiexingguo · 2017-09-30T02:09:34Z

retest this please

xiexingguo · 2017-09-30T06:50:05Z

http://pulpito.ceph.com/xxg-2017-09-29_22:57:51-rados-wip-object-logic-size-distro-basic-smithi/
http://pulpito.ceph.com/xxg-2017-09-29_10:09:01-rados-wip-object-logic-size-distro-basic-smithi/
http://pulpito.ceph.com/xxg-2017-09-29_13:00:30-rados-wip-object-logic-size-distro-basic-smithi/

…_info_t Introduced-by: ceph#15199 Fixes: http://tracker.ceph.com/issues/21618 Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

It is originally introduced in ceph#15199 aiming at improving the pool-based **du** stats. For performance concerns, ceph#19616 did an incomplete revert of that PR and hence comes the clean-up job... Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

It is originally introduced in ceph#15199 aiming at improving the pool-based **du** stats. For performance concerns, ceph#19616 did an incomplete revert of that PR and hence comes the following clean-up job... Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

xiexingguo added core feature labels May 22, 2017

xiexingguo assigned liewegas May 22, 2017

xiexingguo force-pushed the wip-object-logic-size branch from 580be10 to 5b76b55 Compare June 14, 2017 10:21

liewegas requested a review from jdurgin June 14, 2017 13:59

xiexingguo force-pushed the wip-object-logic-size branch 6 times, most recently from fa6e957 to 0e2628a Compare September 18, 2017 05:38

xiexingguo added the DNM label Sep 18, 2017

xiexingguo force-pushed the wip-object-logic-size branch 6 times, most recently from 5ff30cf to 5e91a99 Compare September 24, 2017 11:44

xiexingguo removed the DNM label Sep 24, 2017

liewegas requested changes Sep 25, 2017

View reviewed changes

xiexingguo force-pushed the wip-object-logic-size branch 2 times, most recently from 281be43 to 30055e0 Compare September 26, 2017 07:07

liewegas added the needs-qa label Sep 27, 2017

liewegas approved these changes Sep 27, 2017

View reviewed changes

xiexingguo force-pushed the wip-object-logic-size branch from 30055e0 to de8dad5 Compare September 27, 2017 12:43

xiexingguo added the wip-xxg-testing label Sep 27, 2017

xiexingguo force-pushed the wip-object-logic-size branch 3 times, most recently from 7349b26 to b4b5c54 Compare September 28, 2017 10:22

xiexingguo removed needs-qa wip-xxg-testing labels Sep 28, 2017

xiexingguo force-pushed the wip-object-logic-size branch from b4b5c54 to c11ca2d Compare September 29, 2017 04:25

xiexingguo added 3 commits September 29, 2017 12:30

xiexingguo force-pushed the wip-object-logic-size branch from c11ca2d to 1e4263f Compare September 30, 2017 01:02

xiexingguo merged commit cd6b983 into ceph:master Sep 30, 2017

xiexingguo deleted the wip-object-logic-size branch September 30, 2017 06:50

xiexingguo mentioned this pull request Oct 3, 2017

qa/standalone/scrub/osd-scrub-repair.sh: add extents flag into object_info_t #18094

Merged

liewegas mentioned this pull request Oct 10, 2017

osd: generalize queueing and lock interface for OpWq #16030

Merged

xiexingguo mentioned this pull request Mar 29, 2018

interval_set: kill subset_of #21108

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

osd: fine-grained statistics of logical object space usage #15199

osd: fine-grained statistics of logical object space usage #15199

xiexingguo commented May 22, 2017 •

edited

xiexingguo commented Jun 14, 2017

liewegas commented Jun 14, 2017

liewegas commented Jun 14, 2017

jdurgin commented Jun 14, 2017

liewegas commented Sep 1, 2017

xiexingguo commented Sep 24, 2017

xiexingguo commented Sep 25, 2017

liewegas Sep 25, 2017

liewegas Sep 25, 2017

liewegas Sep 25, 2017

liewegas Sep 25, 2017

xiexingguo commented Sep 26, 2017

xiexingguo commented Sep 27, 2017 •

edited

xiexingguo commented Sep 27, 2017

xiexingguo commented Sep 28, 2017

xiexingguo commented Sep 30, 2017

xiexingguo commented Sep 30, 2017

osd: fine-grained statistics of logical object space usage #15199

osd: fine-grained statistics of logical object space usage #15199

Conversation

xiexingguo commented May 22, 2017 • edited

xiexingguo commented Jun 14, 2017

liewegas commented Jun 14, 2017

liewegas commented Jun 14, 2017

jdurgin commented Jun 14, 2017

liewegas commented Sep 1, 2017

xiexingguo commented Sep 24, 2017

xiexingguo commented Sep 25, 2017

liewegas Sep 25, 2017

Choose a reason for hiding this comment

liewegas Sep 25, 2017

Choose a reason for hiding this comment

liewegas Sep 25, 2017

Choose a reason for hiding this comment

liewegas Sep 25, 2017

Choose a reason for hiding this comment

xiexingguo commented Sep 26, 2017

xiexingguo commented Sep 27, 2017 • edited

xiexingguo commented Sep 27, 2017

xiexingguo commented Sep 28, 2017

xiexingguo commented Sep 30, 2017

xiexingguo commented Sep 30, 2017

xiexingguo commented May 22, 2017 •

edited

xiexingguo commented Sep 27, 2017 •

edited