crush: weight_set and id remapping #14486

ghost · 2017-04-12T16:21:42Z

No description provided.

ghost · 2017-04-14T13:37:43Z

@liewegas here is a proposal for a minimalist integration of the weight_set in Ceph.

For buckets where the weight_set is not empty, the functions CrushWrapper::{insert_item,move_bucket,update_item,remove_item,remove_item_under} set the weight set and the target weight of new items to zero, and they do not adjust the weights of their parents. This allows the user to maintain the weights via the crushmap using the "choose_args" keyword::

// weight_set for pool 0
choose_args 0 {
  {
    bucket_id -5
    weight_set {
       position 0 weights 0x10000 0x20000 0x50000
       position 1 weights 0x30000 0x20000 0x50000
    }
    ids -20 -30 -25
  }

  {
    bucket_id -20
    weight_set {
       position 0 weights 0x10000 0x20000
       position 1 weights 0x30000 0x20000
    }
    ids -45 -23
  }
}

Whenever a choose_args is added to CrushWrapper (by compiling or decoding the crushmap), the buckets for which there exists at least one non empty weight_set are marked as such so that CrushWrapper::{insert_item,move_bucket,update_item,remove_item,remove_item_under} know to treat them differently.

The CrushWrapper::do_rule function takes an additional argument which is used as an index to find the
corresponding choose_args and OSDMap::_pg_to_raw_osds uses the pool number for this argument so that there can be a different choose_args per pool.

I'm moderately happy about this proposal but that's what I have so far ;-)

liewegas · 2017-04-14T13:42:37Z

I like this. It's simple. Basically weight changes just don't have any effect until whatever created the weight_set in the first place goes and runs again. Where do you want to store the choose_args/weight_set? As part of the map proper at the end of CrushWrapper::{encode,decode}? Or as a separate type that the OSDMap manages/stores separately? Will it be decompiled/compiled as part of the crush map (with a syntax like that below)?

ghost · 2017-04-14T13:48:27Z

Where do you want to store the choose_args/weight_set? As part of the map proper at the end of CrushWrapper::{encode,decode}?

yes

Will it be decompiled/compiled as part of the crush map (with a syntax like that below)?

yes

What bothers me is that I can't quite imagine how it will evolve beyond this minimalist proposal. But it's probably a problem for later :-)

liewegas · 2017-04-14T13:59:43Z

Hmm, what happens if you change the structure of the tree? E.g., move a host to a different rack? The weight_set isn't expressive enough to reflect a different size of the bucket, so it has to remove it form the old location, but it'll be 0'ed out in the new location.

ghost · 2017-04-14T14:10:58Z

when adding/removing items in a bucket, each choose_args[-1-bucket_id] must also be modified to remove the corresponding item or add an item with weight zero. This ensure the items count for each choose_args[-1-bucket_id] matches the items count of the map->bucket[-1-bucket_id] in the crushmap. In other words they must be kept in sync at all times. This should not be too much of a problem since adding / removing items/buckets is done via the CrushWrapper API. Unless there is something I'm not seeing ?

liewegas · 2017-04-14T14:14:59Z

yeah, that makes sense. it just means that if you move a bucket between 2 buckets (or an osd to another bucket) that it will be effectively removed from the old location but not added to the new one (until someone recalculates the weight_set values).

ghost · 2017-04-14T14:33:47Z

it will be effectively removed from the old location but not added to the new one (until someone recalculates the weight_set values).

Yes. From a practical point of view this is not a useful behavior. Maybe it would be better to just forbid adding/removing items/buckets via the CrushWrapper API when there are weight_sets. The user has to add/remove items/buckets by decompiling/modifying/compiling the crushmap and is responsible for maintaining the weight_set accordingly.

ghost · 2017-04-15T16:43:39Z

The compiler part is implemented and the syntax is slightly simpler than what was originally proposed 8438556#diff-6c59ef5804a9ab71015e059395d84feeR107 . Please let me know if you see something odd ;-)

ghost · 2017-04-16T11:58:47Z

@liewegas updated the implementation with compile/decompile/encode/decode and a crushtool based test, moving on to disabling crush map updates via the CrushWrapper api when weight sets are present.

ghost · 2017-04-16T15:40:50Z

@liewegas I think it is ready for review :-)

ghost · 2017-04-16T20:22:04Z

teuthology-suite -k distro --priority 1000 --suite rados --subset $(expr $RANDOM % 200)/200 --email loic@dachary.org --ceph wip-multiweight --machine-type smithi

6/226 fail http://pulpito.ceph.com/loic-2017-04-16_20:21:30-rados-wip-multiweight-distro-basic-smithi/

Re-running the failed tests

fail http://pulpito.ceph.com/loic-2017-04-17_06:47:20-rados-wip-multiweight-distro-basic-smithi/

They consistently fail but are unrelated to this pull request

liewegas · 2017-04-17T20:28:58Z

src/crush/CrushWrapper.cc

@@ -1540,6 +1572,38 @@ void CrushWrapper::decode(bufferlist::iterator& blp)
 	class_rname[c.second] = c.first;
      ::decode(class_bucket, blp);
      cleanup_classes();
+


can you add a } if (!blp.end()) { here? i have the class code deployed on a test cluster.

liewegas · 2017-04-17T20:31:16Z

src/test/cli/crushtool/choose-args.crush

+      [ 1.000 2.000 5.000 ]
+      [ 3.000 2.000 5.000 ]
+    ]
+    ids -20 -30 -25


weight_set uses [ ] for the lists.. maybe ids should too?

ids [ -20 -30 -25 ]

liewegas

Minor nit on syntax, but otherwise looks good!

It is bugous and unsupported. Signed-off-by: Loic Dachary <loic@dachary.org>

Signed-off-by: Loic Dachary <loic@dachary.org>

bucket_straw2_choose needs to use weights that may be different from weight_items. For instance to compensate for an uneven distribution caused by a low number of values. Or to fix the probability biais introduced by conditional probabilities (see http://tracker.ceph.com/issues/15653 for more information). We introduce a weight_set for each straw2 bucket to set the desired weight for a given item at a given position. The weight of a given item when picking the first replica (first position) may be different from the weight the second replica (second position). For instance the weight matrix for a given bucket containing items 3, 7 and 13 could be as follows: position 0 position 1 item 3 0x10000 0x100000 item 7 0x40000 0x10000 item 13 0x40000 0x10000 When crush_do_rule picks the first of two replicas (position 0), item 7, 3 are four times more likely to be choosen by bucket_straw2_choose than item 13. When choosing the second replica (position 1), item 3 is ten times more likely to be choosen than item 7, 13. By default the weight_set of each bucket exactly matches the content of item_weights for each position to ensure backward compatibility. bucket_straw2_choose compares items by using their id. The same ids are also used to index buckets and they must be unique. For each item in a bucket an array of ids can be provided for placement purposes and they are used instead of the ids. If no replacement ids are provided, the legacy behavior is preserved. Signed-off-by: Loic Dachary <loic@dachary.org>

If there is no crush_choose_arg_map for a given pool (the default) a NULL pointer is given instead and crush_do_rule behavior remains unchanged. Signed-off-by: Loic Dachary <loic@dachary.org>

A map of crush_choose_arg_map is added to the crushmap text syntax. The key is an integer matching a pool number. Signed-off-by: Loic Dachary <loic@dachary.org>

Adding, removing or move items / buckets via the CrushWrapper API when choose_args is not empty is unlikely to produce the desired outcome. The caller should instead add, remove or move items / buckets in a decompiled crushmap, update the associated choose_arg and upload the new crushmap. Signed-off-by: Loic Dachary <loic@dachary.org>

Signed-off-by: Loic Dachary <loic@dachary.org>

ghost · 2017-04-18T07:50:42Z

@liewegas the ids syntax was changed here and tested here

The extra blp.end is here

Also added 57d4c8d and 18245ec fixing test leakage & valgrind complaints.

liewegas · 2017-04-18T13:39:25Z

Looks good!

Maybe we should deprecate the tree buckets more explicitly somehow so they can be removed.. maybe a bool CrushWrapper::has_tree_buckets(), and a health warning in luminous if they're in use to warn that support will be removed post-luminous?

ghost · 2017-04-18T14:08:29Z

good idea created http://tracker.ceph.com/issues/19654 to not forget about it.

ghost · 2017-04-18T14:09:30Z

teuthology-suite -k distro --priority 1000 --suite rados --subset $(expr $RANDOM % 200)/200 --email loic@dachary.org --ceph wip-multiweight --machine-type smithi

7/226 fail http://pulpito.ceph.com/loic-2017-04-18_13:14:34-rados-wip-multiweight-distro-basic-smithi/

Re-running the failed tests

running http://pulpito.ceph.com/loic-2017-04-19_07:16:50-rados-wip-multiweight-distro-basic-smithi

ghost added feature core labels Apr 12, 2017

tchaikov self-requested a review April 14, 2017 14:46

ghost changed the title ~~DNM: crush multiweight~~ crush: weight_set and id remapping Apr 16, 2017

ghost requested a review from liewegas April 16, 2017 15:37

liewegas reviewed Apr 17, 2017

View reviewed changes

liewegas approved these changes Apr 17, 2017

View reviewed changes

ldachary and others added 7 commits April 18, 2017 09:38

crush: do not use TREE in tests

57d4c8d

It is bugous and unsupported. Signed-off-by: Loic Dachary <loic@dachary.org>

crush: cleanup test memory leaks

18245ec

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: add per pool choose_args when calling do_rule

55fb91d

If there is no crush_choose_arg_map for a given pool (the default) a NULL pointer is given instead and crush_do_rule behavior remains unchanged. Signed-off-by: Loic Dachary <loic@dachary.org>

crush: compile/decompile crush_choose_arg_map

dbe36e0

A map of crush_choose_arg_map is added to the crushmap text syntax. The key is an integer matching a pool number. Signed-off-by: Loic Dachary <loic@dachary.org>

crush: implement CrushWrapper::dump(choose_args)

1b02fef

Signed-off-by: Loic Dachary <loic@dachary.org>

liewegas added the needs-qa label Apr 18, 2017

liewegas added the wip-sage2-testing label Apr 18, 2017

liewegas merged commit 4e8dd17 into ceph:master Apr 19, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crush: weight_set and id remapping #14486

crush: weight_set and id remapping #14486

ghost commented Apr 12, 2017

ghost commented Apr 14, 2017 •

edited by ghost

liewegas commented Apr 14, 2017 via email

ghost commented Apr 14, 2017

liewegas commented Apr 14, 2017 via email

ghost commented Apr 14, 2017

liewegas commented Apr 14, 2017 via email

ghost commented Apr 14, 2017

ghost commented Apr 15, 2017

ghost commented Apr 16, 2017

ghost commented Apr 16, 2017

ghost commented Apr 16, 2017 •

edited by ghost

liewegas Apr 17, 2017

liewegas Apr 17, 2017

liewegas left a comment

ghost commented Apr 18, 2017

liewegas commented Apr 18, 2017

ghost commented Apr 18, 2017

ghost commented Apr 18, 2017 •

edited by ghost

crush: weight_set and id remapping #14486

crush: weight_set and id remapping #14486

Conversation

ghost commented Apr 12, 2017

ghost commented Apr 14, 2017 • edited by ghost

liewegas commented Apr 14, 2017 via email

ghost commented Apr 14, 2017

liewegas commented Apr 14, 2017 via email

ghost commented Apr 14, 2017

liewegas commented Apr 14, 2017 via email

ghost commented Apr 14, 2017

ghost commented Apr 15, 2017

ghost commented Apr 16, 2017

ghost commented Apr 16, 2017

ghost commented Apr 16, 2017 • edited by ghost

liewegas Apr 17, 2017

Choose a reason for hiding this comment

liewegas Apr 17, 2017

Choose a reason for hiding this comment

liewegas left a comment

Choose a reason for hiding this comment

ghost commented Apr 18, 2017

liewegas commented Apr 18, 2017

ghost commented Apr 18, 2017

ghost commented Apr 18, 2017 • edited by ghost

ghost commented Apr 14, 2017 •

edited by ghost

ghost commented Apr 16, 2017 •

edited by ghost

ghost commented Apr 18, 2017 •

edited by ghost