DAOS-18607 placement: generate partial layout for rebuild and EC aggregation#17829
DAOS-18607 placement: generate partial layout for rebuild and EC aggregation#17829
Conversation
|
Errors are Unable to load ticket data |
EC aggregation doesn't require full layout computation, instead it only requires the layout up to the specified group. This patch adds an option and allows the placement function to complete layout computation early. It can significantly reduce computation time for EC aggregation. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
gnailzenh
force-pushed
the
liang/b_pl_partial
branch
from
4bd5fbf to
630fd47
Compare
| } | ||
|
|
||
| if (gen_mode != PRE_REBUILD || !(md->omd_flags & PL_FL_GRP_SPEC)) | ||
| md->omd_grp_spec = PL_GRP_MAX; /* full layout */ |
There was a problem hiding this comment.
This PR looks can work. Just a few concern that the generated layout actually with partial valid data compared to the definition -
73 struct pl_obj_layout {
74 > uint32_t> > ol_ver;
75 > uint32_t> > ol_grp_size;
76 > uint32_t> > ol_grp_nr;
77 > uint32_t> > ol_nr;
78 > /* number of peer targets, this field is added for reint/extension/drain placement */
79 > unsigned int ol_shard_peers;
80 > struct pl_obj_shard> *ol_shards;
81 };
seems 2 options -
- set the ol_nr, ol_grp_nr as actual value, or
- set ol_nr, ol_grp_nr same as current code (pl_obj_layout_alloc(jmop->jmop_grp_size, jmop->jmop_grp_nr) above), but assert it will not exceed valid range when using it in srv_ec_aggregate.c?
And seems for obj_reclaim() it can also use similar method?
Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
|
|
||
| if (md->omd_grp_spec != PL_GRP_MAX) { | ||
| /* may have generated the full layout, but ignore other parts */ | ||
| layout->ol_grp_nr = md->omd_grp_spec + 1; |
There was a problem hiding this comment.
md->omd_grp_spec may be set as a value > jmop->jmop_grp_nr - 1,
seems that either check that case and return failure,
or here set "layout->ol_grp_nr = min(md->omd_grp_spec + 1, jmop->jmop_grp_nr)"?
There was a problem hiding this comment.
probably can change it in a following PR, if it is correct.
There was a problem hiding this comment.
yes, would be nice to add this check in the future.
I will add it while applying this to dsc_obj_open()
|
|
||
| if (md->omd_grp_spec != PL_GRP_MAX) { | ||
| /* may have generated the full layout, but ignore other parts */ | ||
| layout->ol_grp_nr = md->omd_grp_spec + 1; |
There was a problem hiding this comment.
probably can change it in a following PR, if it is correct.
Rebuild and EC aggregation do not require the full object layout; instead, they only need the localities of the shards within the same redundancy group as the current shard. This patch stops the layout computation after generating the layout for the requested redundancy group, saving a significant amount of computation resources. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
gnailzenh
changed the title
Rebuild and EC aggregation do not require the full object layout; instead, they only need the localities of the shards within the same redundancy group as the current shard. This patch stops the layout computation after generating the layout for the requested redundancy group, saving a significant amount of computation resources. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
…egation (#17829) (#17896) EC aggregation doesn't require full layout computation, instead it only requires the layout up to the specified group. This patch adds an option and allows the placement function to complete layout computation early. It can significantly reduce computation time for EC aggregation. Signed-off-by: Liang Zhen <gnailzenh@gmail.com> * DAOS: rebuild scan/reclaim also uses partial layout Signed-off-by: Liang Zhen <gnailzenh@gmail.com> --------- Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
…egation (#17829) (#17879) EC aggregation doesn't require full layout computation, instead it only requires the layout up to the specified group. This patch adds an option and allows the placement function to complete layout computation early. It can significantly reduce computation time for EC aggregation. Signed-off-by: Liang Zhen <gnailzenh@gmail.com>
3 participants
Rebuild and EC aggregation do not require the full object layout;
instead, they only need the localities of the shards within the
same redundancy group as the current shard.
This patch stops the layout computation after generating the layout
for the requested redundancy group, saving a significant amount of
computation resources.
Steps for the author:
After all prior steps are complete: