Support EC in container storage policies #345
Conversation
cthulhu-rider
force-pushed
the
feature/container-policy-ec
branch
3 times, most recently
from
0bb2bf1 to
56eef59
Compare
cthulhu-rider
requested review from
carpawell and
roman-khimov
as code owners
cthulhu-rider
force-pushed
the
feature/container-policy-ec
branch
from
56eef59 to
2b3be32
Compare
Exec `make doc`. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider
force-pushed
the
feature/container-policy-ec
branch
5 times, most recently
from
ffbf6c7 to
730a83a
Compare
| // | ||
| // For each original object, the payload is split into `data_part_num` data | ||
| // and `parity_part_num` parity parts. Each part is the same size. Data parts | ||
| // contain the original payload. If its length is not divisible by |
There was a problem hiding this comment.
Data parts contain the original payload
is it strictly correct? isn't it transformed?
netmap/types.proto
Outdated
| // For each payload part, a part object is created. Original object's ID, | ||
| // signature and header is written in `header.split.parent`, | ||
| // `header.split.parent_signature` and `header.split.parent_header` fields | ||
| // correspondingly. Part index is written in the `__NEOFS__EC_PART_IDX` |
There was a problem hiding this comment.
| // correspondingly. Part index is written in the `__NEOFS__EC_PART_IDX` | |
| // correspondingly. Part index is written in the `__NEOFS__EC_PART_IDX` |
object/service.proto
Outdated
| // specified as follows: | ||
| // - `body.address` is an address of the parent; | ||
| // - `meta_header.x_headers` includes `__NEOFS__EC_RULE_IDX` and | ||
| // `__NEOFS__EC_PART_IDX` by object attribute format. Rule index MUST NOT |
There was a problem hiding this comment.
| // `__NEOFS__EC_PART_IDX` by object attribute format. Rule index MUST NOT | |
| // `__NEOFS__EC_PART_IDX` by object attribute format. Rule index MUST NOT |
object/service.proto
Outdated
| // specified as follows: | ||
| // - `body.address` is an address of the parent; | ||
| // - `meta_header.x_headers` includes `__NEOFS__EC_RULE_IDX` and | ||
| // `__NEOFS__EC_PART_IDX` by object attribute format. Rule index MUST NOT |
There was a problem hiding this comment.
| // `__NEOFS__EC_PART_IDX` by object attribute format. Rule index MUST NOT | |
| // `__NEOFS__EC_PART_IDX` by object attribute format. Rule index MUST NOT |
Refs nspcc-dev/neofs-node#526. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider
force-pushed
the
feature/container-policy-ec
branch
from
730a83a to
3479c5f
Compare
Required to implement decoding of the parent object from its EC parts. Refs nspcc-dev/neofs-node#3422, nspcc-dev/neofs-node#3423. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider
force-pushed
the
feature/container-policy-ec
branch
from
3479c5f to
b924e9b
Compare
| // Name of the linked selector | ||
| string selector = 3 [json_name = "selector"]; | ||
| } | ||
| // Erasure coding rules. Limited to 256 items. |
There was a problem hiding this comment.
Limit to 1? Yeah, I know it's an array and theoretically many things are possible. But I'm not sure what will happen with "EC 3/1 EC 12/4 EC 5/3 EC 4/2" policy. Maybe it'll work, but I doubt it'll be usable. Also it's easier to extend things in future rather than constrain. So maybe 2-4, but no more.
There was a problem hiding this comment.
i doubt various encoding parameters will be in demand too. But multiple settings may be required to bind to different selectors. For example, they can define disjoint subnets for placement
picked 256 from replicas
in total, limit by 4 seems good to me
| string selector = 3 [json_name = "selector"]; | ||
| } | ||
| // Erasure coding rules. Limited to 256 items. | ||
| repeated ECRule ec_rules = 6 [json_name = "ecRules"]; |
There was a problem hiding this comment.
Also it's completely separated from replicas which makes these rules kinda alternative. It can be an implementation detail, but the fact that they're not on the same level can be confusing.
There was a problem hiding this comment.
they are at the same level of policy, just diff fields. I thought that we can add parity field to Replica. Zero stands for old REP rule, non-zero - for EC one. But the storage scheme is so different, so i decided not to merge them. And IINM there wont be any gain in size
| // UTF-8 string topic ID that is used for object notification. | ||
| // DEPRECATED: attribute ignored by servers. | ||
| // * __NEOFS__EC_RULE_IDX \ | ||
| // Index of EC rule in container's `PlacementPolicy.ec_rules` according to |
There was a problem hiding this comment.
Like this index. For me in "REP 3 EC 3/1" the EC rule index is 1. But in fact it's 0.
There was a problem hiding this comment.
that's why it is EC_RULE_IDX, not RULE_IDX
| // specified as follows: | ||
| // - `body.address` is an address of the parent; | ||
| // - `meta_header.x_headers` includes `__NEOFS__EC_RULE_IDX` and | ||
| // `__NEOFS__EC_PART_IDX` by object attribute format. Rule index MUST NOT |
There was a problem hiding this comment.
What if netmap change happens and we're requesting part 0/4 from node that stores 0/1?
No description provided.