Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Blobstor may "lose" objects on config changes #1707

Closed
carpawell opened this issue Aug 18, 2022 · 2 comments
Closed

Blobstor may "lose" objects on config changes #1707

carpawell opened this issue Aug 18, 2022 · 2 comments
Assignees
@cthulhu-rider
Labels
bug Something isn't working discussion Open discussion of some problem neofs-storage Storage node application issues U0 Needs to be resolved immediately
Milestone
v0.33.0

Comments

@carpawell
Copy link
Member

carpawell commented Aug 18, 2022 •
edited
Loading

Context

blobovnicza stores objects in its buckets according to object sizes: it chooses the minimum number (the power of 2) that satisfies the object limit. It creates buckets for objects starting from 32<<10 = 32kb, e.g. if the size limit is 100kb, tree buckets gonna be created: [0:32kb), [32kb:64kb), [64kb:128kb). That buckets will be used for storing and searching objects. Searching will try to iterate buckets according to the current configuration of the blobovnicza.

Problem

Consider situation:

  1. A Node was configured to store objects with size<=N, N%2==0
  2. It gets objects with N/2 < size <= N and stores them in the last of the supported buckets
  3. The node goes offline
  4. Its config changes and now it gonna store "small" objects only if their size<=N/2
  5. It gets Get with the object from 2. and could not return it since it does not know about the last bucket it has

Problems:

  1. Current implementation could not return object
  2. Object is stored but does not satisfy object placement (seems like such objects should be migrated to the FSTree with corresponding metabase changes)

Ideas

  1. Could be solved by just iterating all the buckets it has (but somehow logic with the defined bucket names were implemented, mb it has some nuances, /cc @cthulhu-rider)
  2. We could migrate or not migrate objects. If some objects are expected to be stored in a bolt base it could be unexpected to find them in FSTree after the config change (/cc @realloc)
@carpawell carpawell added bug Something isn't working discussion Open discussion of some problem neofs-storage Storage node application issues labels Aug 18, 2022
@alexchetaev alexchetaev added U0 Needs to be resolved immediately 2022Q4 labels Aug 19, 2022
@carpawell carpawell mentioned this issue Aug 26, 2022
Merged
7 tasks
@cthulhu-rider cthulhu-rider self-assigned this Sep 6, 2022
@cthulhu-rider
Copy link
Contributor

cthulhu-rider commented Sep 6, 2022 •
edited
Loading

1. No problem, we can freely iterate over all buckets. It'd be even faster since we don't know the object size.
2. Currently we don't have to migrate objects which are big in current run and were small in the previous one. We read the object from the particular BLOB storage according to the record in metabase called StorageID. If StorageID is missing - we look up all the BLOBs. With this approach any migration adds overhead only until we approve the rigidity of the object division by size.

cthulhu-rider pushed a commit to cthulhu-rider/neofs-node that referenced this issue Sep 7, 2022
[nspcc-dev#1707] blobovnicza/get: Iterate over all buckets regardless…
dc11d60 
… of config

Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.

Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
@cthulhu-rider cthulhu-rider mentioned this issue Sep 7, 2022
Merged
cthulhu-rider pushed a commit that referenced this issue Sep 8, 2022
@cthulhu-rider
[#1707] blobovnicza/get: Iterate over all buckets regardless of config
8e8265d 
Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.

Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
@fyrchik fyrchik mentioned this issue Sep 8, 2022
Open
7 tasks
@cthulhu-rider
Copy link
Contributor

aprasolova pushed a commit to aprasolova/neofs-node that referenced this issue Oct 19, 2022
@aprasolova
[nspcc-dev#1707] blobovnicza/get: Iterate over all buckets regardless…
79c34d0 
… of config

Re-configuration of Blobovnicza's object size limit must not affect
already stored objects. In previous implementation `Blobovnicza` didn't
see objects stored in buckets which became too big after size limit
re-configuration. For example, lets consider 1st configuration with 64KB
size limit for stored objects. Lets assume that we stored object of 64KB
size, and re-configured `Blobovnicza` with 32KB limit. After reboot
object should be still available, but actually it isn't. This is caused
by `Get` operation algorithm which iterates over configured size ranges
only, and doesn't process any other existing size bucket. By the way,
increasing of the object size limit didn't lead to the problem even in
previous implementation.

Make `Blobovnicza.Get` method to iterate over all size buckets
regardless of current configuration. This covers the described scenario.

Signed-off-by: Leonard Lyubich <leonard@nspcc.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cthulhu-rider cthulhu-rider

Labels
bug Something isn't working discussion Open discussion of some problem neofs-storage Storage node application issues U0 Needs to be resolved immediately
Projects
None yet
Milestone
v0.33.0
Development

No branches or pull requests
4 participants
@fyrchik @carpawell @alexchetaev @cthulhu-rider