New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mark object store bucket as full and move new files into new bucket #22039
Comments
My idea would be to save "ranges" of fileids in a table for each multibucket storage, to split fileids into further buckets With a table looking something like
meaning that files storaged within the multibucket storage "bucket_1" with fileids between 1 and 100 will be stored in bucket "bucket_1", with fileids after 101 in "bucket_1a". Since the fileids are already spread over multibucket this means that there is an overlap between the ranges for different "storage buckets", and the bucket "bucket_1" will have less files in them then their range might suggest. When the storage needs to call the object storage, it would find the range the fileid belongs in to find the final bucket name. When a new object is written, the number of objects the the current, final range can be queries from the filecache and if the configured limit has been exceeded, a new range is created. |
Problem
Things to keep in mind
@rullzer @kesselb @icewind1991 Feedback and ideas are welcome.
The text was updated successfully, but these errors were encountered: