Skip to content

Conversation

yeya24
Copy link
Contributor

@yeya24 yeya24 commented Aug 10, 2023

What this PR does:

We added Index Stats info to the meta.json file and use that to estimate series and chunk size. However, when store gateway uses bucket index, it won't actually download meta.json file from object store. It will generate meta.json file from the information provided by bucket index.

This pr adds additional fields series_max_size and chunk_max_size to bucket index per block to help address this issue. The change is also backward compatible from what I tested.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Copy link
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yeya24
Copy link
Contributor Author

yeya24 commented Aug 10, 2023

The issue of current implementation is that, for old blocks in the bucket index, we don't update series_max_size and chunk_max_size fields because bucket index updater just copies what we have from the old bucket index.
Only those new blocks that don't appear in bucket index will be added with those fields

@alanprot
Copy link
Member

The issue of current implementation is that, for old blocks in the bucket index, we don't update series_max_size and chunk_max_size fields because bucket index updater just copies what we have from the old bucket index.
Only those new blocks that don't appear in bucket index will be added with those fields

This should be fine for this PR?

@yeya24
Copy link
Contributor Author

yeya24 commented Aug 15, 2023

This should be fine for this PR?

Yeah I think it should be fine. It is another story of migration, which can be done separately.

@yeya24 yeya24 merged commit 734cf0b into cortexproject:master Aug 15, 2023
@yeya24 yeya24 deleted the add-index-stats-metadata branch August 15, 2023 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants