Create guide doc for partition compaction #6512
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CharlieTLe
reviewed
Jan 22, 2025
* Purge expired postings cache items due inactivity Signed-off-by: alanprot <alanprot@gmail.com> * Fix comments Signed-off-by: alanprot <alanprot@gmail.com> --------- Signed-off-by: alanprot <alanprot@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
* Update thanos to 4ba0ba403896 Signed-off-by: Daniel Sabsay <sabsay@adobe.com> * run go mod vendor Signed-off-by: Daniel Sabsay <sabsay@adobe.com> --------- Signed-off-by: Daniel Sabsay <sabsay@adobe.com> Co-authored-by: Daniel Sabsay <sabsay@adobe.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…cortexproject#6505) Bumps the actions-dependencies group with 2 updates in the / directory: [actions/upload-artifact](https://github.com/actions/upload-artifact) and [github/codeql-action](https://github.com/github/codeql-action). Updates `actions/upload-artifact` from 4.5.0 to 4.6.0 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@6f51ac0...65c4c4a) Updates `github/codeql-action` from 3.28.0 to 3.28.1 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@48ab28a...b6a472f) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-dependencies - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
* Implement partition compaction grouper Signed-off-by: Alex Le <leqiyue@amazon.com> * fix comment Signed-off-by: Alex Le <leqiyue@amazon.com> * replace level 1 compaction limits with ingestion replication factor Signed-off-by: Alex Le <leqiyue@amazon.com> * fix doc Signed-off-by: Alex Le <leqiyue@amazon.com> * update compaction_visit_marker_timeout default value Signed-off-by: Alex Le <leqiyue@amazon.com> * update default value for compactor_partition_index_size_limit_in_bytes Signed-off-by: Alex Le <leqiyue@amazon.com> * refactor code Signed-off-by: Alex Le <leqiyue@amazon.com> * address comments and refactor Signed-off-by: Alex Le <leqiyue@amazon.com> * address comment Signed-off-by: Alex Le <leqiyue@amazon.com> * address comment Signed-off-by: Alex Le <leqiyue@amazon.com> * update config name Signed-off-by: Alex Le <leqiyue@amazon.com> * Implement partition compaction planner Signed-off-by: Alex Le <leqiyue@amazon.com> * fix after rebase Signed-off-by: Alex Le <leqiyue@amazon.com> * addressed comments Signed-off-by: Alex Le <leqiyue@amazon.com> * updated doc and refactored metric Signed-off-by: Alex Le <leqiyue@amazon.com> * fix test Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…les (cortexproject#6507) * Add cleaner logic to clean partition compaction blocks and related files Signed-off-by: Alex Le <leqiyue@amazon.com> * refactored metrics Signed-off-by: Alex Le <leqiyue@amazon.com> * refactor Signed-off-by: Alex Le <leqiyue@amazon.com> * update logs Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com>
Maintainers would like an additional week to get the partition compactor changes in before the first release candidate for 1.19. Signed-off-by: Charlie Le <charlie_le@apple.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…rtexproject#6514) Signed-off-by: yeya24 <benye@amazon.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…ortexproject#6518) Signed-off-by: alanprot <alanprot@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
cortexproject#6519) Signed-off-by: Alex Le <leqiyue@amazon.com>
…ject#6522) Signed-off-by: Alex Le <leqiyue@amazon.com>
…oject#6523) * expose grpc client connect timeout config Signed-off-by: yeya24 <benye@amazon.com> * changelog Signed-off-by: yeya24 <benye@amazon.com> --------- Signed-off-by: yeya24 <benye@amazon.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…#6510) * Implemented partition compaction end to end with custom compaction lifecycle Signed-off-by: Alex Le <leqiyue@amazon.com> * removed unused variable Signed-off-by: Alex Le <leqiyue@amazon.com> * tweak test Signed-off-by: Alex Le <leqiyue@amazon.com> * tweak test Signed-off-by: Alex Le <leqiyue@amazon.com> * refactor according to comments Signed-off-by: Alex Le <leqiyue@amazon.com> * tweak test Signed-off-by: Alex Le <leqiyue@amazon.com> * check context error inside sharded posting Signed-off-by: Alex Le <leqiyue@amazon.com> * fix lint Signed-off-by: Alex Le <leqiyue@amazon.com> * fix integration test for memberlist Signed-off-by: Alex Le <leqiyue@amazon.com> * make compactor initial wait cancellable Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com>
* Test for nil on expire expanded postings Signed-off-by: alanprot <alanprot@gmail.com> * stopping ingester Signed-off-by: alanprot <alanprot@gmail.com> * refactor the test to not timeout Signed-off-by: alanprot <alanprot@gmail.com> --------- Signed-off-by: alanprot <alanprot@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
* log when a request starts running in querier Signed-off-by: Ahmed Hassan <afayekhassan@gmail.com> * log when a request starts running in querier for frontend processor Signed-off-by: Ahmed Hassan <afayekhassan@gmail.com> --------- Signed-off-by: Ahmed Hassan <afayekhassan@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…6508) Signed-off-by: Friedrich Gonzalez <friedrichg@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
https://github.com/cortexproject/cortex/blob/master/RELEASE.md#show-that-a-release-is-in-progress Signed-off-by: Charlie Le <charlie_le@apple.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Charlie Le <charlie_le@apple.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
…roup (cortexproject#6547) * Fixed blocksGroupWithPartition unable to reuse functions from blocksGroup Signed-off-by: Alex Le <leqiyue@amazon.com> * update tests Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: alanprot <alanprot@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
* Create partial_data Signed-off-by: Justin Jung <jungjust@amazon.com> * Fix lazyquery so that warning message is returned Signed-off-by: Justin Jung <jungjust@amazon.com> * Add QueryPartialData limit Signed-off-by: Justin Jung <jungjust@amazon.com> * Fix broken mock Signed-off-by: Justin Jung <jungjust@amazon.com> * Make response with warnings to be not cached Signed-off-by: Justin Jung <jungjust@amazon.com> * Updated streamingSelect in distributor_queryable Signed-off-by: Justin Jung <jungjust@amazon.com> * Update query.go Signed-off-by: Justin Jung <jungjust@amazon.com> * Update replication_set Signed-off-by: Justin Jung <jungjust@amazon.com> * Lint Signed-off-by: Justin Jung <jungjust@amazon.com> * Lint again Signed-off-by: Justin Jung <jungjust@amazon.com> * Generated doc Signed-off-by: Justin Jung <jungjust@amazon.com> * Changelog Signed-off-by: Justin Jung <jungjust@amazon.com> * Update config description Signed-off-by: Justin Jung <jungjust@amazon.com> * Do not remove warnings from seriesSet Signed-off-by: Justin Jung <jungjust@amazon.com> * Avoid cache only if the warning message contains partial data error Signed-off-by: Justin Jung <jungjust@amazon.com> * Remove context usage for partial data Signed-off-by: Justin Jung <jungjust@amazon.com> * Refactor how partial data info is passed + apply to series and label methods as well Signed-off-by: Justin Jung <jungjust@amazon.com> * Lint + fix tests Signed-off-by: Justin Jung <jungjust@amazon.com> * Fix build Signed-off-by: Justin Jung <jungjust@amazon.com> * Create separate config for ruler partial data Signed-off-by: Justin Jung <jungjust@amazon.com> * Genereta doc Signed-off-by: Justin Jung <jungjust@amazon.com> * Add more tests Signed-off-by: Justin Jung <jungjust@amazon.com> * Change error Signed-off-by: Justin Jung <jungjust@amazon.com> * Fix test Signed-off-by: Justin Jung <jungjust@amazon.com> * Update changelog Signed-off-by: Justin Jung <jungjust@amazon.com> * Update changelog Signed-off-by: Justin Jung <jungjust@amazon.com> * Nit Signed-off-by: Justin Jung <jungjust@amazon.com> * Nit Signed-off-by: Justin Jung <jungjust@amazon.com> --------- Signed-off-by: Justin Jung <jungjust@amazon.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
* add dynamodb kv with timeout enforced Signed-off-by: yeya24 <benye@amazon.com> * add tests Signed-off-by: yeya24 <benye@amazon.com> * docs Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: yeya24 <benye@amazon.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…cortexproject#6564) Bumps the actions-dependencies group with 2 updates in the / directory: [github/codeql-action](https://github.com/github/codeql-action) and [actions/setup-go](https://github.com/actions/setup-go). Updates `github/codeql-action` from 3.28.1 to 3.28.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@b6a472f...6e54559) Updates `actions/setup-go` from 5.2.0 to 5.3.0 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@3041bf5...f111f33) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-dependencies - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
…in the future" (cortexproject#6562) * improve fuzz test for expanded postings cache Signed-off-by: alanprot <alanprot@gmail.com> * create more tests on the expanded postings cache Signed-off-by: alanprot <alanprot@gmail.com> * adding get series call on the test Signed-off-by: alanprot <alanprot@gmail.com> * no use CachedBlockChunkQuerier when query time range is completely after the last sample added in the head Signed-off-by: alanprot <alanprot@gmail.com> * adding comments Signed-off-by: alanprot <alanprot@gmail.com> * increase the number of fuzz test from 100 to 300 Signed-off-by: alanprot <alanprot@gmail.com> * add get series fuzzy testing Signed-off-by: alanprot <alanprot@gmail.com> --------- Signed-off-by: alanprot <alanprot@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
* Filter readOnly ingesters when sharding Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Extend shard on READONLY Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Remove old code Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * Fix test Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> * update changelog Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> --------- Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
Co-authored-by: Charlie Le <charlie_le@apple.com> Signed-off-by: Alex Le <emoc1989@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
alexqyle
force-pushed
the
partition-compaction-doc
branch
from
65285a1 to
a7466d1
Compare
Signed-off-by: Alex Le <leqiyue@amazon.com>
|
Are we good to merge this one? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Create guide doc for partition compaction and update CHANGELOG to include partition compaction
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]