Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mixin: Add block deletion panels for compactor dashboard #3380

Merged

Conversation

Sayrus
Copy link
Contributor

@Sayrus Sayrus commented Oct 29, 2020

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

This PR adds a new row to the Thanos Compact Dashboard. This row contains:

  • The rate at which blocks are deleted from S3: This is useful as it avoids reading logs or manually querying the metric. In 0.15, the deletion is done after downsampling so it allows us to monitor whether it is actually called or not. In 0.16, this process is done concurrently thanks to cmd: compact: clean partial / marked blocks concurrently #3115 . This graph then allows us to monitor if the async context is doing its job.
  • The error rate when trying to delete from S3
  • The rate at which blocks are marked for deletion, not only from GC but also from retention policy

Verification

Our compactor isn't deleting anything at the moment so the graphs are empty.

Notes
Metrics will need to be renamed following #3309 as they use the thanos_compact_* format.

@Sayrus Sayrus force-pushed the add-deletion-metrics-thanos-compact-mixins branch from 8a9ec2d to df18581 Compare October 29, 2020 14:45
@kakkoyun kakkoyun changed the title compact: add deletion metrics to mixins mixin: add deletion metrics for compactor Oct 30, 2020
Copy link
Member

@kakkoyun kakkoyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. I can see the use case.
Let's wait for the renaming PR to be merged.

CHANGELOG.md Outdated Show resolved Hide resolved
@Sayrus Sayrus force-pushed the add-deletion-metrics-thanos-compact-mixins branch from df18581 to 0dba1bb Compare October 30, 2020 10:31
@metalmatze
Copy link
Member

Renaming PRs should be merged.

@Sayrus Sayrus force-pushed the add-deletion-metrics-thanos-compact-mixins branch from 0dba1bb to c4865da Compare November 13, 2020 16:43
@Sayrus
Copy link
Contributor Author

Sayrus commented Nov 13, 2020

Updated for new names 👌

@kakkoyun
Copy link
Member

kakkoyun commented Jan 4, 2021

@Sayrus Looks good so far. You need to run make examples and commit changes to make CI happy.

Signed-off-by: Mathis Raguin <mathis@cri.epita.fr>
@Sayrus Sayrus force-pushed the add-deletion-metrics-thanos-compact-mixins branch from c4865da to aed289d Compare January 5, 2021 09:03
@kakkoyun kakkoyun changed the title mixin: add deletion metrics for compactor mixin: Add block deletion panels for compactor dashboard Jan 5, 2021
CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Member

@kakkoyun kakkoyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Let's be super clear on what we actually add in the changelog.

Then we can merge.

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
@kakkoyun kakkoyun force-pushed the add-deletion-metrics-thanos-compact-mixins branch from 2d53172 to 044241f Compare January 5, 2021 10:49
@kakkoyun kakkoyun merged commit 0aa0711 into thanos-io:master Jan 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants