Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improvements for boltdb-shipper compactor #2640

Conversation

sandeepsukhani
Copy link
Contributor

What this PR does / why we need it:
Does the following improvements to boltdb-shipper compactor:

  1. Run the compaction process on startup instead of waiting for the first tick.
  2. Make the interval at which compaction runs configurable and reduce the default interval to 2 hours because lesser the files faster the query response.
  3. Run the compactor service when using loki with boltdb-shipper in a single binary mode without clustering.

1. Run the compaction process on startup instead of waiting for the first tick.
2. Run the compactor service when using loki with boltdb-shipper in single binary mode without clustering.
@codecov-commenter
Copy link

Codecov Report

Merging #2640 into master will decrease coverage by 1.47%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2640      +/-   ##
==========================================
- Coverage   62.87%   61.40%   -1.48%     
==========================================
  Files         170      170              
  Lines       15051    13187    -1864     
==========================================
- Hits         9464     8097    -1367     
+ Misses       4827     4349     -478     
+ Partials      760      741      -19     
Impacted Files Coverage Δ
pkg/loki/loki.go 0.00% <0.00%> (ø)
pkg/storage/stores/shipper/compactor/compactor.go 0.00% <0.00%> (ø)
pkg/logql/step_evaluator.go 57.14% <0.00%> (-9.53%) ⬇️
pkg/logql/marshal/labels.go 66.66% <0.00%> (-8.34%) ⬇️
pkg/logproto/timestamp.go 40.00% <0.00%> (-6.81%) ⬇️
pkg/logentry/stages/labeldrop.go 53.33% <0.00%> (-6.67%) ⬇️
pkg/chunkenc/encoding_helpers.go 59.25% <0.00%> (-6.37%) ⬇️
pkg/logql/marshal/tail.go 66.66% <0.00%> (-6.07%) ⬇️
pkg/promtail/api/types.go 14.28% <0.00%> (-5.72%) ⬇️
pkg/logql/vector.go 81.81% <0.00%> (-5.69%) ⬇️
... and 162 more

Copy link
Collaborator

@slim-bean slim-bean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sandeepsukhani sandeepsukhani merged commit 5b361a0 into grafana:master Sep 18, 2020
@oleksandrsemak
Copy link

Hey @sandeepsukhani @slim-bean is it right example of compactor? It has to be in config: ?
Image: grafana/loki:1.6.0
field compactor not found in type main.Config

@slim-bean
Copy link
Collaborator

The compactor is not available in a release yet, only in master builds. Hold tight though, a release is coming soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants