Blocksconvert cleaner (#3283)

* Extracted plan_processor_service.go from builder.go.

To be used by cleaner, which is also processing a plan.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Comments.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Moved fetcher back, since cleaner doesn't use it.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Added new "cleaner" target to blocksconvert.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Remove forgotten gauge from builder.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Register cleaner flags.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Set userID into context for store methods to work.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Parse 'finished' status without block ID.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Cleaner now removes entries / chunks directly, without using store.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Ignore object not found error when deleting chunks.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Don't delete chunks starting outside of the plan range.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Delete index entries first, chunks second.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Delete index entries first, chunks second.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Improve comment.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Count number of deleted index entries.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Use correct From/Through when getting chunk write entries.
Remove label entries as well.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Fix test.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Make lint happy.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Make lint happy.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Make lint happy again.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* Moved additional config options to NewService method.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

* CHANGELOG.md

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

.gitignore

@@ -19,5 +19,7 @@ cortex
 query-tee
 test-exporter
 
+!tools/blocksconvert
+
 # This is custom Makefile modification, if it exists.
 Makefile.local

CHANGELOG.md

@@ -58,6 +58,7 @@
 * [ENHANCEMENT] Shuffle sharding: improved shuffle sharding in the write path. Shuffle sharding now should be explicitly enabled via `-distributor.sharding-strategy` CLI flag (or its respective YAML config option) and guarantees stability, consistency, shuffling and balanced zone-awareness properties. #3090 #3214
 * [ENHANCEMENT] Ingester: added new metric `cortex_ingester_active_series` to track active series more accurately. Also added options to control whether active series tracking is enabled (`-ingester.active-series-enabled`, defaults to false), and how often this metric is updated (`-ingester.active-series-update-period`) and max idle time for series to be considered inactive (`-ingester.active-series-idle-timeout`). #3153
 * [ENHANCEMENT] Blocksconvert – Builder: download plan file locally before processing it. #3209
+* [ENHANCEMENT] Blocksconvert – Cleaner: added new tool for deleting chunks data. #3283
 * [ENHANCEMENT] Store-gateway: added zone-aware replication support to blocks replication in the store-gateway. #3200
 * [ENHANCEMENT] Store-gateway: exported new metrics. #3231
   - `cortex_bucket_store_cached_series_fetch_duration_seconds`

cmd/blocksconvert/main.go

@@ -17,6 +17,7 @@ import (
 	"github.com/cortexproject/cortex/pkg/util/services"
 	"github.com/cortexproject/cortex/tools/blocksconvert"
 	"github.com/cortexproject/cortex/tools/blocksconvert/builder"
+	"github.com/cortexproject/cortex/tools/blocksconvert/cleaner"
 	"github.com/cortexproject/cortex/tools/blocksconvert/scanner"
 	"github.com/cortexproject/cortex/tools/blocksconvert/scheduler"
 )
@@ -29,6 +30,7 @@ type Config struct {
 	ScannerConfig   scanner.Config
 	BuilderConfig   builder.Config
 	SchedulerConfig scheduler.Config
+	CleanerConfig   cleaner.Config
 }
 
 func main() {
@@ -38,6 +40,7 @@ func main() {
 	cfg.ScannerConfig.RegisterFlags(flag.CommandLine)
 	cfg.BuilderConfig.RegisterFlags(flag.CommandLine)
 	cfg.SchedulerConfig.RegisterFlags(flag.CommandLine)
+	cfg.CleanerConfig.RegisterFlags(flag.CommandLine)
 	cfg.ServerConfig.RegisterFlags(flag.CommandLine)
 	flag.Parse()
 
@@ -62,6 +65,8 @@ func main() {
 		targetService, err = builder.NewBuilder(cfg.BuilderConfig, cfg.SharedConfig, util.Logger, registry)
 	case "scheduler":
 		targetService, err = scheduler.NewScheduler(cfg.SchedulerConfig, cfg.SharedConfig, util.Logger, registry, serv.HTTP, serv.GRPC)
+	case "cleaner":
+		targetService, err = cleaner.NewCleaner(cfg.CleanerConfig, cfg.SharedConfig, util.Logger, registry)
 	default:
 		err = fmt.Errorf("unknown target")
 	}