Promtool: Add support for compaction analysis #8940

fpetkovski · 2021-06-15T09:56:23Z

This commit extends the promtool tsdb analyze command to help
troubleshoot high Prometheus disk usage. The command will now plot a distribution of
how full chunks are relative to the maximum capacity of 120 samples per chunk.

Here is a sample output:

$ promtool tsdb analyze

...

Fullness: Amount of chunks
     10%: #############
     20%: ######
     30%: 
     40%: 
     50%: #####################################
     60%: ######
     70%: 
     80%: 
     90%: 
    100%: ###################################

fpetkovski · 2021-06-15T09:57:45Z

cmd/promtool/main.go

@@ -142,6 +142,10 @@ func main() {
 	dumpMinTime := tsdbDumpCmd.Flag("min-time", "Minimum timestamp to dump.").Default(strconv.FormatInt(math.MinInt64, 10)).Int64()
 	dumpMaxTime := tsdbDumpCmd.Flag("max-time", "Maximum timestamp to dump.").Default(strconv.FormatInt(math.MaxInt64, 10)).Int64()

+	tsdbAnalyzeCompaction := tsdbCmd.Command("analyze-compaction", "Plot a distribution of how full TSDB chunks are, relative to the maximum capacity of 120 samples.")
+	analyzeCompactionDB := tsdbAnalyzeCompaction.Arg("db path", "Database path (default is "+defaultDBPath+").").Default(defaultDBPath).String()
+	analyzeCompactionBlockID := tsdbAnalyzeCompaction.Arg("block ID", "The TSDB block to analyze").Default("").String()


We could also consider adding this analysis to the existing tsdb analyze command.

IMHO it would be fit in the scope of tsdb analyze but lets see what others think :)

@dgl do you have any thoughts on this one?

Yeah, makes sense to include it to me.

+1 can we merge to tsdb analyze then? 🤗

bwplotka · 2021-06-29T13:14:54Z

PTAL @codesome @cstyan , I will look in my free time too! 🙈

bwplotka

Nice, LGTM when added to tsdb analyze (:

bwplotka · 2021-07-01T18:08:08Z

cmd/promtool/main.go

@@ -142,6 +142,10 @@ func main() {
 	dumpMinTime := tsdbDumpCmd.Flag("min-time", "Minimum timestamp to dump.").Default(strconv.FormatInt(math.MinInt64, 10)).Int64()
 	dumpMaxTime := tsdbDumpCmd.Flag("max-time", "Maximum timestamp to dump.").Default(strconv.FormatInt(math.MaxInt64, 10)).Int64()

+	tsdbAnalyzeCompaction := tsdbCmd.Command("analyze-compaction", "Plot a distribution of how full TSDB chunks are, relative to the maximum capacity of 120 samples.")
+	analyzeCompactionDB := tsdbAnalyzeCompaction.Arg("db path", "Database path (default is "+defaultDBPath+").").Default(defaultDBPath).String()
+	analyzeCompactionBlockID := tsdbAnalyzeCompaction.Arg("block ID", "The TSDB block to analyze").Default("").String()


+1 can we merge to tsdb analyze then? 🤗

fpetkovski · 2021-07-02T06:37:43Z

@bwplotka @dgl the compaction analysis should now be merged into the tsdb analyze command. Changed the commit message and PR description to reflect this change.

bwplotka

Nice! LGTM, just small tiny nits.

Plus: Can we mention new functionality in tsdb analysis command help?

bwplotka · 2021-07-02T08:40:39Z

cmd/promtool/tsdb.go

+
+	fmt.Printf("\nCompaction analysis:\n")
+	fmt.Println("Fullness: Amount of chunks")
+	// Normalize absolute counts to percentages and print them out


Suggested change

// Normalize absolute counts to percentages and print them out

// Normalize absolute counts to percentages and print them out.

bwplotka · 2021-07-02T08:40:46Z

cmd/promtool/tsdb.go

+				return err
+			}
+			chunkSize := math.Min(float64(chk.NumSamples()), maxSamplesPerChunk)
+			// Calculate the bucket for the chunk and increment it in the histogram


Suggested change

// Calculate the bucket for the chunk and increment it in the histogram

// Calculate the bucket for the chunk and increment it in the histogram.

bwplotka · 2021-07-02T08:40:52Z

cmd/promtool/tsdb.go

+		}
+
+		for _, chk := range chks {
+			// Load the actual data of the chunk


Suggested change

// Load the actual data of the chunk

// Load the actual data of the chunk.

bwplotka · 2021-07-02T08:41:30Z

cmd/promtool/tsdb.go

+}
+
+func analyzeCompaction(block tsdb.BlockReader, indexr tsdb.IndexReader) (err error) {
+	postingsr, err := indexr.Postings("", "")


Suggested change

postingsr, err := indexr.Postings("", "")

postingsr, err := indexr.Postings(index.AllPostingsKey())

This commit extends the promtool tsdb analyze command to help troubleshoot high Prometheus disk usage. The command now plots a distribution of how full chunks are relative to the maximum capacity of 120 samples per chunk. Signed-off-by: fpetkovski <filip.petkovsky@gmail.com>

bwplotka

One more thing ;p

cmd/promtool/tsdb.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>

bwplotka · 2021-07-02T10:08:56Z

Thanks!

roidelapluie · 2021-07-05T09:04:33Z

This pull request seems a massive regression on the time spent by tsdb analysis, as it now read the chunks. I think we should put this option behind a flag.

fpetkovski · 2021-07-05T09:10:21Z

Hi @roidelapluie, thanks for the review. What do you think about an --extended flag which would enable the compaction analysis?

fpetkovski requested a review from codesome as a code owner June 15, 2021 09:56

fpetkovski commented Jun 15, 2021

View reviewed changes

fpetkovski force-pushed the promtool-analyze-compaction branch 3 times, most recently from ecbac81 to ee54a9c Compare June 15, 2021 10:54

simonpasquier requested a review from dgl June 22, 2021 15:14

fpetkovski changed the title ~~Extend promtool to support compaction analysis~~ Promtool: Add support for compaction analysis Jun 24, 2021

bwplotka approved these changes Jul 1, 2021

View reviewed changes

fpetkovski force-pushed the promtool-analyze-compaction branch 5 times, most recently from 0d5fe71 to 00bcde1 Compare July 2, 2021 06:34

fpetkovski force-pushed the promtool-analyze-compaction branch from 00bcde1 to 3a7828d Compare July 2, 2021 06:39

fpetkovski mentioned this pull request Jul 2, 2021

WIP: Add compaction analysis blog post prometheus/docs#1982

Open

bwplotka approved these changes Jul 2, 2021

View reviewed changes

fpetkovski force-pushed the promtool-analyze-compaction branch from 3a7828d to 0f47b5f Compare July 2, 2021 09:04

bwplotka approved these changes Jul 2, 2021

View reviewed changes

cmd/promtool/tsdb.go Outdated Show resolved Hide resolved

Update cmd/promtool/tsdb.go

e355d87

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>

bwplotka merged commit 7c125aa into prometheus:main Jul 2, 2021

fpetkovski mentioned this pull request Sep 8, 2021

promtool: add extended flag for tsdb analysis #9312

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Promtool: Add support for compaction analysis #8940

Promtool: Add support for compaction analysis #8940

fpetkovski commented Jun 15, 2021 •

edited

fpetkovski Jun 15, 2021

simonpasquier Jun 15, 2021

fpetkovski Jun 21, 2021

dgl Jun 30, 2021

bwplotka Jul 1, 2021

bwplotka commented Jun 29, 2021

bwplotka left a comment

bwplotka Jul 1, 2021

fpetkovski commented Jul 2, 2021 •

edited

bwplotka left a comment

bwplotka Jul 2, 2021

bwplotka Jul 2, 2021

bwplotka Jul 2, 2021

bwplotka Jul 2, 2021

bwplotka left a comment

bwplotka commented Jul 2, 2021

roidelapluie commented Jul 5, 2021 •

edited

fpetkovski commented Jul 5, 2021

	// Normalize absolute counts to percentages and print them out
	// Normalize absolute counts to percentages and print them out.

	// Calculate the bucket for the chunk and increment it in the histogram
	// Calculate the bucket for the chunk and increment it in the histogram.

	// Load the actual data of the chunk
	// Load the actual data of the chunk.

	postingsr, err := indexr.Postings("", "")
	postingsr, err := indexr.Postings(index.AllPostingsKey())

Promtool: Add support for compaction analysis #8940

Promtool: Add support for compaction analysis #8940

Conversation

fpetkovski commented Jun 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka commented Jun 29, 2021

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fpetkovski commented Jul 2, 2021 • edited

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka commented Jul 2, 2021

roidelapluie commented Jul 5, 2021 • edited

fpetkovski commented Jul 5, 2021

fpetkovski commented Jun 15, 2021 •

edited

fpetkovski commented Jul 2, 2021 •

edited

roidelapluie commented Jul 5, 2021 •

edited