promtool tsdb analyze maps whole db, but only looks at one block

[bobrik]

What did you do?

I ran promtool tsdb analyze /path/to/data 01E8BRZKNZG99J3PJDW31MZ1WR.

What did you expect to see?

I expected promtool to mmap a single block I picked and analyze it.

What did you see instead? Under which circumstances?

The whole 50TiB of my prometheus tsdb were mmaped into memory over the 20 minute time span. Lots of disk I/O happened.

Similar things to happen when you don't supply the block ID argument: the whole tsdb is loaded, then the latest block is picked and analyzed.

System information

Linux 5.15

Prometheus version

prometheus, version 2.36.0 (branch: master, revision: 0cb07b796a07b652485fc8b3504eb149a44af381)
  build user:       builder@k8s-pdx-managed-7pc7j.cluster.local
  build date:       20220607-08:35:43
  go version:       go1.18.2
  platform:         linux/amd64

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[bobrik]

Here's a dumb solution to speed things up if anyone is facing the same issue:

1. Remove IsDir check from promtool:

diff --git a/tsdb/db.go b/tsdb/db.go
index ab0822d73..b4ae9de20 100644
--- a/tsdb/db.go
+++ b/tsdb/db.go
@@ -1718,9 +1718,6 @@ func (db *DB) CleanTombstones() (err error) {
 }

 func isBlockDir(fi fs.DirEntry) bool {
-       if !fi.IsDir() {
-               return false
-       }
        _, err := ulid.ParseStrict(fi.Name())
        return err == nil
 }

2. Create a temporary directory, and symlink a single block there:

mkdir /tmp/derp
ln -s /disk/data/prometheus/performance/data/01E8BRZKNZG99J3PJDW31MZ1WR /tmp/derp/

3. Analyze your temporary directory:

/tmp/promtool tsdb analyze /tmp/derp 01E8BRZKNZG99J3PJDW31MZ1WR

[ffjlabo]

@bobrik @gouthamve Hi! I'd like to work on this!