Why was not first block compacted ?

[shogosanz]

What did you do?
I executed this command.

$./prometheus --storage.tsdb.min-block-duration 2m

What did you expect to see?
tsdb compact older 3 blocks.

What did you see instead? Under which circumstances?
But, tsdb did not compact first block.

(15:52:45)prometheus-2.0.0.darwin-amd64$ rm -rf data/

(15:52:57)prometheus-2.0.0.darwin-amd64$./prometheus --storage.tsdb.min-block-duration 2m

(15:52:59)prometheus-2.0.0.darwin-amd64$cd data

(15:53:12)data$ls -lh
total 8
drwxr-xr-x  6   204B 12  4 15:55 01C0G4XKE0XJBSBC0DVWM4352F
drwxr-xr-x  6   204B 12  4 15:57 01C0G518MCD3ZM7KHCS9WJ7JE2
drwxr-xr-x  6   204B 12  4 15:59 01C0G54XTBYTBBHYG0H161A853
-rw-------  1     5B 12  4 15:52 lock
drwxr-xr-x  3   102B 12  4 15:53 wal

?
Why were not blocks compacted ?
↓

(16:01:01)data$ls -lh
total 8
drwxr-xr-x  6   204B 12  4 15:55 01C0G4XKE0XJBSBC0DVWM4352F
drwxr-xr-x  6   204B 12  4 16:01 01C0G58KDVPQ1S5ZZN6JWZGB9G
-rw-------  1     5B 12  4 15:52 lock
drwxr-xr-x  3   102B 12  4 15:53 wal

2nd block, 3rd block and 4th block were compacted.

(16:01:10)data$cat 01C0G4XKE0XJBSBC0DVWM4352F/meta.json | grep compaction -A1
	"compaction": {
		"level": 1,
(16:01:25)data$cat 01C0G58KDVPQ1S5ZZN6JWZGB9G/meta.json | grep compaction -A1
	"compaction": {
		"level": 2,

Environment

• System information:

Darwin 16.7.0 x86_64

• Prometheus version:

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:15:39
  go version:       go1.9.2

• Prometheus configuration file:

scrape_configs:
  - job_name: 'zisaku'
    static_configs:
      - targets: ['localhost:8080']

• Logs:

(15:52:57)prometheus-2.0.0.darwin-amd64$./prometheus --storage.tsdb.min-block-duration 2m
level=info ts=2017-12-04T06:52:58.877851223Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2017-12-04T06:52:58.877906911Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:15:39)"
level=info ts=2017-12-04T06:52:58.877920009Z caller=main.go:217 host_details=(darwin)
level=info ts=2017-12-04T06:52:58.879486455Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-12-04T06:52:58.879503236Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2017-12-04T06:52:58.879511208Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2017-12-04T06:52:58.8846527Z caller=main.go:326 msg="TSDB started"
level=info ts=2017-12-04T06:52:58.884684782Z caller=main.go:394 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2017-12-04T06:52:58.88510628Z caller=main.go:371 msg="Server is ready to receive requests."
level=info ts=2017-12-04T06:55:23.584700925Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1512370320000 maxt=1512370440000
level=info ts=2017-12-04T06:55:24.017933606Z caller=head.go:345 component=tsdb msg="head GC completed" duration=2.033178ms
level=info ts=2017-12-04T06:55:24.018003846Z caller=head.go:354 component=tsdb msg="WAL truncation completed" duration=2.702µs
level=info ts=2017-12-04T06:57:23.596295509Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1512370440000 maxt=1512370560000
level=info ts=2017-12-04T06:57:24.016639436Z caller=head.go:345 component=tsdb msg="head GC completed" duration=1.133484ms
level=info ts=2017-12-04T06:57:24.016697952Z caller=head.go:354 component=tsdb msg="WAL truncation completed" duration=2.073µs
level=info ts=2017-12-04T06:59:23.595439185Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1512370560000 maxt=1512370680000
level=info ts=2017-12-04T06:59:24.011797734Z caller=head.go:345 component=tsdb msg="head GC completed" duration=1.206721ms
level=info ts=2017-12-04T06:59:24.011890934Z caller=head.go:354 component=tsdb msg="WAL truncation completed" duration=2.221µs
level=info ts=2017-12-04T07:01:23.60381518Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1512370680000 maxt=1512370800000
level=info ts=2017-12-04T07:01:24.022876718Z caller=head.go:345 component=tsdb msg="head GC completed" duration=1.29613ms
level=info ts=2017-12-04T07:01:24.022954668Z caller=head.go:354 component=tsdb msg="WAL truncation completed" duration=3.468µs
level=info ts=2017-12-04T07:01:24.027297457Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512370440000 maxt=1512370800000
level=info ts=2017-12-04T07:03:23.59994402Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1512370800000 maxt=1512370920000
level=info ts=2017-12-04T07:03:24.004821545Z caller=head.go:345 component=tsdb msg="head GC completed" duration=790.349µs
level=info ts=2017-12-04T07:03:24.004865171Z caller=head.go:354 component=tsdb msg="WAL truncation completed" duration=1.185µs
^Clevel=warn ts=2017-12-04T07:04:48.358025022Z caller=main.go:377 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2017-12-04T07:04:48.358096373Z caller=main.go:384 msg="See you next time!"
level=info ts=2017-12-04T07:04:48.360030537Z caller=targetmanager.go:87 component="target manager" msg="Stopping target manager..."
level=info ts=2017-12-04T07:04:48.360108592Z caller=targetmanager.go:99 component="target manager" msg="Target manager stopped"
level=info ts=2017-12-04T07:04:48.360146949Z caller=manager.go:455 component="rule manager" msg="Stopping rule manager..."
level=info ts=2017-12-04T07:04:48.360168209Z caller=manager.go:461 component="rule manager" msg="Rule manager stopped"
level=info ts=2017-12-04T07:04:48.360195763Z caller=notifier.go:483 component=notifier msg="Stopping notification handler..."

Please give me some advice on what I should do.
Thank you and regards.

[gouthamve]

We compact blocks in time-ranges and it looks like the 2nd and 3rd blocks in one range while the 1st is not. See: https://github.com/prometheus/tsdb/blob/master/compact.go#L165-L191

Also, it makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.