2.0-rc1: Can't set retention to less than 20 hours

[haizaar]

What did you do?

./prometheus --config.file ./prometheus-test.yml --storage.tsdb.path=/tmp/data --storage.tsdb.retention=10h

What did you expect to see?
Expected to see prometheus running with retention configured to 10h.

What did you see instead? Under which circumstances?
Prometheus exited with error. See the log below.

Environment

• System information:
  Reproduces both on GKE 1.7.6 and local dev Ubuntu 16.04

• Prometheus version:

prometheus, version 2.0.0-rc.1 (branch: HEAD, revision: 5ab8834befbd92241a88976c790ace7543edcd59)
  build user:       root@1f56dd8b6f7b
  build date:       20171017-12:34:15
  go version:       go1.9.1

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     10s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

• Logs:

level=info ts=2017-10-24T05:07:40.291231027Z caller=main.go:216 msg="Starting prometheus" version="(version=2.0.0-rc.1, branch=HEAD, revision=5ab8834befbd92241a88976c790ace7543edcd59)"
level=info ts=2017-10-24T05:07:40.291272148Z caller=main.go:217 build_context="(go=go1.9.1, user=root@1f56dd8b6f7b, date=20171017-12:34:15)"
level=info ts=2017-10-24T05:07:40.291291973Z caller=main.go:218 host_details="(Linux 4.12.14-041214-generic #201709200843 SMP Wed Sep 20 12:46:23 UTC 2017 x86_64 hera (none))"
level=info ts=2017-10-24T05:07:40.293175522Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-10-24T05:07:40.293239206Z caller=main.go:315 msg="Starting TSDB"
level=info ts=2017-10-24T05:07:40.293234522Z caller=targetmanager.go:68 component="target manager" msg="Starting target manager..."
level=error ts=2017-10-24T05:07:40.293466253Z caller=main.go:324 msg="Opening storage failed" err="create leveled compactor: at least one range must be provided"

[zemek]

I believe this is because the default for --storage.tsdb.min-block-duration is 2 hours.

The default for --storage.tsdb.max-block-duration is 10% of the retention period (which for a 10 hour retention would make the max-block-duration = 1 hour)

Since you can't have the max-block-duration be less than the min-block-duration, try setting --storage.tsdb.min-block-duration=30m (or instead set the max-block-duration to be longer)

[haizaar]

Thanks for the detailed answer - makes sense.
Now I'm able to configure the retention to be as short as I need. It's for federated clusters, that have p8s only as a proxy to be scrapped into dedicated monitoring cluster.

[fabxc]

Reopening since the short retention period should work out of the box. The *-block-duration flags are mostly for debugging purposes at this point.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.