scrape: configurable startTimeMargin

[thomschke]

Proposal

Use case. Why is this important?

Sometimes I'm using Prom/Grafana to persist/visualize historical timeseries data
with the help of a stateful metric-exporter that provide data step-by-step (include timestamp)

For this use case I have to increase "maxAheadTime"
but in the moment it's a fixed value of 10s.

prometheus/scrape/scrape.go

Line 136 in 105ed5c

const maxAheadTime = 10 * time.Minute

[brian-brazil]

maxAheadTime is from now, not the TSDB's last ingested timestamp. It only prevents you ingesting data from the future, as a safeguard.

[thomschke]

Sure, the lower bound of TSDB (startTimeMargin) is the problem - sorry :-)

prometheus/cmd/prometheus/main.go

Lines 574 to 575 in 3241c52

	startTimeMargin := int64(2 * time.Duration(cfg.tsdb.MinBlockDuration).Seconds() * 1000)
	localStorage.Set(db, startTimeMargin)

prometheus/storage/tsdb/tsdb.go

Line 162 in 3241c52

startTime = time.Now().Unix() * 1000

[thomschke]

A workaround is by increasing storage.tsdb.min-block-duration (but this has sideeffects)
(see coreos/prometheus-operator#1375)

[thomschke]

Hi @brian-brazil, can you explain a little bit the drawbacks of increasing storage.tsdb.min-block-duration in production?

prometheus/cmd/prometheus/main.go

Lines 156 to 157 in f34b260

	a.Flag("storage.tsdb.min-block-duration", "Minimum duration of a data block before being persisted. For use in testing.").
	Hidden().Default("2h").SetValue(&cfg.tsdb.MinBlockDuration)