tsdb.retention use problem

[timchenxiaoyu]

version=2.5.0, branch=HEAD, revision=67dc912ac8b24f94a1fc478f352d25179c94ab9bset
set --storage.tsdb.retention=4h

get more than 4h monitor data ,why

[codesome]

Blocks are deleted as a whole when it goes beyond retention period.

In your case, assuming block range was 2h, and prometheus started at 2h20m, then the blocks would be 2h20m->4h20m,4h20m->6h20m,6h20m->8h20m. So when the block 2h20m-4h20m goes beyond retention period (8h20m - 4h = 4h20m), that block will be deleted. In your graph you have not touched that time yet.

[timchenxiaoyu]

more than two blocks.

[aixeshunter]

I have same issuse, could someone tell me where is the code about tsdb retention?

[codesome]

https://github.com/prometheus/tsdb/blob/79869d9a4d3d29c494d626aa626724688bbcbe3d/db.go#L515-L518

[aixeshunter]

https://github.com/prometheus/tsdb/blob/79869d9a4d3d29c494d626aa626724688bbcbe3d/db.go#L515-L518

@codesome thanks!

[simonpasquier]

IIUC here what happens. When the retention kicks in, we have:

0. The WAL range is [t-2h, t]
1. The last (most recent) block is [t-4h, t-2h]
2. The (last - 1) block is [t-6h, t-4h]
3. The (last - 2) block is [t-8h, t-6h]
4. The (last - 3) block is [t-10h, t-8h]

TSDB doesn't account for data in the WAL for retention adn uses maxTime of the latest block (eg t-2h) as the reference time so if you have a retention period of 4h, it will delete all blocks that have their upper bound less than t - 2h - 4h = t - 6h which would only be the block at index 4 in the above example. Conclusion: you can have up to 10h of data.

prometheus/vendor/github.com/prometheus/tsdb/db.go

Lines 339 to 356 in 68f0787

	func (db *DB) beyondRetention(meta *BlockMeta) bool {
	if db.opts.RetentionDuration == 0 {
	return false
	}
	db.mtx.RLock()
	blocks := db.blocks[:]
	db.mtx.RUnlock()
	if len(blocks) == 0 {
	return false
	}
	last := blocks[len(db.blocks)-1]
	mint := last.Meta().MaxTime - int64(db.opts.RetentionDuration)
	return meta.MaxTime < mint
	}

[timchenxiaoyu]

as you say will delete index4[t-10h, t-8h] ,but i can get [t-12h, t-10h] mostly

[timchenxiaoyu]

after hour later ,I only get [t-10h, t-8h]. strange!

[timchenxiaoyu]

I Know . now is Wed Dec 19 10:50:02 CST 2018
but recent block is "maxTime": 1545177600000, 2018/12/19 8:0:0 not 2018/12/19 10:0:0
wal log contain more than two hour