disk space usage metrics

[anarcat]

I have recently made some tests with prometheus, and one of my key concerns is disk space usage. Because there is no downsampling (planned? prometheus/tsdb#56, #1381), I worry that altering the retention period to a much larger range (say one year) will take an abusive amount of disk space (and hurt performance but that's another question).

I have kept an eye on disk space by running du -sch /var/lib/prometheus by hand, and I know others are running prometheus on its own partition which makes monitoring easier, but i would like to avoid complicating the deployment...

Shouldn't there be a metric exposed by prometheus about the disk space it uses? it probably knows how many chunks it has - how hard would it be to expand that to the actual disk space in use?

(edit: i found out about process_resident_memory_bytes which seems to be giving a good metric for memory usage).

Thanks!

[anarcat]

I have just noticed this excellent tutorial which partly covers the requirement here, although it is not shipped out of the box.

The only way I could find to do this (and it's in tsdb) is with a crude patch that walks the filesystem, which is not much faster than the above implementation. The patch is in anarcat/tsdb@2ba6cb8 but i'm worried it will hurt performance.

We could assume the disk cache will be primed and that listing directories in the chunks storage will be fast enough, but I'm not familiar enough with the new storage engine to assert that confidently.

[brian-brazil]

It should be fast, but I think it'd be better to be tracking this internally.

[anarcat]

yeah well, i couldn't figure out how to do this - the Chunk objects seem to be variable length so they're not useful, and i suspect only some of those are kept in memory anyways...

[simonpasquier]

Relates to prometheus/tsdb#343