New prometheus metric storing information about series age

[bkupidura]

Suggested scenario for Prometheus HA is to have multiple instances of Prometheus scraping same targets. But there is no way to distinguish which instance is oldest.

This can be problem for ex. graphing tools (grafana).

1. Spawn 2 instances of prometheus (A, B)
2. Instance A is down
3. Spawn new instance (C)
4. Grafana should use 'B' to serve graphs, till 'C' will catch up with series

Prometheus process_start_time_seconds can't be treated as reliable for this purpose.

We can use load-balancer to disable 'fresh' instances, but to make this more autonomous (no operator intervention needed) we need metric which gives us information which Prometheus stores oldest series.

We can expose a few additional metrics:

1. oldest_metric timestamp
2. local storage size
3. number of series in storage

Option 2) sounds ridiculous and not reliable
Option 3) based on @brian-brazil opinion will not be easy to implement in Prometheus 2.X

[brian-brazil]

We will want to know when the DB was created later for use when querying long term storage (don't need to query what we already have locally).

[gouthamve]

Yes, oldest_timestamp is a good metric to have. I am not sure how this would work in 1.x but it is trivial to implement in 2.0.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.