Metric tracking crash recoveries?

[mattbostock]

There's a metric to track inconsistencies in the local storage:

prometheus_local_storage_inconsistencies_total

prometheus/storage/local/persistence.go

Lines 242 to 247 in 70490fe

	dirtyCounter: prometheus.NewCounter(prometheus.CounterOpts{
	Namespace: namespace,
	Subsystem: subsystem,
	Name: "inconsistencies_total",
	Help: "A counter incremented each time an inconsistency in the local storage is detected. If this is greater zero, restart the server as soon as possible.",
	}),

...but it's not incremented (as far as I can tell) when the storage is found dirty during startup (and crash recovery is invoked):

prometheus/storage/local/persistence.go

Lines 700 to 708 in 70490fe

	defer func() {
	if p.dirty {
	log.Warn("Persistence layer appears dirty.")
	err = p.recoverFromCrash(fingerprintToSeries)
	if err != nil {
	sm = nil
	}
	}
	}()

The help for the above metric says:

A counter incremented each time an inconsistency in the local storage is detected. If this is greater zero, restart the server as soon as possible.

So it probably doesn't make sense to use it to indicate that crash recovery took place when Prometheus started.

Should we have a separate metric dedicated to this purpose?

[beorn7]

As discussed in person, I think a metric like that would make sense. It's usually obvious from the logs that it happened, but you have to check the logs to find out. So the fact that you should check the logs would be something for a ticketing alert. Could be prometheus_local_storage_started_dirty or something.

[mattbostock]

Thanks, I'll raise a PR soon.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.