prometheus_sd_configs_failed_total is not actionable, gauge would be better

[juliusv]

The metric prometheus_sd_configs_failed_total (

prometheus/discovery/manager.go

Lines 44 to 49 in a9050c4

	failedConfigs = prometheus.NewCounter(
	prometheus.CounterOpts{
	Name: "prometheus_sd_configs_failed_total",
	Help: "Total number of service discovery configurations that failed to load.",
	},
	)

) currently tracks the total number of SD configurations that were not able to be applied during configuration (re)load. However, the actual rate of that counter depends highly on the configuration reload rate, so a bad SD config might stick around forever without actually causing any counter increments. This means that this metric is not very useful for alerts. We should introduce a gauge metric that tells us how many SD configurations are currently invalid / were not able to be instantiated.

[infoverload]

I would like to work on this

[geekodour]

@infoverload , are you working on this?

[infoverload]

@geekodour hey, yes I am!

[palash25]

@juliusv I don't see any referenced PR by the original claimer on this issue, would it be okay if I take a crack at this?

[juliusv]

@palash25 Yes, please go ahead, I haven't heard anything in a while either from @infoverload about this.