Combining metrics

[hryamzik]

I have a metric that comes from different hosts exported by node_exporter. Due to the way it's exported it has two different labels: env and instance. I'd like to remove them for a set of metrics but once I do so a query error appears:

(label_replace(label_replace(domain_check_expiration_date,"instance", "", "instance", ".*"),"env", "", "env", ".*") - time()) / 3600/24 < 14

Error executing query: duplicated label set in output of label_replace(): domain_check_expiration_date{job="node", name="example.org"}

This metric comes from two hosts for redundancy reasons but it should be aggregated as a single one. How can I do this? I'm fine to move to a blackbox_exporter but I don't see cmd support there and expect to get same issues with duplicates.

[brian-brazil]

What are you expecting as the output? The error is because what you've requested results in clashing time series, and there's no sane semantics for that.

[brian-brazil]

Also as a general note adding additional duplicate exporters for redundancy is often more hassle than it's worth. In this case even if you have a single exporter that fails, you'll have many days to get it back up and running which is more than enough time to get domains renewed.

[hryamzik]

I have some other metrics that should be duplicated. At the moment I have redundancy for all the prometheus stack including prometheus itself and an alert manager and I'm very happy with the way it works – no duplicated alerts at all.

What are you expecting as the output? The error is because what you've requested results in clashing time series, and there's no sane semantics for that.

Good point. In this particular case I'd like to get min value but once I apply the min function all the labels got lost. So at least it's possible to create a personal rule for each domain and use the min function. Is there a way to avoid this?

[brian-brazil]

Are you specifying an ignoring clause on your min?

[hryamzik]

I believe you meant without – I wasn't able to create a valid queue with ignoring.

And it did work, thank you so much!

(min(domain_check_expiration_date) without (job,env,instance) - time() )/3600/24

[hryamzik]

Oops, with without I still get two results but they are similar, should this result in a single alert?

[brian-brazil]

Are you sure you've removed all the labels you need to?

[hryamzik]

I am, I have now two labels left:

{host="some_ip_here",name="example.org"}

And the values are expectedly equal.

[hryamzik]

Oops, my bad, IP were different. Closing this now, thank you again!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.