Cannot use arithmetic operators for this case…

[stapelberg]

The manual mentions “Between two instant vectors, a binary arithmetic operator only applies to vector elements that have identical sets of labels between the two vectors. Vector elements that don't find an exact label match on the other side get dropped from the result. The metric name of the result vector is carried over from the left hand side of the expression.”

I have a distributed system which uses raft to process the same state on every of the three nodes. Now, I want to graph the commands which were processed by the master, because that will exclude snapshot recoveries. I’ve exported two metrics for this: a counter for the processed messages, with a label for the message type, and a gauge which is 1 when the node is the master and 0 otherwise.

Now, I tried doing this:

sum(messages_processed) by (command, instance) * sum(raft_isleader) by (instance)

However, this yields 0 results. Once I remove the “command” from the first sum’s by clause, it works, because then the labels match. But I want to keep the command in there, because that should go into the legend :).

Am I missing something here or is this a shortcoming of how you implemented arithmetic operators? Would it be possible to change the behavior so that one could either:

1. specify which label should be used, e.g. sum(messages_processed) by (command, instance) * by (instance) sum(raft_isleader) by (instance)
2. Only require that all labels from the left-hand side are present on the right-hand side, not that the right-hand side contains all labels the left-hand side contains.

Thanks!

[brian-brazil]

This is #393. This requires a modifier as this is a non-standard (though common) use case, and your first suggestion would be a very common mistake leading to unexpected behaviour.

[stapelberg]

Can you clarify what the mistake is in my first suggestion please? :)

[brian-brazil]

It'd work, but it'd cause confusion in the much more common case where someone has mismatched their labels. I think it's better for the language to make the mistake obvious when you first create the expression, rather than producing a likely incorrect and unexpected result and causing confusion.

For example:

sum by (instance, method)(http_latency_sum) 
  / by (instance) 
sum by (instance) (http_latency_count)

is a mistake, and I believe it should fail. Thus I propose adding a modifier to explicitly request the behaviour you're looking for, as mismatching labels is much more common than having a master/slave setup you need to do that sort of math on.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.