Using operators with aggregation values

[wrouesnel]

It seems like there's some missing functionality with aggregations and operators. Something like this:
node_load1 - (sum(node_load1) by (job) / count(node_load1) by (job))
would be part of calculating the deviation of load by jobs. But instead prometheus returns no data when you run it.

It's not intuitive to me why: the result is a bunch of vectors labelled as jobs containing the average. If I then ask to subtract that from node_load1, the operation seems like it should be to match all labels by job and do the subtraction between the vectors, resulting in a bunch of (in this case) instance vectors.

[fabxc]

You are aggregating away all labels but job. Vector entries in a binary operation are matched based on all labels of the two entries. As you are aggregating away everything but job that is no longer possible for the outer operation so you have to specify what to match on.

Vector matching docs: http://prometheus.io/docs/querying/operators/#vector-matching

If and how you can modify the matching behavior depends on the shape of your data.
But most likely you are looking for something like this:

node_load1 - on(job) group_left(instance) (
  sum(node_load1) by (job) / count(node_load1) by (job)
)

[wrouesnel]

Ah ok, group_left and group_right were what I was missing. That clarifies things a lot.

[fabxc]

Cool, closing here then.
Feel free to ask any follow up questions though.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.