Attempt to filter out duplicates

[paroxp]

What

Some of the metrics are duplicated at the prometheus level... We can add
some group by closures, to make sure we're finding unique information,
where we need to.

How to review

• Take te queries and run through grafana
• Sanity check

[AP-Hunt]

I compared the number of series (pre-sum) with and without the group by for cf_service_instance_info metric in Ireland: 485 vs 241. Seems like a good optimisation.

I'm not quite sure how to check the organization metrics. Can you clarify that, please?

[paroxp]

Sure, if you check the performance page in london now, there are ~280 orgs on record.

  sum by (type) (group by (organization_name,type) (label_replace(
    label_replace(
      cf_organization_info{organization_name!~"(AIVENBACC|BACC|ACC|ASATS|SMOKE).*"},
      "type", "billable", "quota_name", "(gds-non-chargeable|small|medium|large|xlarge|2xlarge|4xlarge|8xlarge)"
    ),
    "type", "trial", "quota_name", "default"
  )))

The above, groups the orgs by organization_name name. But when you do only org name, the type is not being returned. So I group by both. The above results in 139 orgs total. 67 trial and 72 billable.

If you run the previous query

  sum by (type) (label_replace(
    label_replace(
      cf_organization_info{organization_name!~"(AIVENBACC|BACC|ACC|ASATS|SMOKE).*"},
      "type", "billable", "quota_name", "(gds-non-chargeable|small|medium|large|xlarge|2xlarge|4xlarge|8xlarge)"
    ),
    "type", "trial", "quota_name", "default"
  ))

you will see basically the double numbers.

Now, if you're asking about the weird structure of the query...

I use two label_replace functions, to create a dividing line between trial orgs and the billable ones. So, if an org is one of: gds-non-chargeable|small|medium|large|xlarge|2xlarge|4xlarge|8xlarge is set to billable, and then any with default is simply a trial... I don't like the way it's done, as it will need to be updated if and when we change quotas... But seems to be the only way at the moment...

Another weirdness here is: organization_name!~"(AIVENBACC|BACC|ACC|ASATS|SMOKE).*" which essentially filters out orgs where the name starts with any of AIVENBACC|BACC|ACC|ASATS|SMOKE so that we don't over count our tests.

Let me know if that's not ... somehow ... easier xD

[AP-Hunt]

Thanks, Rafal. That makes sense.