Fix JaegerQueryReqsFailing alert rule missing 'operation' in query

[james-ryans]

Which problem is this PR solving?

• Resolves [Bug]: JaegerQueryReqsFailing alert specifies 'operation' in message and it isn't aggregated by the alert expr #4771

Description of the changes

• Add group by operation to JaegerQueryReqsFailing rule
• Here's the difference between before and after the changes. With operation group by, the alert will be distinguished by operation

- Tested manually because PR at #1745 says creating the automated tests are not easy.

How was this change tested?

• Manually tested in local
• Ran curl -s http://localhost:9090/api/v1/alerts | jq and returns

{
  "status": "success",
  "data": {
    "alerts": [
      {
        ...
        "annotations": {
          "message": "prometheus localhost:14269 is seeing 100.00% query errors on get_trace.\n"
        },
        ...
      }
    ]
  }
}

Before it was

{
  "status": "success",
  "data": {
    "alerts": [
      {
        ...
        "annotations": {
          "message": "prometheus localhost:14269 is seeing 100.00% query errors on .\n"
        },
        ...
      }
    ]
  }
}

• Ran pint lint prometheus_alerts.yml

james_ryans@james-ryans ~/g/s/g/j/jaeger (spm_alert_operation)> pint lint monitoring/jaeger-mixin/prometheus_alerts.yml 
level=info msg="Problems found" Information=8
level=info msg="1 problem(s) not visible because of --min-severity=warning flag"

Before changes it was

pint lint monitoring/jaeger-mixin/prometheus_alerts.yml
monitoring/jaeger-mixin/prometheus_alerts.yml:62-64 Bug: template is using "operation" label but the query removes it (alerts/template)
 62 |       "message": |
 63 |         {{ $labels.job }} {{ $labels.instance }} is seeing {{ printf "%.2f" $value }}% query errors on {{ $labels.operation }}.
 64 |     "expr": "100 * sum(rate(jaeger_query_requests_total{result=\"err\"}[1m])) by (instance, job, namespace) / sum(rate(jaeger_query_requests_total[1m])) by (instance, job, namespace)> 1"

level=info msg="Problems found" Bug=1 Information=8
level=info msg="1 problem(s) not visible because of --min-severity=warning flag"
level=fatal msg="Execution completed with error(s)" error="found 1 problem(s) with severity Bug or higher"

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[albertteoh]

Thanks!

By the way, I don't think this relates to SPM.

[codecov]

Codecov Report

All modified lines are covered by tests ✅

see 1 file with indirect coverage changes

📢 Thoughts on this report? Let us know!.

[james-ryans]

By the way, I don't think this relates to SPM.

Oops, I've changed the title