Usage of Set Rule Group HTTP API

[balbirthomas]

Hi I am trying to upload an alert rule group to Mimir Alertmanager using its HTTP API.

When I use a curl command line such as

$ curl  -d @cpu_overuse.yaml -H "Content-Type: application/yaml" http://<ENDPOINT-IP>:9009/prometheus/config/v1/rules/anonymous

I see the following error

2022-05-17T09:37:23.283Z [mimir] level=error ts=2022-05-17T09:37:23.283026808Z caller=api.go:470 org_id=anonymous msg="unable to unmarshal rule group payload"
 err="yaml: mapping values are not allowed in this context"

I infer this is coming from here. I would like to write scripts using the HTTP API to control and configure Mimir.

I do find that many other HTTP APIs do work as expected. I am however unable to make the set-rule-group HTTP API work using the curl command line shown above. I am only using curl to learn and explore this API. As far as I can tell I am providing all requisite HTTP headers. I am not providing X-Scope-OrgID since I have disabled multi-tenancy.

I can confirm mimirtool does work when used to upload the very same rule group file. Also I have checked the same rule group file using mimirtool lint, check and prepare, all of which passed.

Could this be a bug or am I missing something ?

[pracucci]

Does it work if you substitute -d with --data-binary to your curl command? Reason is that -d doesn't preserve carriage returns and newlines.

We mention it in https://grafana.com/docs/mimir/latest/operators-guide/reference-http-api/#set-alertmanager-configuration: if it fixes your issue, we should mention it for https://grafana.com/docs/mimir/latest/operators-guide/reference-http-api/#set-rule-group too.

[balbirthomas]

I am pasting it below and also linking it here

namespace: anonymous
groups:
    - name: cpuoveruse
      rules:
        - alert: cpu_over_use
          expr: 'process_cpu_seconds_total > 10'
          labels:
            severity: Low
          annotations:
            description: '{{ $labels.instance }} of job {{ $labels.job }} has used too much CPU.'
            summary: Instance {{ $labels.instance }} CPU over use

(A pointless alert rule only for testing and learning).

[pracucci]

I think I understood the issue. The API you're calling allows you to create 1 rule group, not namespace (which is a set of groups). So the actual content of the file you upload should be:

name: cpuoveruse
rules:
  - alert: cpu_over_use
    expr: 'process_cpu_seconds_total > 10'
    labels:
      severity: Low
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has used too much CPU.'
      summary: Instance {{ $labels.instance }} CPU over use

[balbirthomas]

I can confirm that you are absolutely right. Thank you very very much. My apologies that I may have misread the documentation. In retrospect it is implicit in the set rule group docs based on the use of the singular (group) rather than plural (groups). I had falsely assumed that the format would be identical to a prometheus alert rule file.

I wonder if it would make sense to add emphasis in the set rule group docs that only a single group (without groups) is supported by this API particularly since this HTTP response error message may not lead to this conclusion.

Thank you again.

[pracucci]

I wonder if it would make sense to add emphasis in the set rule group docs that only a single group (without groups) is supported by this API particularly since this HTTP response error message may not lead to this conclusion.

Definitely! I'm working on a PR right now ;)

[pracucci]

PR to improve the doc: #1869