Unable to post alert rule group to Mimir set rule group endpoint

[balbirthomas]

Hi,

I am learning the use of Mimir. I have enabled Alertmanager in monolithic mode, with multi-tenancy disabled. I have also configured Mimir alertmanager using its HTTP API endpoint. I have now
created a simple ( pointless ) alert rule file ( attached ) and am trying to upload it to Mimir, using its HTTP API. However I keep getting unable to decoded rule group (HTTP 400) error. I have checked my alert rule file using promtool and the check passes. I am using the Mimir HTTP API as suggested in the reference documentation using the command line

$ curl  -d @cpu_overuse.yaml -H "Content-Type: application/yaml" http://<ENDPOINT-IP>:9009/prometheus/config/v1/rules/anonymous

What am I missing ?

regards

[pracucci]

I would recommend you to upload the rules configuration file using mimirtool instead of manually crafting the request with curl:
https://grafana.com/docs/mimir/latest/operators-guide/tools/mimirtool/#rules

Could you give a try to mimirtool and let us know if it fits your use case, please?

[balbirthomas]

Thank you. I just tried mimirtool. However there still seems to be an error shown below. Note first I tried to list existing rules as a test to check mimirtool is able to connect with Mimir.

$ mimirtool rules list --address=http://<ENDPOINT-IP>:9009 --id=anonymous
Namespace | Rule Group
$ mimirtool rules load cpu_overuse.yaml --address=http://<ENDPOINT-IP>:9009 --id=anonymous
group: 'cpuoveruse', ns: 'cpu_overuse'
/api/v1/rules/cpu_overuse/cpuoveruse
ERRO[0000] server returned HTTP status 400 Bad Request: GetRuleGroup unsupported in rule local store  fields.msg="GetRuleGroup unsupported in rule local store" status="400 Bad Request"
mimirtool: error: load operation unsuccessful, unable to contact Grafana Mimir API: server returned HTTP status 400 Bad Request: GetRuleGroup unsupported in rule local store, try --help

Also I tried changing the group name and/or rule file name to anonymous but still got the same error.

[balbirthomas]

I do see that the error message is being generated in this file. Does this mean loading alert rules into Mimir Alertmanager which is using backend: "filesystem" in alertmanager_storage is not supported ?

[balbirthomas]

Hi,

I found the issue. In the getting started tutorial ruler_storage backend is set to local. I had left it like this. However changing this backend to filesystem fixes the issue.

[pracucci]

Correct! The local backend doesn't support the APIs to update rules config file. Reason is that local is designed for gitops-like deployments, where the source of truth is what's configured on the filesystem (so shouldn't be rewritten via API).