Loki push_api lib is dropping a valid alert rules file

[sed-i]

While debugging cos-config I found out that the following rule never reaches loki (doesn't end up in relation data).
This sample file is taken from the official docs:
https://grafana.com/docs/loki/latest/rules/#example

groups:
  - name: should_fire
    rules:
      - alert: HighPercentageError
        expr: |
          sum(rate({app="foo", env="production"} |= "error" [5m])) by (job)
            /
          sum(rate({app="foo", env="production"}[5m])) by (job)
            > 0.05
        for: 10m
        labels:
            severity: page
        annotations:
            summary: High request latency
  - name: credentials_leak
    rules: 
      - alert: http-credentials-leaked
        annotations: 
          message: "{{ $labels.job }} is leaking http basic auth credentials."
        expr: 'sum by (cluster, job, pod) (count_over_time({namespace="prod"} |~ "http(s?)://(\\w+):(\\w+)@" [5m]) > 0)'
        for: 10m
        labels: 
          severity: critical

[dstathis]

@sed-i could you give some reproduction steps? I attempted to reproduce this and it worked just fine.

[sed-i]

Feel free to close this issue if you're able to see the forwarded rule when curling loki's /loki/loki/api/v1/rules endpoint.

[rbarry82]

@sed-i still reproducible?

[sed-i]

That file is in the cos-config repo so it was easy to retest:

$ juju deploy --channel=edge cos-configuration-k8s cos-config --config git_repo=https://github.com/canonical/cos-configuration-k8s-operator --config git_branch=main --config loki_alert_rules_path="tests/samples/loki_alert_rules/"
$ juju deploy --channel=edge loki-k8s loki
$ juju relate loki cos-config
$ curl -s 10.1.50.221:3100/loki/api/v1/rules | grep  "HighPercentageError\|http\-credentials\-leaked"
        - alert: HighPercentageError
        - alert: http-credentials-leaked

Closing.