Loki ruler not working with s3 ruler storage

[yashaswee]

Describe the bug
The alerts are not being fired when configuring ruler with helm charts and the storage is s3. I have locally tested and I am able to send alerts with local storage but it is not working with s3 storage.
These are values for loki I am using.

  values: |
      loki:
        resources:
          limits:
            cpu: 2
            memory: xx
          requests:
            cpu: 1
            memory: xx
        enabled: true
        podAnnotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "http-metrics"
          iam.amazonaws.com/role: 'grafana-loki'
        config:
          schema_config:
            configs:
            - from: 2021-06-01
              store: boltdb-shipper
              object_store: s3
              schema: v11
              index:
                prefix: loki_index_
                period: 24h
          storage_config:
            aws:
              s3: s3://region/bucket-name
            boltdb_shipper:
              active_index_directory: /data/loki/index
              shared_store: s3
              cache_location: /data/loki/boltdb-cache
          ruler:
            storage:
              type: s3
              s3:
                s3: s3://region/bucket-name
            alertmanager_url: http://alertmanager.default.svc.cluster.local:9093
            notification_timeout: 1m
            rule_path: /tmp/scratch
            ring:
              kvstore:
                store: inmemory
            enable_api: true
        alerting_groups:
          - name: testing
            rules: 
              - alert: TestAlert
                expr: |
                  rate({container="testing"}  [1m]) > 0
                for: 5m
                labels:
                    severity: warning
                    team: devops
                annotations:
                    title: "testing"

In the logs it says

Jun 30 19:13:04  loki level=info ts=2021-07-01T00:13:04.524299398Z caller=module_service.go:59 msg=initialising module=ruler
Jun 30 19:13:04 loki level=info ts=2021-07-01T00:13:04.524324997Z caller=ruler.go:403 msg="ruler up and running"
Jun 30 23:14:30  loki level=info ts=2021-07-01T04:14:30.810076529Z caller=modules.go:477 msg="RulerStorage is not configured in single binary mode and will not be started."
Jun 30 23:14:31  loki level=info ts=2021-07-01T04:14:31.771784452Z caller=modules.go:502 msg="RulerStorage is nil.  Not starting the ruler."
Jun 30 23:36:23 loki level=info ts=2021-07-01T04:36:23.289889019Z caller=modules.go:477 msg="RulerStorage is not configured in single binary mode and will not be started."
Jun 30 23:36:24 loki level=info ts=2021-07-01T04:36:24.244241725Z caller=modules.go:502 msg="RulerStorage is nil.  Not starting the ruler."

We are running on single binary mode and all the loki components are running from the single pod . These logs does not always show up so I am not sure if they are showing the correct information.
Also, I am not able to see the rules in the rule_path which is /tmp/scratch in our case. If I am using storage as local the rules can se seen in /tmp/scratch/..data/rules/loki-alerting-rules.yaml

loki-promtail-bnr54                1/1     Running   0          14h
loki-0                             1/1     Running   0          14h

I am using loki-stack helm chart of version 2.4.1. SInce I am using helm-chart I believe defining a tenant_id in rule_path will not make sense.

To Reproduce
Steps to reproduce the behavior:

1. Run the loki helm charts with the above values

Expected behavior
Should see alerts being triggered for alerts defined in loki ruler.

Environment:

• Infrastructure: kubernetes cluster version 18.xx
• Deployment tool: helm

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

[owen-d]

Can you ensure the ruler configs are actually mounted into the ruler pod (not just in your helm config)? Your helm config looks good but the logs insinuate that Loki isn't actually seeing the ruler configurations so I'm guessing the problem is somewhere in between.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[mars64]

I ran across this issue (no alerts trigger when expected) with the loki-distributed chart, but I think ultimately I had misunderstood Loki's intention here.

At first glance, since I was using s3 for shared storage elsewhere a la:

loki:
  config: |
    storage_config:
      aws:
        s3: s3://us-west-2
        bucketnames: my-loki-s3-bucket
      boltdb_shipper:
        shared_store: s3

... it seemed that I ought to also use s3 for the ruler storage. As far as I can tell, this is incorrect if you use the loki-distributed chart for ruler configuration (and probably the other charts too).

The reason none of my alerts fired was because I never posted the rules.yaml files to my bucket -- I had configured them in .loki.ruler.directories.fake , which are only mapped in as a config-map!

So since you say alerts worked "locally" but not "via s3", my intuition is that you probably made the same mistake as me!

I was able to resolve by:

loki:
  config: |
    ruler:
      alertmanager_url: {{ .Values.ruler.alertmanagerUrl }}
      enable_alertmanager_v2: true
      enable_api: true
      external_url: {{ .Values.ruler.alertmanagerExternalUrl }}
      ring:
        kvstore:
          store: memberlist
          prefix: rulers/
      rule_path: /tmp/loki
      storage:
        type: local
        local:
          directory: /etc/loki/rules/fake

and my test ruler:

loki:
  ruler:
    enabled: true
    directories:
      # in single tenant mode, the tenant ID is "fake"
      # https://grafana.com/docs/loki/latest/operations/multi-tenancy/ 
      # override this section in env-specific values
      fake: 
        testRules.yaml: |
          groups:
            - name: test1
              rules:
                - alert: test1
                  expr: |
                    1 > 0
                  for: 0m
                  labels:
                    # testing, post to #catchall for now
                    slack: slack-catchall
                    severity: warning
                  annotations:
                    summary: TEST: meowburgers and cheese

[GounGG]

The same problem does not take effect in S3, and there is no good example for our reference, disappointed.

[LinTechSo]

Hi, any updates ? same issue in loki 2.4.2

[dragoangel]

same answer as in: grafana/helm-charts#1020

need to use cortextool to put rules at s3, not plain files, yes, quite not obvious, https://grafana.com/docs/loki/latest/rules/#interacting-with-the-ruler

Update: actually cortextool also not working, created bug issue: #9351
Update 2.0: new Loki Helm chart v5.5.0 fixing issues fully. In my issue 9351 I shared working configuration that can be managed via Grafana fully.

[rgg1993]

Has anyone found a solution to this? I've been trying to figure out how to configure Loki rules to use an s3 bucket so I can use Grafana Alert State History (https://grafana.com/docs/grafana/latest/alerting/set-up/configure-alert-state-history/) and it's been a pain in the neck. I've also checked the documentation regarding cortextool and it's not clear how to interact with it (besides marking valid loki rules as invalid) Any help is really appreciated :)

So far this is my ruler storage in loki configuration:

ruler:
  storage:
    type: s3
    s3:
      s3: s3://my-bucket

Everything works ok, but when i check /loki/api/v1/rules it says "no rule groups found"