-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Loki ruler not working with s3 ruler storage #3922
Comments
Can you ensure the ruler configs are actually mounted into the ruler pod (not just in your helm config)? Your helm config looks good but the logs insinuate that Loki isn't actually seeing the ruler configurations so I'm guessing the problem is somewhere in between. |
This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
I ran across this issue (no alerts trigger when expected) with the At first glance, since I was using
... it seemed that I ought to also use The reason none of my alerts fired was because I never posted the So since you say alerts worked "locally" but not "via s3", my intuition is that you probably made the same mistake as me! I was able to resolve by:
and my test ruler:
|
The same problem does not take effect in S3, and there is no good example for our reference, disappointed. |
Hi, any updates ? same issue in loki 2.4.2 |
same answer as in: grafana/helm-charts#1020 need to use cortextool to put rules at s3, not plain files, yes, quite not obvious, https://grafana.com/docs/loki/latest/rules/#interacting-with-the-ruler Update: actually cortextool also not working, created bug issue: #9351 |
Has anyone found a solution to this? I've been trying to figure out how to configure Loki rules to use an s3 bucket so I can use Grafana Alert State History (https://grafana.com/docs/grafana/latest/alerting/set-up/configure-alert-state-history/) and it's been a pain in the neck. I've also checked the documentation regarding cortextool and it's not clear how to interact with it (besides marking valid loki rules as invalid) Any help is really appreciated :) So far this is my ruler storage in loki configuration:
Everything works ok, but when i check /loki/api/v1/rules it says "no rule groups found" |
Describe the bug
The alerts are not being fired when configuring ruler with helm charts and the storage is s3. I have locally tested and I am able to send alerts with local storage but it is not working with s3 storage.
These are values for loki I am using.
In the logs it says
We are running on single binary mode and all the loki components are running from the single pod . These logs does not always show up so I am not sure if they are showing the correct information.
Also, I am not able to see the rules in the rule_path which is
/tmp/scratch
in our case. If I am using storage as local the rules can se seen in/tmp/scratch/..data/rules/loki-alerting-rules.yaml
I am using loki-stack helm chart of version 2.4.1. SInce I am using helm-chart I believe defining a tenant_id in rule_path will not make sense.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Should see alerts being triggered for alerts defined in loki ruler.
Environment:
Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.
The text was updated successfully, but these errors were encountered: