Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki Ruler - Provide a way of accessing alert firing timestamp or current time in annotations #4980

Open
dwilliams782 opened this issue Dec 20, 2021 · 3 comments
Labels
keepalive An issue or PR that will be kept alive and never marked as stale.

Comments

@dwilliams782
Copy link

dwilliams782 commented Dec 20, 2021

Is your feature request related to a problem? Please describe.

Note: I raised this in Slack as well as it's somewhere between a FR and a question.

We are using Loki Ruler to send alerts to Alert Manager. These alerts contain an annotation which links back to Grafana / Loki, example:

groups:
  - name: quota_exceeded
    rules:
      - alert: QUOTA_EXCEEDED
        expr: |
          sum by (project) (count_over_time({job="gcplog"} |= "QUOTA_EXCEEDED" [5m]) > 0)
        for: 1m
        annotations:
          summary: "A GCP Quota has been exceeded. See logs for details."
          loki: <loki-url>
       

The alert expression and annotations have been truncated to hide our domains but you see the point - we pass a loki URL which I would like to be able to set the from and to parameters using the time the alert fires +-15 mins so we can point to an accurate timeframe for the logs.

Describe the solution you'd like

#4603 adds some nice datetime functions like now and unixEpoch, which work well if you add them as a label. Adding them as a label to use in the URL would work but it isn't ideal as it'll interfere with the sum by aggregation, so I'd like a way to be able to call these functions.

In prometheus, you can simply query time():

{{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}

but with logQL, any attempts to use a function generates a:

parse error at line 1, col 1: syntax error: unexpected IDENTIFIER

Ideally there will be a way to generate a timestamp we can use in a similar fashion to promQL, maybe:

{{ with query "(unixEpoch now) * 1000" }}{{ . }}{{ end }} (grafana wants the from/to in ms not s)

Describe alternatives you've considered
I tried all the methods I could find for promQL but none work with logQL, and I wasn't able to iterate over the alerts to get .StartsAt.

@stale
Copy link

stale bot commented Mar 2, 2022

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

  • Mark issues as revivable if we think it's a valid issue but isn't something we are likely
    to prioritize in the future (the issue will still remain closed).
  • Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

@stale stale bot added the stale A stale issue or PR that will automatically be closed. label Mar 2, 2022
@stale stale bot closed this as completed Apr 18, 2022
@fculpo
Copy link

fculpo commented Apr 18, 2023

I also need this in order to build grafana preconfigured URLs

Using Alertmanager .startsAt is not an option as I need to group alerts and each alert in the group must have their own correct URL in annotations (sending to OpsGenie)

@slim-bean slim-bean added keepalive An issue or PR that will be kept alive and never marked as stale. and removed stale A stale issue or PR that will automatically be closed. labels Apr 20, 2023
@slim-bean slim-bean reopened this Apr 20, 2023
@slim-bean slim-bean changed the title Loki Ruler - Provide a way of accessing alert firing timestamp or current time Loki Ruler - Provide a way of accessing alert firing timestamp or current time in annotations Apr 20, 2023
@jmesny
Copy link

jmesny commented Jul 30, 2023

Hello.
I was also trying to produce such Grafana links for my alerts but without success, and it led me here...
Having this possibility would be a great improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
keepalive An issue or PR that will be kept alive and never marked as stale.
Projects
None yet
Development

No branches or pull requests

4 participants