Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alerting: Add alert instance labels to Loki log lines in addition to stream labels #65403

Merged
merged 1 commit into from
Mar 28, 2023

Conversation

alexweav
Copy link
Contributor

What is this feature?

Right now, we merge instance labels with a few pre-defined stream labels like ruleUID and from="state-history".

A consequence of this, is that it becomes impossible to calculate the exact set of labels attached to the corresponding alert instance. For example, the user might have created labels with keys called from, which collides with the stream labels. Then, it's impossible to disambiguate stream labels from what the underlying instance labels were, so it's hard to track the entry back to the original instance.

This PR puts the instance labels, on their own, into the log lines too, under a block called labels. You can query it like

{arbitrary="labels"} | json | labels.myLabelKey="myLabelValue"

Why do we need this feature?

The data that Loki indexes should be separate from the semantics of what actually gets recorded. Helps with querying. This also allows the UI to link an ASH entry back to an instance, which could be useful in the future.

Who is this feature for?

More clear queries of state history rows.

Which issue(s) does this PR fix?:

n/a

Special notes for your reviewer:

Please check that:

  • It works as expected from a user's perspective.
  • If this is a pre-GA feature, it is behind a feature toggle.
  • The docs are updated, and if this is a notable improvement, it's added to our What's New doc.
  • There are no known compatibility issues with older supported versions of Grafana, or plugins.
  • It passes the Hosted Grafana feature readiness review for observability, scalability, performance, and security.

@alexweav alexweav added type/bug area/alerting Grafana Alerting area/backend add to changelog backport v9.4.x Mark PR for automatic backport to v9.4.x labels Mar 27, 2023
@alexweav alexweav added this to the 9.4.x milestone Mar 27, 2023
@alexweav alexweav requested a review from a team March 27, 2023 21:18
Copy link
Contributor

@yuri-tceretian yuri-tceretian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alexweav alexweav merged commit de1637a into main Mar 28, 2023
@alexweav alexweav deleted the alexweav/labels-line branch March 28, 2023 13:57
grafanabot pushed a commit that referenced this pull request Mar 28, 2023
…stream labels (#65403)

Add instance labels to log line

(cherry picked from commit de1637a)
alexweav added a commit that referenced this pull request Mar 28, 2023
…ition to stream labels (#65454)

Alerting: Add alert instance labels to Loki log lines in addition to stream labels (#65403)

Add instance labels to log line

(cherry picked from commit de1637a)

Co-authored-by: Alexander Weaver <weaver.alex.d@gmail.com>
@zerok zerok modified the milestones: 9.4.x, 9.4.8, 9.5.0 Apr 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

4 participants