Alerting: No longer index state history log streams by instance labels #65474

alexweav · 2023-03-28T17:12:26Z

What is this feature?

This is a continuation of #65403 where we stop merging instance labels into the log stream labels. Going forward, log streams correspond 1:1 to rules.

Queries for labels will rely on the functionality added in #65403, they now filter the log line itself rather than labels.

Why do we need this feature?

Big reduction in cardinality.
Easier to reason about limits. Number of streams = number of rules, which already has limits.
Stream labels are well-known and predictable.

Which issue(s) does this PR fix?:

n/a

Special notes for your reviewer:

Consider reviewing this PR commit-by-commit

Please check that:

It works as expected from a user's perspective.
If this is a pre-GA feature, it is behind a feature toggle.
The docs are updated, and if this is a notable improvement, it's added to our What's New doc.
There are no known compatibility issues with older supported versions of Grafana, or plugins.
It passes the Hosted Grafana feature readiness review for observability, scalability, performance, and security.

… of straight logQL

grafanabot · 2023-03-28T17:13:02Z

Hello @alexweav!
Backport pull requests need to be either:

Pull requests which address bugs,
Urgent fixes which need product approval, in order to get merged,
Docs changes.

Please, if the current pull request addresses a bug fix, label it with the type/bug label.
If it already has the product approval, please add the product-approved label. For docs changes, please add the type/docs label.
If none of the above applies, please consider removing the backport label and target the next major/minor release.
Thanks!

alexweav · 2023-03-28T17:21:14Z

pkg/services/ngalert/state/historian/loki.go

+type Selector struct {
+	// Label to Select
+	Label string
+	Op    Operator
+	// Value that is expected
+	Value string
+}
+
+func NewSelector(label, op, value string) (Selector, error) {
+	if !isValidOperator(op) {
+		return Selector{}, fmt.Errorf("'%s' is not a valid query operator", op)
+	}
+	return Selector{Label: label, Op: Operator(op), Value: value}, nil
+}
+
+func selectorString(selectors []Selector) string {
+	if len(selectors) == 0 {
+		return "{}"
+	}
+	// Build the query selector.
+	query := ""
+	for _, s := range selectors {
+		query += fmt.Sprintf("%s%s%q,", s.Label, s.Op, s.Value)
+	}
+	// Remove the last comma, as we append one to every selector.
+	query = query[:len(query)-1]
+	return "{" + query + "}"
+}
+
+func isValidOperator(op string) bool {
+	switch op {
+	case "=", "!=", "=~", "!~":
+		return true
+	}
+	return false
+}


copied verbatim from loki_http.go

alexweav · 2023-03-28T17:21:48Z

pkg/services/ngalert/state/historian/loki_test.go

+	t.Run("selector string", func(t *testing.T) {
+		selectors := []Selector{{"name", "=", "Bob"}, {"age", "=~", "30"}}
+		expected := "{name=\"Bob\",age=~\"30\"}"
+		result := selectorString(selectors)
+		require.Equal(t, expected, result)
+
+		selectors = []Selector{}
+		expected = "{}"
+		result = selectorString(selectors)
+		require.Equal(t, expected, result)
+	})
+
+	t.Run("new selector", func(t *testing.T) {
+		selector, err := NewSelector("label", "=", "value")
+		require.NoError(t, err)
+		require.Equal(t, "label", selector.Label)
+		require.Equal(t, Eq, selector.Op)
+		require.Equal(t, "value", selector.Value)
+
+		selector, err = NewSelector("label", "invalid", "value")
+		require.Error(t, err)
+	})


copied from below. not modified

alexweav · 2023-03-28T17:26:43Z

pkg/services/ngalert/state/historian/loki_test.go

-			require.Equal(t, exp, res[0].Stream)
-		})
-
-		t.Run("groups streams based on combined labels", func(t *testing.T) {


not needed any more. the type system now enforces this automatically

gotjosh

LGTM

JohnnyQQQQ

LGTM

grafanabot · 2023-03-29T16:55:41Z

The backport to v9.4.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-65474-to-v9.4.x origin/v9.4.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x a416100abc5f649ada57201cb855d79116f40c8e
# Push it to GitHub
git push --set-upstream origin backport-65474-to-v9.4.x
git switch main
# Remove the local backport branch
git branch -D backport-65474-to-v9.4.x

Then, create a pull request where the base branch is v9.4.x and the compare/head branch is backport-65474-to-v9.4.x.

#65474) * Remove private labels * No longer index by instance labels * Labels are now invariant, only build them once * Remove bucketing since everything is in a single stream * Refactor statesToStreams to only return a single unified log stream * Don't query on labels that no longer exist * Move selector logic to loki layer, genericize client to work in terms of straight logQL * Add support for line-level label filters in query * Combine existing selector tests for better parallelism * Tests for logQL construction * Underscore instead of dot for unwrapping labels in logql (cherry picked from commit a416100)

#65563) Alerting: No longer index state history log streams by instance labels (#65474) * Remove private labels * No longer index by instance labels * Labels are now invariant, only build them once * Remove bucketing since everything is in a single stream * Refactor statesToStreams to only return a single unified log stream * Don't query on labels that no longer exist * Move selector logic to loki layer, genericize client to work in terms of straight logQL * Add support for line-level label filters in query * Combine existing selector tests for better parallelism * Tests for logQL construction * Underscore instead of dot for unwrapping labels in logql (cherry picked from commit a416100)

alexweav added 10 commits March 28, 2023 10:16

Remove private labels

4c0569d

No longer index by instance labels

22f9ef7

Labels are now invariant, only build them once

f591de6

Remove bucketing since everything is in a single stream

8faa3a7

Refactor statesToStreams to only return a single unified log stream

adfb73b

Don't query on labels that no longer exist

15efec2

Move selector logic to loki layer, genericize client to work in terms…

b084ebf

… of straight logQL

Add support for line-level label filters in query

5fa4fd9

Combine existing selector tests for better parallelism

a139b1e

Tests for logQL construction

a26d967

alexweav added area/alerting Grafana Alerting area/backend add to changelog backport v9.4.x Mark PR for automatic backport to v9.4.x labels Mar 28, 2023

alexweav added this to the 9.4.x milestone Mar 28, 2023

alexweav requested a review from a team as a code owner March 28, 2023 17:12

grafanabot added the missing-labels label Mar 28, 2023

alexweav commented Mar 28, 2023

View reviewed changes

gotjosh approved these changes Mar 28, 2023

View reviewed changes

armandgrillet added the product-approved Pull requests that are approved by product/managers and are allowed to be backported label Mar 28, 2023

grafanabot removed the missing-labels label Mar 28, 2023

JohnnyQQQQ approved these changes Mar 29, 2023

View reviewed changes

Underscore instead of dot for unwrapping labels in logql

bc47095

alexweav added the kata:state-history label Mar 29, 2023

alexweav merged commit a416100 into main Mar 29, 2023
9 checks passed

alexweav deleted the alexweav/labels-no-index branch March 29, 2023 16:52

grafanabot added the backport-failed Failed to generate backport PR. Please resolve conflicts and create one manually. label Mar 29, 2023

alexweav mentioned this pull request Mar 29, 2023

Alerting: No longer index state history log streams by instance labels #65563

Merged

5 tasks

zerok modified the milestones: 9.4.x, 9.4.8, 9.5.0 Apr 12, 2023

rwwiv mentioned this pull request Jun 23, 2023

Alerting: Move rule UID from Loki stream labels into log lines #70637

Merged

3 tasks

grafana-delivery-bot bot mentioned this pull request Jun 26, 2023

[v10.0.x] Alerting: Move rule UID from Loki stream labels into log lines #70686

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alerting: No longer index state history log streams by instance labels #65474

Alerting: No longer index state history log streams by instance labels #65474

alexweav commented Mar 28, 2023

grafanabot commented Mar 28, 2023

alexweav Mar 28, 2023

alexweav Mar 28, 2023 •

edited

alexweav Mar 28, 2023

gotjosh left a comment

JohnnyQQQQ left a comment

grafanabot commented Mar 29, 2023

Alerting: No longer index state history log streams by instance labels #65474

Alerting: No longer index state history log streams by instance labels #65474

Conversation

alexweav commented Mar 28, 2023

Consider reviewing this PR commit-by-commit

grafanabot commented Mar 28, 2023

alexweav Mar 28, 2023

Choose a reason for hiding this comment

alexweav Mar 28, 2023 • edited

Choose a reason for hiding this comment

alexweav Mar 28, 2023

Choose a reason for hiding this comment

gotjosh left a comment

Choose a reason for hiding this comment

JohnnyQQQQ left a comment

Choose a reason for hiding this comment

grafanabot commented Mar 29, 2023

alexweav Mar 28, 2023 •

edited