Rulegroups #2842

gouthamve · 2017-06-14T07:56:34Z

The initial implementation of rule-groups. Some keypoints:

Group name should be unique in a single file.
Promtool can be used to update the rules.
Uses ghodss/yaml instead of go-yaml, I can see that it is already vendored.

cc @brian-brazil @fabxc @beorn7 @juliusv @grobie @SuperQ

This change is

Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

brian-brazil · 2017-06-14T08:06:45Z

cmd/promtool/main.go

@@ -20,8 +20,12 @@ import (
 	"path/filepath"
 	"strings"

+	yaml "github.com/ghodss/yaml"


Why are you using a non-standard yaml library?

I believe the goal was to switch over to this library, as the semantics are much closer to the standard lib encoding/json package, where the Marshal/Unmarshal interface is a lot saner.

For the better or worse, the fact that it is already vendored is thanks to kubernetes/client-go.

That's still under discussion, I don't believe we got to consensus there.

It was an experiment of mine in the first commit. Turns out, error messages for YAML parsing errors are a lot clearer. Validation and custom marshalling becomes tons better.

I haven't seen any drawbacks yet.

It's just a small wrapper btw.

It's a wrapper. It makes error reporting better, it allows us to do marshalling saner. We should probably migrate other places to it. Look at our config packages. That's where the insanity lies.

If we're going to migrate we should complete the discussion elsewhere and then migrate. Introducing things only in new code has a tendency to end up with a split world for rather a long time.

It's vendored already, it's entirely local to the package and gives us an immediate benefit. Any config/ refactoring is far away. I see no reason to not do this.

Changing the rest of our code to also use this new library being something that is far away is a strong reason not to do this.

We do not have consensus on this, and using multiple different libraries within our organisation that do the same thing is to be avoided.

Moved back to go-yaml. PTAL.

brian-brazil · 2017-06-14T08:06:45Z

pkg/rulefmt/testdata/test.yaml

@@ -0,0 +1,59 @@
+version: 1


Do we really need a version number at this stage? It's noise.

brian-brazil · 2017-06-14T08:06:45Z

pkg/rulefmt/testdata/test.yaml

+  - alert: HighErrors
+    expr: |
+      sum without(instance) (rate(errors_total[5m]))
+      / sum without(instance) (rate(requests_total[5m]))


nit:

sum without(instance) (rate(errors_total[5m])) / sum without(instance) (rate(requests_total[5m]))

I think YAML doesn't get it if the indentation on the first line is offset by this =/ one of the warts.

Ugh, is this confirmed? Any workaround? This would be a serious drawback in readability.

I did update the file now to reflect something better. Everything has to be in the same indentation level.

That kind of constraint goes straight to my worries about nesting one kind of language in another one, especially if the outer one is one that cares about white space like this. I don't know what conclusion to draw from this though, as there might not be a better way.

brian-brazil · 2017-06-14T08:06:45Z

rules/manager.go

@@ -270,18 +271,10 @@ func typeForRule(r Rule) ruleType {
 // In the future a single group will be evaluated sequentially to properly handle


Comment needs updating

brian-brazil · 2017-06-14T08:06:45Z

rules/manager.go

@@ -270,18 +271,10 @@ func typeForRule(r Rule) ruleType {
 // In the future a single group will be evaluated sequentially to properly handle
 // rule dependency.
 func (g *Group) Eval(ts time.Time) {


The different rule groups need to have offset runtimes, like we do for targets.

brian-brazil · 2017-06-14T08:06:45Z

rules/manager.go

 			}
-			rules = append(rules, rule)
+
+			// Groups need not be unique across filenames.


What is this going to look like in the UI and in metrics?

Will the labels of a group change every time it's rescheduled in a different directory?

I don't think non-unique names works with our current setup.

What do you mean labels?

I haven't yet touched the UI. But it would be simple to add the grouping there as I now added the filename also to the Group object. Currently in the UI, all the rules will be listed without any reference to the group/file.

We will have a metric for how long the rule groups took to evaluate. The labels of a given group in that metric should be consistent across Prometheus invocations.

Can we not have name, filename as the labels?

The filename can change as Prometheus is rescheduled.

brian-brazil · 2017-06-14T08:06:45Z

pkg/rulefmt/rulefmt.go

+}
+
+// Validate the rule and return a list of encountered errors.
+func (r *Rule) Validate() (errs []error) {


You need to validate that the labelnames/annotations names are valid, and for rules that label values are valid.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

fabxc · 2017-06-16T09:13:14Z

rules/manager.go

-			return nil, err
+		rgs, errs := rulefmt.ParseFile(fn)
+		if errs != nil {
+			return nil, tsdb.MultiError(errs)


Not a good idea to pull in tsdb as a dep just for a multierror type. In fact, it's probably a good idea to have loadGroups also just returns a straight slice of errors so the caller can print a single log line per encountered issue or similar. That's why we made it a slice in rulefmt to begin with.

fabxc · 2017-06-16T09:13:14Z

rules/manager.go

+	l := model.LabelSet{
+		"name":     model.LabelValue(g.name),
+		"filename": model.LabelValue(g.file),
+	}
 	return l.Fingerprint()


Can we rename it to hash() and use pkg/labels.Labels.Hash()?

fabxc · 2017-06-16T09:13:14Z

cmd/promtool/main.go

@@ -171,16 +176,96 @@ func checkRules(t cli.Term, filename string) (int, error) {
 		return 0, fmt.Errorf("is a directory")
 	}

+	rgs, errs := rulefmt.ParseFile(filename)
+	if errs != nil {
+		return 0, tsdb.MultiError(errs)


Same about import tsdb for this and probably better returning the actual slice so the errors can be logged as one per line. tsdb.MultiError will just concat them with ; which will be impossible to debug with.

fabxc · 2017-06-16T09:13:14Z

cmd/promtool/main.go

 	}
-	return len(rules), nil
+
+	return ioutil.WriteFile(filename+".yaml", y, 0777)


0666 for files, 0777 for dirs. That's what we are going with in general I think.

fabxc · 2017-06-16T09:13:14Z

cmd/promtool/main.go

+	}
+
+	yamlRG.Groups[0].Rules = yamlRules
+	y, err := yaml.Marshal(yamlRG)


Because we are now using the plain YAML lib again this will generate rule files with basically random key ordering right?

@brian-brazil if yes, this is pretty bad UX just because we don't want to use a 100 LOC wrapper lib that's vendored already anyway.

Using go-yaml the fields are ordered as we declare them. ghodss/yaml produces the fields in alphabetical order.

Okay, let's move forward then. Somewhat sure I've seen things wildely mixed – but maybe I recall incorrectly.

brian-brazil

The offset logic isn't included here yet.

brian-brazil · 2017-06-16T10:14:19Z

cmd/promtool/main.go

@@ -243,6 +328,11 @@ func main() {
 		Run:  CheckMetricsCmd,
 	})

+	app.Register("update-rules", &cli.Command{
+		Desc: "update the rules to the new YAML format",


update rule files to...

to be consistent with the above.

brian-brazil · 2017-06-16T10:14:19Z

pkg/rulefmt/rulefmt.go

+type RuleGroup struct {
+	Name     string         `yaml:"name"`
+	Interval model.Duration `yaml:"interval,omitempty"`
+	Rules    []Rule         `yaml:"rules"`


This should have the XXX checkOverflow stuff.

brian-brazil · 2017-06-16T10:14:19Z

rules/manager.go

-	var (
-		wg sync.WaitGroup
-	)
-
 	for i, rule := range g.rules {


Should we check done after each rule?

I don't see why we need to. This is all going sequentially and the function wouldn't exit until all the rules are evaluated.

I think Brian's point was to actually do exit if done indicates the group should terminate.

* Move fingerprint to Hash() * Move away from tsdb.MultiError * 0777 -> 0666 for files * checkOverflow of extra fields Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

fabxc · 2017-06-16T11:15:48Z

I don't think we need offset evaluation for the first iteration. Let's focus on working down breaking changes for now.

gouthamve · 2017-06-16T11:23:02Z

@brian-brazil This takes care of the offset no? https://github.com/Gouthamve/prometheus/blob/6b70a4d85031ecfb1404c3702237ebc0e7d3f2c8/rules/manager.go#L169

fabxc · 2017-06-16T11:26:58Z

Mh, I thought this was alluding to "offset evaluation" where we evaluate recording rules not against now but now-5m for example. That can be relevant for metrics that are lagging behind like Cloudwatch.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

brian-brazil · 2017-06-16T11:36:20Z

No, that's what I was talking about. I'd thought it hadn't been merged into this PR.

My "offset eval" is still at the idea stage, I want to wait until we've got cloudwatch&friends outputting timestamps and see if it still makes sense.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

fabxc · 2017-06-19T11:27:35Z

👍

fabxc and others added 7 commits Jun 7, 2017

pkg/rulefmt: Add rule group parsing

Loading status checks…

c843a0c

Move rules to new format

e8f5566

Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>

Add update-rules command to promtool

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

cea1e99

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Make sure groups are unique in a single file

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

a48a018

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Add tests

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

Loading status checks…

2d1e925

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Update check-rules to new format.

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

1c08743

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Add License Header

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

Loading status checks…

c884881

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

brian-brazil reviewed Jun 14, 2017

View changes

gouthamve added 4 commits Jun 14, 2017

Validate labels and annotations

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

e893c89

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Add file name to group.

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

Loading status checks…

8cca666

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Reflect the grouping in the UI

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

Loading status checks…

5ff283a

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Move back to go-yaml

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

Loading status checks…

dc69645

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

fabxc reviewed Jun 16, 2017

View changes

brian-brazil reviewed Jun 16, 2017

View changes

gouthamve added 2 commits Jun 19, 2017

Merge remote-tracking branch 'upstream/dev-2.0' into rulegroups

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

37e7b69

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

Remove version from RuleGroups

Verified

This commit was signed with a verified signature.

gouthamve Goutham Veeramachaneni

GPG key ID: F1C217E8E9023CAD Learn about signing commits

Loading status checks…

592cb00

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

fabxc merged commit ab1bc9b into prometheus:dev-2.0 Jun 19, 2017
2 of 3 checks passed

2 of 3 checks passed

ci/circleci CircleCI is running your tests
Details

codacy/pr Good work! A positive pull request.
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

		@@ -270,18 +271,10 @@ func typeForRule(r Rule) ruleType {
		// In the future a single group will be evaluated sequentially to properly handle

prometheus / prometheus

Rulegroups #2842

Rulegroups #2842

Conversation

gouthamve commented Jun 14, 2017 • edited

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

brancz Jun 14, 2017 Member

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

fabxc Jun 14, 2017 Contributor

This comment has been minimized.

fabxc Jun 14, 2017 Contributor

This comment has been minimized.

fabxc Jun 14, 2017 Contributor

This comment has been minimized.

brian-brazil Jun 14, 2017 • edited Contributor

This comment has been minimized.

fabxc Jun 15, 2017 Contributor

This comment has been minimized.

brian-brazil Jun 15, 2017 Contributor

This comment has been minimized.

gouthamve Jun 16, 2017 Author Member

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

fabxc Jun 14, 2017 Contributor

This comment has been minimized.

grobie Jun 14, 2017 Member

This comment has been minimized.

gouthamve Jun 14, 2017 Author Member

This comment has been minimized.

juliusv Jun 15, 2017 Member

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

gouthamve Jun 14, 2017 Author Member

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

gouthamve Jun 14, 2017 Author Member

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

brian-brazil Jun 14, 2017 Contributor

This comment has been minimized.

fabxc Jun 16, 2017 Contributor

This comment has been minimized.

fabxc Jun 16, 2017 Contributor

This comment has been minimized.

fabxc Jun 16, 2017 Contributor

This comment has been minimized.

fabxc Jun 16, 2017 Contributor

This comment has been minimized.

fabxc Jun 16, 2017 • edited Contributor

This comment has been minimized.

gouthamve Jun 16, 2017 Author Member

This comment has been minimized.

fabxc Jun 16, 2017 Contributor

brian-brazil left a comment

This comment has been minimized.

brian-brazil Jun 16, 2017 Contributor

This comment has been minimized.

brian-brazil Jun 16, 2017 Contributor

This comment has been minimized.

brian-brazil Jun 16, 2017 Contributor

This comment has been minimized.

gouthamve Jun 16, 2017 Author Member

This comment has been minimized.

fabxc Jun 16, 2017 Contributor

This comment has been minimized.

gouthamve Jun 16, 2017 Author Member

gouthamve commented Jun 14, 2017 •

edited

brian-brazil Jun 14, 2017
Contributor

brancz Jun 14, 2017
Member

brian-brazil Jun 14, 2017
Contributor

fabxc Jun 14, 2017
Contributor

fabxc Jun 14, 2017
Contributor

fabxc Jun 14, 2017
Contributor

brian-brazil Jun 14, 2017 •

edited

Contributor

fabxc Jun 15, 2017
Contributor

brian-brazil Jun 15, 2017
Contributor

gouthamve Jun 16, 2017
Author Member

brian-brazil Jun 14, 2017
Contributor

brian-brazil Jun 14, 2017
Contributor

fabxc Jun 14, 2017
Contributor

grobie Jun 14, 2017
Member

gouthamve Jun 14, 2017
Author Member

juliusv Jun 15, 2017
Member

brian-brazil Jun 14, 2017
Contributor

brian-brazil Jun 14, 2017
Contributor

brian-brazil Jun 14, 2017
Contributor

gouthamve Jun 14, 2017
Author Member

brian-brazil Jun 14, 2017
Contributor

gouthamve Jun 14, 2017
Author Member

brian-brazil Jun 14, 2017
Contributor

brian-brazil Jun 14, 2017
Contributor

fabxc Jun 16, 2017
Contributor

fabxc Jun 16, 2017
Contributor

fabxc Jun 16, 2017
Contributor

fabxc Jun 16, 2017
Contributor

fabxc Jun 16, 2017 •

edited

Contributor

gouthamve Jun 16, 2017
Author Member

fabxc Jun 16, 2017
Contributor

brian-brazil Jun 16, 2017
Contributor

brian-brazil Jun 16, 2017
Contributor

brian-brazil Jun 16, 2017
Contributor

gouthamve Jun 16, 2017
Author Member

fabxc Jun 16, 2017
Contributor

gouthamve Jun 16, 2017
Author Member