feat: Add Constraint schema validation testing #2092

willbeason · 2022-06-06T16:42:02Z

Add feature to "gator verify" which allows testing the schema of
Constraints.

Before it was possible to test that Constraints were valid (by simply omitting Cases),
but now it is also possible to test that specific Constraints are invalid. For example -
that fields are not of the correct type, or that required fields are missing.

Fixes: #1989
Signed-off-by: Will Beason willbeason@google.com

codecov-commenter · 2022-06-06T16:47:42Z

Codecov Report

Merging #2092 (d024aa0) into master (c998cea) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2092      +/-   ##
==========================================
- Coverage   54.43%   54.42%   -0.02%     
==========================================
  Files         111      111              
  Lines        9478     9502      +24     
==========================================
+ Hits         5159     5171      +12     
- Misses       3930     3937       +7     
- Partials      389      394       +5

Flag	Coverage Δ
unittests	`54.42% <100.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/gator/runner.go	`87.44% <100.00%> (-9.42%)`	⬇️
pkg/readiness/list.go	`79.41% <0.00%> (-11.77%)`	⬇️
pkg/readiness/object_tracker.go	`83.98% <0.00%> (+1.06%)`	⬆️
...onstrainttemplate/constrainttemplate_controller.go	`58.99% <0.00%> (+2.39%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c998cea...d024aa0. Read the comment docs.

maxsmythe · 2022-06-07T04:15:51Z

pkg/gator/runner.go

-	results, err := r.runCases(ctx, suiteDir, filter, t)
+	err := r.tryAddConstraint(ctx, suiteDir, t)
+	var results []CaseResult
+	if err == nil && !t.Invalid {


It would be more clear to check for the invalid constraint error here and fail the test as-necessary here. That avoids double-checking t.Invalid and makes it easier to reason about what happens if the error is nil and t.Invalid is true.

maxsmythe · 2022-06-23T03:39:39Z

@ritazh @sozercan @chewong LGTY?

Add feature to "gator verify" which allows testing the schema of Constraints. Fixes: open-policy-agent#1989 Signed-off-by: Will Beason <willbeason@google.com>

Signed-off-by: Max Smythe <smythe@google.com>

ritazh · 2022-06-23T22:50:48Z

pkg/gator/errors.go

@@ -38,4 +38,6 @@ var (
 	// ErrConvertingTemplate means we were able to parse a template, but not convert
 	// it into the version-independent format.
 	ErrConvertingTemplate = errors.New("unable to convert template")
+
+	ErrValidConstraint = errors.New("constraint did not fail schema validation")


Suggested change

ErrValidConstraint = errors.New("constraint did not fail schema validation")

ErrValidConstraint = errors.New("constraint failed schema validation")

I think the original is correct. This error is saying that someone wanted a constraint to fail validation but it didn't

wow that was not obvious to me LOL. how about:

Suggested change

ErrValidConstraint = errors.New("constraint did not fail schema validation")

ErrValidConstraint = errors.New("constraint should have failed schema validation")

Also it's missing comments like the rest of the vars

I like it, changing.

@ritazh updated

Signed-off-by: Max Smythe <smythe@google.com>

ritazh

LGTM

* feat: Add Constraint schema validation testing Add feature to "gator verify" which allows testing the schema of Constraints. Fixes: open-policy-agent#1989 Signed-off-by: Will Beason <willbeason@google.com> * fix lint errors and comments Signed-off-by: Max Smythe <smythe@google.com> * tweak error message Signed-off-by: Max Smythe <smythe@google.com> Co-authored-by: Will Beason <willbeason@google.com> Co-authored-by: Max Smythe <smythe@google.com> Co-authored-by: Rita Zhang <rita.z.zhang@gmail.com> Signed-off-by: Christoph Fraundorfer <christoph.fraundorfer@allianzdirect.de>

* feat: Add Constraint schema validation testing Add feature to "gator verify" which allows testing the schema of Constraints. Fixes: open-policy-agent#1989 Signed-off-by: Will Beason <willbeason@google.com> * fix lint errors and comments Signed-off-by: Max Smythe <smythe@google.com> * tweak error message Signed-off-by: Max Smythe <smythe@google.com> Co-authored-by: Will Beason <willbeason@google.com> Co-authored-by: Max Smythe <smythe@google.com> Co-authored-by: Rita Zhang <rita.z.zhang@gmail.com>

* feat: Add Constraint schema validation testing Add feature to "gator verify" which allows testing the schema of Constraints. Fixes: open-policy-agent#1989 Signed-off-by: Will Beason <willbeason@google.com> * fix lint errors and comments Signed-off-by: Max Smythe <smythe@google.com> * tweak error message Signed-off-by: Max Smythe <smythe@google.com> Co-authored-by: Will Beason <willbeason@google.com> Co-authored-by: Max Smythe <smythe@google.com> Co-authored-by: Rita Zhang <rita.z.zhang@gmail.com> Signed-off-by: davis-haba <davishaba@google.com>

willbeason requested review from maxsmythe and a team June 6, 2022 16:42

willbeason self-assigned this Jun 6, 2022

willbeason modified the milestones: Mutation Beta, gator beta Jun 6, 2022

maxsmythe requested changes Jun 7, 2022

View reviewed changes

mrueg mentioned this pull request Jun 22, 2022

chore: Update to opa v0.41 #2093

Merged

maxsmythe force-pushed the invalid-constraint-schema branch 2 times, most recently from dbf18ab to 24ea191 Compare June 23, 2022 02:33

maxsmythe requested review from chewong, ritazh and sozercan June 23, 2022 03:39

maxsmythe approved these changes Jun 23, 2022

View reviewed changes

chewong approved these changes Jun 23, 2022

View reviewed changes

chewong force-pushed the invalid-constraint-schema branch from 24ea191 to 1989b6b Compare June 23, 2022 15:50

Will Beason and others added 2 commits June 23, 2022 11:04

feat: Add Constraint schema validation testing

17790d4

Add feature to "gator verify" which allows testing the schema of Constraints. Fixes: open-policy-agent#1989 Signed-off-by: Will Beason <willbeason@google.com>

fix lint errors and comments

0583470

Signed-off-by: Max Smythe <smythe@google.com>

chewong force-pushed the invalid-constraint-schema branch from 1989b6b to 0583470 Compare June 23, 2022 18:04

ritazh reviewed Jun 23, 2022

View reviewed changes

tweak error message

1fc53b4

Signed-off-by: Max Smythe <smythe@google.com>

maxsmythe requested a review from ritazh June 24, 2022 18:09

ritazh approved these changes Jun 24, 2022

View reviewed changes

Merge branch 'master' into invalid-constraint-schema

d024aa0

maxsmythe merged commit 838ee89 into open-policy-agent:master Jun 24, 2022

eyenx mentioned this pull request Oct 4, 2022

chore(gatekeeper): update to 3.9.* adfinis/helm-charts#824

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Constraint schema validation testing #2092

feat: Add Constraint schema validation testing #2092

willbeason commented Jun 6, 2022

codecov-commenter commented Jun 6, 2022 •

edited

maxsmythe Jun 7, 2022

maxsmythe commented Jun 23, 2022

ritazh Jun 23, 2022

maxsmythe Jun 23, 2022

ritazh Jun 24, 2022

ritazh Jun 24, 2022

maxsmythe Jun 24, 2022

maxsmythe Jun 24, 2022

ritazh left a comment

	ErrValidConstraint = errors.New("constraint did not fail schema validation")
	ErrValidConstraint = errors.New("constraint failed schema validation")

feat: Add Constraint schema validation testing #2092

feat: Add Constraint schema validation testing #2092

Conversation

willbeason commented Jun 6, 2022

codecov-commenter commented Jun 6, 2022 • edited

Codecov Report

maxsmythe Jun 7, 2022

Choose a reason for hiding this comment

maxsmythe commented Jun 23, 2022

ritazh Jun 23, 2022

Choose a reason for hiding this comment

maxsmythe Jun 23, 2022

Choose a reason for hiding this comment

ritazh Jun 24, 2022

Choose a reason for hiding this comment

ritazh Jun 24, 2022

Choose a reason for hiding this comment

maxsmythe Jun 24, 2022

Choose a reason for hiding this comment

maxsmythe Jun 24, 2022

Choose a reason for hiding this comment

ritazh left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jun 6, 2022 •

edited