Setup test fixtures for resolvers #5317

egor-ryashin · 2024-07-20T12:20:48Z

No description provided.

begelundmuller

Some other comments:

Put .yaml and .csv files into a separate runtime/resolvers/testdata directory
Add support for ClickHouse and Druid tests
Do you think it makes sense to have some kind of --update flag that will write out resolver results to the test files? To make it easier to author new tests. E.g. see this for an example: https://ieftimov.com/posts/testing-in-go-golden-files/

runtime/resolvers/resolvers_test.go

begelundmuller · 2024-07-22T09:25:42Z

runtime/security.go

 type SecurityClaims struct {
 	// UserAttributes about the current user (or service account). Usually exposed through templating as {{ .user }}.
-	UserAttributes map[string]any
+	UserAttributes map[string]any `yaml:"user_attributes"`
 	// AdditionalRules are optional security rules to apply *in addition* to the built-in rules and the rules defined on the requested resource.
 	// These are currently leveraged by the admin service to enforce restrictions for magic auth tokens.
 	AdditionalRules []*runtimev1.SecurityRule


Can we remove the yaml parsing from this type (can be handled with a separate type in the tests)? It makes it look like this type can be serialized/deserialized to YAML, which it can't (because the proto type runtimev1.SecurityRule is not YAML compatible)

I wonder why we cannot do it YAML compatible, more copies of structs - more structs to maintain - harder to maintain when 2 structs are almost equal (finding the distinction is harder) - more error-prone.

As written above, the *runtimev1.SecurityRule proto-generated type is not safe for use with YAML, so we don't want to give the impression that the SecurityClaims type is YAML-serializable. Additionally, it's just a bad practice to hack an internal type that used in many places for a single specific use case (in this case, test fixtures) – it creates unnecessary coupling.

In this case, UserAttributes is the only field we need to be able to populate from the tests, so it is simple to parse it from the test fixture and create a SecurityClaims value for it.

begelundmuller · 2024-07-22T09:26:22Z

runtime/resolvers/resolvers_test.go

+type Project struct {
+	Sources    map[string]yaml.Node
+	Models     map[string]yaml.Node
+	Dashboards map[string]yaml.Node
+	APIs       map[string]yaml.Node
+}


Instead of hard-coding the types, can we just use full paths as key types? We support quite a lot more resource types than just these four now. See this comment:
#5218 (comment)

I personally don't see much difference:

sources: 1.yaml 2.yaml dashboards: 1.yaml 2.yaml

- sources/1.yaml: - sources/2.yaml: - dashboards/1.yaml: - dashboards/2.yaml:

We can easily add a new field to Project struct if more resources is required.
That could make sense if you'd like to process all files equally with a single for-loop and it can save a few lines of code but you'll have to repeat yourself if you need multiple similar-type entries in the future, ie:

- sources/1.yaml: - sources/2.yaml: - dashboards/1.yaml: - dashboards/2.yaml:

IMO reducing nesting one level for all the test files definitely improves readability here. The very slight repetition of the directory name does not hurt readabilty IMO, and anyway we don't require specific directory names anymore, so the files can just be created at the root level.

This also has the additional benefit of simplifying the test fixture parsing code.

runtime/resolvers/resolvers_test.go

begelundmuller · 2024-07-22T09:46:07Z

runtime/resolvers/resolvers_test.yaml

+    options:
+      claims:
+        user_attributes:


For the sake of cleaner tests, I think we should flatten the nesting here and just have user_attributes as a root property of the test type

It relates to the previous argument If you like to have more test structures then it's less production code coverage.

I understand that we would like to see a more concise schema but it's already have a diverse structure, simplifying it is short term goal - eventually we will need to test more production fields anyway.

This doesn't impact production code coverage at all – it just makes the test fixture simpler to read/write. Internally the test fixture parser would create a SecurityClaims value, so the production code coverage is exactly the same.

nishantmonu51 · 2024-07-31T19:10:52Z

@egor-ryashin : Any updates on this PR ?

egor-ryashin · 2024-08-01T08:11:33Z

@nishantmonu51 Most of comments I'll resolve in 1d, the first one "updates" can take longer.

egor-ryashin · 2024-08-01T19:47:42Z

Put .yaml and .csv files into a separate runtime/resolvers/testdata directory

Add support for ClickHouse and Druid tests

Do you think it makes sense to have some kind of --update flag that will write out resolver results to the test files? To make it easier to author new tests. E.g. see this for an example: https://ieftimov.com/posts/testing-in-go-golden-files/

Done (replaced with the public https source - no need to commit data).
Done.

egor-ryashin · 2024-08-02T14:04:21Z

Merge conflicts resolved.

egor-ryashin · 2024-08-05T21:13:58Z

Updates having this issue go-yaml/yaml#430
Floats are overwritten with integers breaking unit tests consequently, trying to workaround.

egor-ryashin · 2024-08-07T16:53:00Z

Updates is done.

begelundmuller · 2024-08-12T09:49:57Z

go.mod

@@ -261,6 +261,7 @@ require (
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/mattn/go-runewidth v0.0.15 // indirect
 	github.com/mattn/go-sqlite3 v2.0.3+incompatible // indirect
+	github.com/maxatome/go-testdeep v1.14.0 // indirect


Indirect dependency added without change to a direct dependency – try running go mod tidy

begelundmuller · 2024-08-12T09:51:40Z

runtime/drivers/druid/druidsqldriver/druid_api_sql_driver.go

+		if strings.Contains(err.Error(), c.dsn) { // avoid returning the actual DSN with the password which will be logged
+			return nil, fmt.Errorf("invalid dsn")
+		}


Perhaps return the original error, but with something like strings.Replace(errMsg, c.dsn, "<masked>")?

begelundmuller · 2024-08-12T10:02:36Z

runtime/resolvers/clickhouse_resolvers_test.yaml

@@ -0,0 +1,91 @@
+project:


Please move the .yaml test fixtures into a separate directory at runtime/resolvers/testdata since the Go docs recommend putting non-Go test files in a separate testdata directory

begelundmuller · 2024-08-12T10:03:28Z

runtime/testruntime/testruntime.go

+	// logger := zap.NewNop()
 	// nolint
-	// logger, err := zap.NewDevelopment()
-	// require.NoError(t, err)
+	logger, err := zap.NewDevelopment()
+	require.NoError(t, err)


Please revert before merging – makes the CI test output difficult to read

begelundmuller · 2024-08-12T10:06:59Z

runtime/resolvers/resolvers_test.go

+func TestResolvers(t *testing.T) {
+
+	entries, err := os.ReadDir("./")


nit: redundant newline

begelundmuller · 2024-08-12T10:40:12Z

runtime/testruntime/testruntime.go

+	InstanceOptions
+	OLAPDriver string
+	OLAPDSN    string
+	TempDir    string


This appears unused

begelundmuller · 2024-08-12T10:50:08Z

runtime/security.go

 type SecurityClaims struct {
 	// UserAttributes about the current user (or service account). Usually exposed through templating as {{ .user }}.
-	UserAttributes map[string]any
+	UserAttributes map[string]any `yaml:"user_attributes"`
 	// AdditionalRules are optional security rules to apply *in addition* to the built-in rules and the rules defined on the requested resource.
 	// These are currently leveraged by the admin service to enforce restrictions for magic auth tokens.
 	AdditionalRules []*runtimev1.SecurityRule


As written above, the *runtimev1.SecurityRule proto-generated type is not safe for use with YAML, so we don't want to give the impression that the SecurityClaims type is YAML-serializable. Additionally, it's just a bad practice to hack an internal type that used in many places for a single specific use case (in this case, test fixtures) – it creates unnecessary coupling.

In this case, UserAttributes is the only field we need to be able to populate from the tests, so it is simple to parse it from the test fixture and create a SecurityClaims value for it.

begelundmuller · 2024-08-12T10:54:14Z

runtime/resolvers/resolvers_test.go

+type Project struct {
+	Sources    map[string]yaml.Node
+	Models     map[string]yaml.Node
+	Dashboards map[string]yaml.Node
+	APIs       map[string]yaml.Node
+}


IMO reducing nesting one level for all the test files definitely improves readability here. The very slight repetition of the directory name does not hurt readabilty IMO, and anyway we don't require specific directory names anymore, so the files can just be created at the root level.

This also has the additional benefit of simplifying the test fixture parsing code.

begelundmuller · 2024-08-12T10:55:20Z

runtime/resolvers/resolvers_test.yaml

+    options:
+      claims:
+        user_attributes:


This doesn't impact production code coverage at all – it just makes the test fixture simpler to read/write. Internally the test fixture parser would create a SecurityClaims value, so the production code coverage is exactly the same.

begelundmuller · 2024-08-12T10:58:08Z

runtime/resolvers/resolvers_test.go

+type Resolvers struct {
+	Project    Project
+	Connectors map[string]*testruntime.InstanceOptionsForResolvers
+	Tests      map[string]*Test
+}
+
+type Test struct {
+	Resolver string
+	Options  runtime.ResolveOptions
+	Result   []map[string]any
+}


I strongly believe it would simplify the code and readability here to combine this into a single struct with anonymous sub-structs to represent the YAML structure – and avoid using internal types.

This is also what we do for the resource YAML parsing, and it serves us well. Look at the AlertYAML type here as an example:

rill/runtime/compilers/rillv1/parse_alert.go

Line 18 in d66a28f

type AlertYAML struct {

OK, but I'll need to copy all the data to ResolveOptions to call a method here:

res, err := rt.Resolve(context.Background(), &ropts)

I see SecurityClaims have a oneof field, now it makes sense.

egor-ryashin · 2024-08-21T19:24:19Z

All done.

begelundmuller · 2024-08-22T14:40:48Z

Can you take a look at the CI failures?

egor-ryashin · 2024-08-23T10:06:22Z

There's a weird error currently

        	            	rill.runtime.v1.Model/ad_bids_mini: failed to create model: code: 27, message: Cannot parse input: expected ',' before: 'Z,,msn.com,2,1,2,cars,,iphone\n2,2022-01-03T14:49:50.459Z,,msn.com,2.5,10,1,cars,,iphone\n1,2022-01-02T11:58:12.475Z,Yahoo,yahoo.com,2,100,1,cars,1,\n3,2022-01-04T': (at row 1)

It's not reproduced locally with the same docker image.

egor-ryashin · 2024-09-03T09:37:00Z

There's a new exception emerged after the merge:

ORDER BY or PRIMARY KEY clause is missing. Consider using extended storage definition syntax with ORDER BY or PRIMARY KEY clause. With deprecated old syntax (highly not recommended) storage MergeTree requires 3 to 4 parameters: 
            	            	name of column with date,
            	            	[sampling element of primary key],
            	            	primary key expression,
            	            	index granularity
            	            	
            	            	Syntax for the MergeTree table engine:
            	            	
            	            	CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]
            	            	(
            	            	    name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1] [TTL expr1],

I'm figuring that out.

egor-ryashin · 2024-09-03T14:17:47Z

Solved here #5591

egor-ryashin · 2024-09-05T14:15:26Z

All done.

resolvers-test-fixture

698fcc1

egor-ryashin requested a review from begelundmuller July 20, 2024 12:20

egor-ryashin mentioned this pull request Jul 20, 2024

Runtime: Setup test fixtures for resolvers and migrate existing metrics tests #5218

Closed

begelundmuller requested changes Jul 22, 2024

View reviewed changes

egor-ryashin added 2 commits August 1, 2024 15:13

resolvers tests - clickhouse

c76d60c

resolvers tests - druid

4836668

egor-ryashin added 3 commits August 2, 2024 11:42

resolvers tests - style

c5e1560

Merge remote-tracking branch 'origin/main' into resolvers-test-fixture

eb344c1

resolvers tests - merge fix

eeab03b

egor-ryashin added 2 commits August 2, 2024 19:18

resolvers tests - merge fix

4b7baa6

resolvers tests - style

7979b7d

egor-ryashin requested a review from begelundmuller August 2, 2024 17:05

egor-ryashin marked this pull request as ready for review August 5, 2024 11:12

resolvers tests - druid

a980ad2

resolvers tests - updates

a2989ec

begelundmuller requested changes Aug 12, 2024

View reviewed changes

egor-ryashin added 8 commits August 12, 2024 20:58

resolvers tests - go.mod

7fce8af

resolvers tests - masked

a9d5ec7

resolvers tests - testdata

ead0534

resolvers tests - zap

0dd4820

resolvers tests - filepath

5976e37

resolvers tests - reconcile errors

a028a8a

resolvers tests - reconcile errors

2246de9

resolvers tests - reconcile errors

27456cd

egor-ryashin added 4 commits August 21, 2024 21:53

resolvers tests - csv

578e919

resolvers tests - error update

4ffb84c

resolvers tests - gocritic

ac3bb1c

resolvers tests - filepath

3d9ad74

resolvers tests - style

10d5848

egor-ryashin added 3 commits September 2, 2024 18:55

resolvers tests - clickhouse image

4f3da82

Merge remote-tracking branch 'origin/main' into resolvers-test-fixture

a457f04

merge fix

5b51e56

egor-ryashin added 2 commits September 3, 2024 16:55

clickhouse image version

594ba0c

Merge remote-tracking branch 'origin/main' into resolvers-test-fixture

1da4e0d

egor-ryashin added 6 commits September 4, 2024 21:21

resolvers tests - CH CSV ISO ts parsing

5188c19

resolvers tests - .env

ddf6c00

resolvers tests - url mask

ba55ff7

resolvers tests - .env fail fast

2191e3c

resolvers tests - testClaims

bd11fd8

resolvers tests - testClaims

d38f6b0

egor-ryashin requested a review from begelundmuller September 5, 2024 09:36

begelundmuller approved these changes Sep 19, 2024

View reviewed changes

begelundmuller merged commit 370c29d into main Sep 19, 2024
7 checks passed

begelundmuller deleted the resolvers-test-fixture branch September 19, 2024 11:52

		func TestResolvers(t *testing.T) {

		entries, err := os.ReadDir("./")

Setup test fixtures for resolvers #5317

Setup test fixtures for resolvers #5317

Conversation

egor-ryashin commented Jul 20, 2024

begelundmuller left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nishantmonu51 commented Jul 31, 2024

egor-ryashin commented Aug 1, 2024

egor-ryashin commented Aug 1, 2024

egor-ryashin commented Aug 2, 2024

egor-ryashin commented Aug 5, 2024

egor-ryashin commented Aug 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

egor-ryashin Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

egor-ryashin Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

egor-ryashin commented Aug 21, 2024

begelundmuller commented Aug 22, 2024

egor-ryashin commented Aug 23, 2024

egor-ryashin commented Sep 3, 2024 • edited Loading

egor-ryashin commented Sep 3, 2024

egor-ryashin commented Sep 5, 2024

egor-ryashin commented Aug 7, 2024 •

edited

Loading

egor-ryashin Aug 15, 2024 •

edited

Loading

egor-ryashin Aug 15, 2024 •

edited

Loading

egor-ryashin commented Sep 3, 2024 •

edited

Loading