migrate auth_test to common #14041

chaochn47 · 2022-05-13T22:59:53Z

Related to #13637

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1202764322183583

chaochn47 · 2022-05-13T23:02:46Z

@serathius Can I get some early feebacks if the common test framework auth implementation makes sense to you? Thanks!

codecov-commenter · 2022-05-13T23:30:47Z

Codecov Report

Merging #14041 (d5f2696) into main (9cc2f64) will decrease coverage by 1.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #14041      +/-   ##
==========================================
- Coverage   75.80%   74.65%   -1.16%     
==========================================
  Files         457      457              
  Lines       37421    37391      -30     
==========================================
- Hits        28367    27914     -453     
- Misses       7294     7747     +453     
+ Partials     1760     1730      -30

Flag	Coverage Δ
all	`74.65% <ø> (-1.16%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
etcdctl/ctlv3/command/txn_command.go	`6.14% <0.00%> (-57.02%)`	⬇️
etcdctl/ctlv3/command/defrag_command.go	`26.92% <0.00%> (-50.00%)`	⬇️
server/proxy/grpcproxy/auth.go	`44.44% <0.00%> (-44.45%)`	⬇️
etcdctl/ctlv3/command/printer_simple.go	`19.13% <0.00%> (-33.96%)`	⬇️
server/etcdserver/api/v3rpc/auth.go	`55.43% <0.00%> (-26.09%)`	⬇️
etcdctl/ctlv3/command/ep_command.go	`26.11% <0.00%> (-25.56%)`	⬇️
server/etcdserver/apply/apply_auth.go	`71.00% <0.00%> (-17.00%)`	⬇️
etcdctl/ctlv3/command/auth_command.go	`54.23% <0.00%> (-16.95%)`	⬇️
etcdctl/ctlv3/command/role_command.go	`61.29% <0.00%> (-9.68%)`	⬇️
server/etcdserver/apply/apply.go	`73.39% <0.00%> (-9.18%)`	⬇️
... and 30 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

ahrtr · 2022-05-14T01:18:55Z

cc @mitake

tests/framework/e2e/etcdctl.go

tests/framework/integration.go

serathius · 2022-05-15T13:26:09Z

tests/framework/integration/cluster.go

@@ -179,6 +179,8 @@ type Cluster struct {

 	mu            sync.Mutex
 	clusterClient *clientv3.Client
+
+	clusterAuthClients []*clientv3.Client


There is already a lot of different types of clients stored just to call .Close(), maybe we should just store list destructor functions? Even better would be to use t.Cleanup(func()) for call client.Close.

maybe we should just store list destructor functions

I think that's a good idea. Maybe we could optimize it in a follow-up PR?

I mean this PR could just avoid introducing clusterAuthClients []*clientv3.Client, by using cleanup []func(). We can migrate others in future.

okay. Make sense.

tests/framework/interface.go

chaochn47 · 2022-08-04T07:52:20Z

The failed test is unrelated to the PR.

--- FAIL: TestDowngradeUpgradeClusterOf3 (41.84s)

{
	"level": "warn",
	"ts": "2022-08-04T07:33:45.478Z",
	"caller": "embed/serve.go:120",
	"msg": "stopping grpc server due to error",
	"error": "accept tcp 127.0.0.1:20000: use of closed network connection"
}

ref. https://github.com/etcd-io/etcd/runs/7667198807?check_suite_focus=true

The symptom looks like similar to #14275.

However, I think this type of flake is inevitable without embed server restart retry given we run multiple tests in parallel. Kernel and tests races on TCP connection closing, opening.

See this test causes 41.84s while waiting for client connecting to the dead server. Retry server should prevent this type of flake. Do you have any thoughts on this? @serathius

ahrtr · 2022-08-08T21:46:06Z

Thanks @chaochn47 for the PR. More than 2K line of code change is too big, could you try to break down this PR into small ones?

server/auth/store.go

tests/common/alarm_test.go

ahrtr · 2022-09-05T07:18:01Z

Sorry for the late response. Are you still working on this PR?

chaochn47 · 2022-09-06T17:05:49Z

Thanks @ahrtr for the reminder

Yeah, I am working on the second common test framework changes mentioned above. #14331 but a little confused how to address the last comment raised by Merek. Will keep the PR updated.

chaochn47 · 2022-10-26T05:01:29Z

The latest code is on top of #14626, once it is merged, rebase will make the code diff smaller.

chaochn47 · 2022-11-22T03:12:26Z

Impacted by a flaky test case

=== FAIL: integration TestPeriodicCheckDetectsCorruption
    corrupt_test.go:100: 
        	Error Trace:	corrupt_test.go:100
        	Error:      	Not equal: 
        	            	expected: []*etcdserverpb.AlarmMember{(*etcdserverpb.AlarmMember)(0xc00bb8f9b0)}
        	            	actual  : []*etcdserverpb.AlarmMember{(*etcdserverpb.AlarmMember)(0xc00bb8f950)}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,3 +2,3 @@
        	            	  (*etcdserverpb.AlarmMember)({
        	            	-  MemberID: (uint64) 12970263741109204454,
        	            	+  MemberID: (uint64) 11662817823488512165,
        	            	   Alarm: (etcdserverpb.AlarmType) 2,
        	Test:       	TestPeriodicCheckDetectsCorruption

fuweid · 2022-11-22T03:18:10Z

Impacted by a flaky test case


=== FAIL: integration TestPeriodicCheckDetectsCorruption

    corrupt_test.go:100: 

        	Error Trace:	corrupt_test.go:100

        	Error:      	Not equal: 

        	            	expected: []*etcdserverpb.AlarmMember{(*etcdserverpb.AlarmMember)(0xc00bb8f9b0)}

        	            	actual  : []*etcdserverpb.AlarmMember{(*etcdserverpb.AlarmMember)(0xc00bb8f950)}

        	            	

        	            	Diff:

        	            	--- Expected

        	            	+++ Actual

        	            	@@ -2,3 +2,3 @@

        	            	  (*etcdserverpb.AlarmMember)({

        	            	-  MemberID: (uint64) 12970263741109204454,

        	            	+  MemberID: (uint64) 11662817823488512165,

        	            	   Alarm: (etcdserverpb.AlarmType) 2,

        	Test:       	TestPeriodicCheckDetectsCorruption

Found that root cause and will file pr later.

Signed-off-by: Chao Chen <chaochn@amazon.com>

tests/common/auth_test.go

ahrtr · 2022-11-23T06:09:20Z

This PR is too big. So it is almost impossible to carefully check each line of code change. But overall looks clear and good to me.

Please try to use github.com/stretchr/testify to simplify the code.

Suggestion for similar PR in future: Please try to migrate the cases step by step.

I am OK if other maintainers agree to merge this PR. So defer to @serathius and @spzala to merge or comment this PR.

ahrtr · 2022-11-23T06:15:48Z

Another point is that please try to reuse the common functions in auth_util.go. If the existing functions do not meet some requirements, then please try to enhance the common functions, so that we can simplify the test cases as much as possible.

serathius · 2022-11-23T08:54:28Z

tests/e2e/ctl_v3_auth_test.go

-	}
-}
-
-func authCredWriteKeyTest(cx ctlCtx) {


Cant find this tests. Was it migrated under new name or removed?

It was migrated to TestAuthWriteKey.

serathius · 2022-11-23T09:06:09Z

tests/common/auth_test.go

+		assert.Equal(t, 3, len(memberList.Members), "want 3 member but got %d", len(memberList.Members))
+		id := memberList.Members[0].ID
+
+		// 5 seconds is the minimum required amount of time peer is considered active


Where this come from? I don't see it in original authTestMemberRemove code

Looks like it's not needed.

serathius · 2022-11-23T09:09:23Z

tests/common/auth_test.go

+func TestAuthLeaseTestTimeToLiveExpired(t *testing.T) {
+	tcs := []struct {
+		name       string
+		JWTEnabled bool


I don't see this used.

Thanks for catching that. Will fix this.

serathius · 2022-11-23T09:15:34Z

tests/common/auth_test.go

+	}
+}
+
+func TestAuthLeaseGrantLeases(t *testing.T) {


Note, there are couple of tests for leases that do not really tests authorization. They are normal lease testing scenarios with only difference that authorization is enabled.

We should investigate if we can expand standard lease tests and treat authorization as part of cluster setup. This way we will not need to duplicate test scenario (one for lease without auth, second with auth enabled).

Agreed. I think we can move this post the migration to think about it in the long term.

serathius · 2022-11-23T09:17:04Z

tests/common/auth_test.go

+					t.Fatalf("want leaseID %v expired but not", leaseID)
+				}
+				gresp, err := rootAuthClient.Get(ctx, "key", config.GetOptions{})
+				if err != nil || len(gresp.Kvs) != 0 {


Note, no need to fix it in this PR. This checking for empty response will really benefit from rewrite to testify.

ack. Will fix it.

serathius · 2022-11-23T09:22:24Z

tests/common/auth_test.go

+		authEnable(ctx, cc, t)
+		rootAuthClient := testutils.MustClient(clus.Client(WithAuth("root", "root")))
+		authSetupDefaultTestUser(ctx, rootAuthClient, t)
+		if _, err := rootAuthClient.RoleGet(ctx, "test-role"); err != nil {


Original test validated here contents of the role, like what keys it can write and what keys it can read.

expected := []string{ "Role test-role", "KV Read:", "foo", "KV Write:", "foo", }

serathius · 2022-11-23T09:23:51Z

tests/common/auth_test.go

+		if err != nil {
+			t.Fatal(err)
+		}
+		assert.ElementsMatch(t, resp.Roles, []string{"test-role"})


Original test also validated user name in response.

serathius · 2022-11-23T09:24:06Z

tests/common/auth_test.go

+		if err != nil {
+			t.Fatal(err)
+		}
+		assert.ElementsMatch(t, resp.Roles, []string{"test-role"})


serathius

Reviewed and left couple fo comments. I would highly encourage migrating one tests at a time to make review quick. This one review iteration took me over 30 minutes and we will need more iterations.

If you send a separate PR that just migrates one test, I expect that review would take 2 minutes and I good give more detailed feedback.

chaochn47 · 2022-12-14T23:20:59Z

Thanks for the review, I will split those test cases into multiple incremental PRs.

Each PR contains 2 test cases.

stale · 2023-03-18T08:59:26Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

serathius reviewed May 15, 2022

View reviewed changes

tests/framework/e2e/etcdctl.go Outdated Show resolved Hide resolved

serathius reviewed May 15, 2022

View reviewed changes

tests/framework/e2e/etcdctl.go Outdated Show resolved Hide resolved

serathius reviewed May 15, 2022

View reviewed changes

tests/framework/integration.go Outdated Show resolved Hide resolved

serathius reviewed May 15, 2022

View reviewed changes

tests/framework/interface.go Outdated Show resolved Hide resolved

chaochn47 force-pushed the auth_test branch from 6ef93e8 to 9db3134 Compare August 4, 2022 06:31

chaochn47 marked this pull request as ready for review August 4, 2022 06:31

chaochn47 changed the title ~~[WIP] migrate auth_test to common: initial commit~~ migrate auth_test to common Aug 4, 2022

chaochn47 force-pushed the auth_test branch 2 times, most recently from a174737 to 394540f Compare August 4, 2022 07:23

chaochn47 requested a review from serathius August 4, 2022 16:13

chaochn47 force-pushed the auth_test branch from 394540f to 8ec0c1c Compare August 4, 2022 16:21

ahrtr reviewed Aug 8, 2022

View reviewed changes

server/auth/store.go Outdated Show resolved Hide resolved

ahrtr reviewed Aug 8, 2022

View reviewed changes

tests/common/alarm_test.go Outdated Show resolved Hide resolved

This was referenced Aug 10, 2022

logging RoleGrantPermission key and range end #14330

Merged

common tests framework: cluster client creation could fail with invalid auth #14331

Merged

chaochn47 force-pushed the auth_test branch from 8ec0c1c to 1f4d6fc Compare October 26, 2022 05:00

chaochn47 force-pushed the auth_test branch from 1f4d6fc to b342f3d Compare October 26, 2022 05:27

chaochn47 mentioned this pull request Oct 26, 2022

common test: update auth test interface #14626

Merged

chaochn47 force-pushed the auth_test branch 2 times, most recently from 2810a71 to 4e312bc Compare October 27, 2022 03:34

chaochn47 force-pushed the auth_test branch from 4e312bc to 993ae27 Compare November 22, 2022 02:35

chaochn47 force-pushed the auth_test branch from 993ae27 to dc7ede3 Compare November 22, 2022 02:54

tests/common: migrate auth test

d5f2696

Signed-off-by: Chao Chen <chaochn@amazon.com>

chaochn47 force-pushed the auth_test branch from dc7ede3 to d5f2696 Compare November 22, 2022 18:45

ahrtr reviewed Nov 23, 2022

View reviewed changes

tests/common/auth_test.go Show resolved Hide resolved

serathius reviewed Nov 23, 2022

View reviewed changes

tests/common/auth_test.go

if err != nil {

t.Fatal(err)

}

assert.ElementsMatch(t, resp.Roles, []string{"test-role"})

Copy link

Member

serathius Nov 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

serathius reviewed Nov 23, 2022

View reviewed changes

chaochn47 mentioned this pull request Dec 14, 2022

tests/common: migrate auth tests #1 #14998

Merged

chaochn47 mentioned this pull request Dec 19, 2022

tests/common: migrate auth tests #2 #15028

Merged

chaochn47 mentioned this pull request Feb 14, 2023

migrate 6 auth test cases to common #3 #15320

Merged

This was referenced Feb 23, 2023

migrate auth test to common #4 #15351

Merged

migration auth tests to common #5 #15378

Merged

migrate auth tests to common #6 #15379

Merged

stale bot added the stale label Mar 18, 2023

chaochn47 mentioned this pull request Mar 28, 2023

migrate auth tests to common #7 #15578

Merged

chaochn47 mentioned this pull request May 10, 2023

migrate e2e auth tests to common #8 #15867

Merged

ahrtr closed this in #15867 May 13, 2023

chaochn47 deleted the auth_test branch May 13, 2023 06:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate auth_test to common #14041

migrate auth_test to common #14041

chaochn47 commented May 13, 2022 •

edited

Loading

chaochn47 commented May 13, 2022

codecov-commenter commented May 13, 2022 •

edited

Loading

ahrtr commented May 14, 2022

serathius May 15, 2022

chaochn47 May 16, 2022

serathius May 16, 2022 •

edited

Loading

chaochn47 May 16, 2022

chaochn47 commented Aug 4, 2022 •

edited

Loading

ahrtr commented Aug 8, 2022 •

edited

Loading

ahrtr commented Sep 5, 2022

chaochn47 commented Sep 6, 2022

chaochn47 commented Oct 26, 2022

chaochn47 commented Nov 22, 2022

fuweid commented Nov 22, 2022

ahrtr commented Nov 23, 2022

ahrtr commented Nov 23, 2022

serathius Nov 23, 2022

chaochn47 Dec 12, 2022

serathius Nov 23, 2022

chaochn47 Dec 12, 2022

serathius Nov 23, 2022

chaochn47 Dec 12, 2022

serathius Nov 23, 2022 •

edited

Loading

chaochn47 Dec 12, 2022

serathius Nov 23, 2022

chaochn47 Dec 12, 2022

serathius Nov 23, 2022

serathius Nov 23, 2022

serathius Nov 23, 2022

serathius left a comment

chaochn47 commented Dec 14, 2022

stale bot commented Mar 18, 2023

migrate auth_test to common #14041

migrate auth_test to common #14041

Conversation

chaochn47 commented May 13, 2022 • edited Loading

chaochn47 commented May 13, 2022

codecov-commenter commented May 13, 2022 • edited Loading

Codecov Report

ahrtr commented May 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serathius May 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chaochn47 commented Aug 4, 2022 • edited Loading

ahrtr commented Aug 8, 2022 • edited Loading

ahrtr commented Sep 5, 2022

chaochn47 commented Sep 6, 2022

chaochn47 commented Oct 26, 2022

chaochn47 commented Nov 22, 2022

fuweid commented Nov 22, 2022

ahrtr commented Nov 23, 2022

ahrtr commented Nov 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serathius Nov 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serathius left a comment

Choose a reason for hiding this comment

chaochn47 commented Dec 14, 2022

stale bot commented Mar 18, 2023

chaochn47 commented May 13, 2022 •

edited

Loading

codecov-commenter commented May 13, 2022 •

edited

Loading

serathius May 16, 2022 •

edited

Loading

chaochn47 commented Aug 4, 2022 •

edited

Loading

ahrtr commented Aug 8, 2022 •

edited

Loading

serathius Nov 23, 2022 •

edited

Loading