[release-3.4] server,test: refresh cache on each NewAuthStore

[vivekpatani]

• permissions were incorrectly loaded on restarts.
• server/auth: users/authentication data not loaded on member startup #14355
• Backport of server/auth: refresh cache on each NewAuthStore #14358

Signed-off-by: vivekpatani 9080894+vivekpatani@users.noreply.github.com

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

[mitake]

TestV3AuthRestartMember is failing only in linux-amd64-integration-1-cpu. I guess it's a non deterministic issue because the test case finished successfully in other checks of integration test. Let me check.

[mitake]

LGTM, defer to @ahrtr . Do you think it's better to revisit timeout config?

[ahrtr]

It seems the new integration test case is failing. It might be related to the Restart method, please try to figure out the root cause.

=== RUN   TestV3AuthRestartMember
{"level":"warn","ts":"2022-09-04T19:32:36.671Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-a7ac07e9-893e-4e9a-a568-0cb48cf737ce/localhost:57870851442980026570","attempt":0,"error":"rpc error: code = Unavailable desc = transport is closing"}
    testutil.go:55: goroutine 34432 [running]:
        go.etcd.io/etcd/pkg/testutil.FatalStack(0xc002c6d680, 0xc0029c54a0, 0x4d)
        	/home/runner/work/etcd/etcd/pkg/testutil/testutil.go:54 +0x6f
        go.etcd.io/etcd/pkg/testutil.AssertEqual(0xc002c6d680, 0x0, 0x0, 0xf31560, 0xc00018d040, 0x0, 0x0, 0x0)
        	/home/runner/work/etcd/etcd/pkg/testutil/assert.go:35 +0x153
        go.etcd.io/etcd/pkg/testutil.AssertNil(...)
        	/home/runner/work/etcd/etcd/pkg/testutil/assert.go:39
        go.etcd.io/etcd/integration.TestV3AuthRestartMember(0xc002c6d680)
        	/home/runner/work/etcd/etcd/integration/v3_auth_test.go:471 +0x9c5
        testing.tRunner(0xc002c6d680, 0x109e968)
        	/opt/hostedtoolcache/go/1.16.15/x64/src/testing/testing.go:1203 +0xe5
        created by testing.(*T).Run
        	/opt/hostedtoolcache/go/1.16.15/x64/src/testing/testing.go:1248 +0x2b3

[mitake]

@ahrtr It's only failing in linux-amd64-integration-1-cpu. linux-amd64-integration-2-cpu and linux-amd64-integration-4-cpu don't cause the failure and on my local env the test case runs successfully. Probably we should fix the timeout config? How do you think @vivekpatani ?

[vivekpatani]

@mitake quoting myself from the 3.5 PR, I can't seem to reproduce this, can you by any chance reproduce this?

I've tried on docker (amd64) and on my baremetal machine (M1 arch), and have consistently failed and reproducing this.

[serathius]

Yep, v3.4 tests are definitely flaky.

[ahrtr]

Please try to add code something below, probably we should add a timeout also,

clus.Members[0].WaitOK(t)

[ahrtr]

Let me do this in a separate PR.

[ahrtr]

FYI. #14442

[ahrtr]

LGTM

Thank you @vivekpatani

[vivekpatani]

sorry for the delay, thanks for merging @ahrtr