Bug: Auth revision w/ JWT tokens

[davissp14]

Etcd version: 3.5

It seems that jwt auth tokens are tied to a particular auth revision. If the auth revision is incremented after a token has been issued, connections fail with:

"level":"warn","ts":"2021-08-19T21:28:01.156Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://xxx/#initially=[xxxx:10000]","attempt":0,"error":"rpc error: code = Unknown desc = auth: revision in header is old"}

The connections will continue to fail until the TTL is hit and the token is renewed. Given our use-case this makes JWT tokens pretty much unusable.

In order to reproduce

Generate certs

 #!/bin/bash

# ES256
# Private key
openssl ecparam -genkey -name prime256v1 -noout -out ecdsa-p256-private.pem

# Public key
openssl ec -in ecdsa-p256-private.pem -pubout -out ecdsa-p256-public.pem

Configure the auth token and boot the cluster
auth_token=jwt,pub-key=ecdsa-p256-public.pem.pub,priv-key=ecdsa-p256-private.pem,sign-method=ES256

Configure auth
Setup your root user, grant role, etc.

Establish a connection

1. Establish a connection using any client.

2. In a separate terminal or through etcdctl, add a new user or make a change that will bump the auth revision.

3. The original connection established by the client will become unusable until the auth token is renewed.

[davissp14]

cc:// @mitake Thoughts?

[mitake]

@davissp14 thanks for sharing the detailed way to reproduce, let me check it on my side.

[mitake]

I'll work on in after merging #13262

[mitake]

the above PR was merged, I'll work on this PR later this week