Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auth failed with error "auth: revision in header is old" #13976

Closed
ahrtr opened this issue Apr 23, 2022 · 19 comments
Closed

Auth failed with error "auth: revision in header is old" #13976

ahrtr opened this issue Apr 23, 2022 · 19 comments
Labels
help wanted

Comments

@ahrtr
Copy link
Member

ahrtr commented Apr 23, 2022 •
edited
Loading

etcd version

3.4.16

Background

The original issue is 13503. There is a long back-and-forth conversation, but not much progress, so I raise a new issue to get the latest error message included.

There are 3 members in the etcd cluster, and the original version was 3.3.11. In the beginning, the users saw the data inconsistency issue, which could be caused by 11651. The issue was fixed in 3.3.21, 3.4.8 and 3.5.0. So the user upgraded to 3.4.16, and then see the error "auth: revision in header is old" (see detailed log below).

Please refer to the following two files to get the detailed info on how the user configures & deploys the etcd cluster,
configmap.txt
statefulset.txt

Help wanted

If you run into the same issue, please provide the etcd version & detailed steps to reproduce this issue. Thanks.

It also would be helpful if anyone can reproduce this issue in 3.5 and main branch.

Error message

**2022-03-24 02:15:14.037286 I | rafthttp: added peer af60d29c10528d99
raft2022/03/24 02:15:14 INFO: e83baa9da4c00535 switched to configuration voters=(4962605495301377754 12637332122132188569 16734156434475844917)
2022-03-24 02:15:14.037486 I | etcdserver/membership: added member e83baa9da4c00535 [https://eric-data-distributed-coordinator-ed-2.eric-data-distributed-coordinator-ed-peer.eiffelesc.svc.cluster.local:2380] to cluster 1d35520433a8d4ea
2022-03-24 02:15:14.038398 W | etcdserver: failed to apply request "header:<ID:1070308296661102795 username:\"root\" auth_revision:11 > lease_revoke:<id:0d997fb99eb3f316>" with response "size:28" took (112.397µs) to execute, err is lease not found
2022-03-24 02:15:14.038441 W | etcdserver: failed to apply request "header:<ID:1070308296661102796 username:\"root\" auth_revision:11 > lease_revoke:<id:0d997fb99eb3f314>" with response "size:28" took (18.443µs) to execute, err is lease not found
2022-03-24 02:15:14.038796 W | etcdserver: failed to apply request "header:<ID:10203326865992843127 username:\"root\" auth_revision:11 > auth_role_revoke_permission:<role:\"kms_role\" key:\"/kms/\" range_end:\"\\000\" > " with response "" took (7.084µs) to execute, err is auth: permission is not granted to the role
2022-03-24 02:15:14.038829 W | etcdserver: failed to apply request "header:<ID:10203326865992843128 username:\"root\" auth_revision:12 > auth_role_revoke_permission:<role:\"kms_role\" key:\"/shelter/\" range_end:\"\\000\" > " with response "" took (3.462µs) to execute, err is auth: permission is not granted to the role
2022-03-24 02:15:14.038923 N | auth: role kms_role's permission of key /kms/ is updated as READWRITE
2022-03-24 02:15:14.039001 N | auth: role kms_role's permission of key /shelter/ is updated as READWRITE
2022-03-24 02:15:14.039060 W | auth: user eric-sec-key-management is already granted role kms_role**
2022-03-24 02:15:14.041660 I | embed: ClientTLS: cert = /run/sec/certs/server/srvcert.pem, key = /run/sec/certs/server/srvprivkey.pem, trusted-ca = /data/combinedca/cacertbundle.pem, client-cert-auth = true, crl-file = 
2022-03-24 02:15:14.041741 I | embed: listening for peers on [::]:2380
2022-03-24 02:15:14.089214 I | rafthttp: peer 44deb767abd70eda became active
2022-03-24 02:15:14.089282 I | rafthttp: established a TCP streaming connection with peer 44deb767abd70eda (stream Message reader)
raft2022/03/24 02:15:14 INFO: raft.node: e83baa9da4c00535 elected leader 44deb767abd70eda at term 4
2022-03-24 02:15:14.167093 I | rafthttp: established a TCP streaming connection with peer 44deb767abd70eda (stream MsgApp v2 reader)
2022-03-24 02:15:14.167519 I | rafthttp: peer af60d29c10528d99 became active
2022-03-24 02:15:14.167548 I | rafthttp: established a TCP streaming connection with peer af60d29c10528d99 (stream MsgApp v2 reader)
2022-03-24 02:15:14.169870 I | rafthttp: established a TCP streaming connection with peer af60d29c10528d99 (stream Message reader)
2022-03-24 02:15:14.175822 I | etcdserver: e83baa9da4c00535 initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
2022-03-24 02:15:14.216796 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/counters/requests/2022/03, err is auth: revision in header is old
2022-03-24 02:15:14.216948 W | etcdserver: failed to apply request "header:<ID:1070308296661102798 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/counters/requests/2022/03\" value_size:47 >" with response "" took (40.243µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.216959 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/sys/counters/requests/2022/03, err is auth: revision in header is old
2022-03-24 02:15:14.216975 W | etcdserver: failed to apply request "header:<ID:1070308296661102799 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/sys/counters/requests/2022/03\" value_size:47 >" with response "" took (5.927µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217077 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/lock/d997fb99eb3ff8a, err is auth: revision in header is old
2022-03-24 02:15:14.217138 W | etcdserver: failed to apply request "header:<ID:10203326865992843149 username:\"eric-sec-key-management\" auth_revision:15 > txn:<compare:<target:CREATE key:\"/shelter/core/lock/d997fb99eb3ff8a\" create_revision:0 > success:<request_put:<key:\"/shelter/core/lock/d997fb99eb3ff8a\" lease:979954829138067338 >> failure:<>>" with response "" took (21.485µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217211 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/lock/d997fb99eb3ff91, err is auth: revision in header is old
2022-03-24 02:15:14.217239 W | etcdserver: failed to apply request "header:<ID:10203326865992843156 username:\"eric-sec-key-management\" auth_revision:15 > txn:<compare:<target:CREATE key:\"/kms/core/lock/d997fb99eb3ff91\" create_revision:0 > success:<request_put:<key:\"/kms/core/lock/d997fb99eb3ff91\" lease:979954829138067345 >> failure:<>>" with response "" took (12.327µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217249 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/leader/dd6333b5-361d-feca-e0e1-705f79d02baf, err is auth: revision in header is old
2022-03-24 02:15:14.217274 W | etcdserver: failed to apply request "header:<ID:1070308296661102800 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/core/leader/dd6333b5-361d-feca-e0e1-705f79d02baf\" > " with response "" took (7.386µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217280 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/leader/358aa84d-03ff-32fa-35b1-e597af763118, err is auth: revision in header is old
2022-03-24 02:15:14.217293 W | etcdserver: failed to apply request "header:<ID:1070308296661102801 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/kms/core/leader/358aa84d-03ff-32fa-35b1-e597af763118\" > " with response "" took (4.155µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217299 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/lock/eda7fb99d081ad9, err is auth: revision in header is old
2022-03-24 02:15:14.217312 W | etcdserver: failed to apply request "header:<ID:1070308296661102802 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/core/lock/eda7fb99d081ad9\" > " with response "" took (4.091µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217317 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/lock/eda7fb99d081adb, err is auth: revision in header is old
2022-03-24 02:15:14.217330 W | etcdserver: failed to apply request "header:<ID:1070308296661102803 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/kms/core/lock/eda7fb99d081adb\" > " with response "" took (4.027µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217339 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/lock/d997fb99eb3ff8a, err is auth: revision in header is old
2022-03-24 02:15:14.217356 W | etcdserver: failed to apply request "header:<ID:10203326865992843159 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/core/lock/d997fb99eb3ff8a\" value_size:36 lease:979954829138067338 >" with response "" took (5.035µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217362 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/lock/d997fb99eb3ff91, err is auth: revision in header is old
2022-03-24 02:15:14.217375 W | etcdserver: failed to apply request "header:<ID:10203326865992843161 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/core/lock/d997fb99eb3ff91\" value_size:36 lease:979954829138067345 >" with response "" took (4.078µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217385 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/leader/b5f8487c-ce11-a394-7122-4ddbccc3399c, err is auth: revision in header is old
2022-03-24 02:15:14.217398 W | etcdserver: failed to apply request "header:<ID:10203326865992843178 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/core/leader/b5f8487c-ce11-a394-7122-4ddbccc3399c\" value_size:1608 >" with response "" took (4.385µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217403 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/leader/da7d6abf-5261-d22f-4213-b11115faa115, err is auth: revision in header is old
2022-03-24 02:15:14.217417 W | etcdserver: failed to apply request "header:<ID:10203326865992843179 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/core/leader/da7d6abf-5261-d22f-4213-b11115faa115\" value_size:1609 >" with response "" took (4.821µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217423 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/cluster/feature-flags, err is auth: revision in header is old
2022-03-24 02:15:14.217439 W | etcdserver: failed to apply request "header:<ID:10203326865992843181 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/core/cluster/feature-flags\" value_size:69 >" with response "" took (4.149µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217447 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/cluster/feature-flags, err is auth: revision in header is old
2022-03-24 02:15:14.217459 W | etcdserver: failed to apply request "header:<ID:10203326865992843183 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/core/cluster/feature-flags\" value_size:69 >" with response "" took (4.032µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217466 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/logical/94a702ce-a10f-2931-94fd-e62d565eee60/casesensitivity, err is auth: revision in header is old
2022-03-24 02:15:14.217482 W | etcdserver: failed to apply request "header:<ID:10203326865992843226 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/logical/94a702ce-a10f-2931-94fd-e62d565eee60/casesensitivity\" value_size:69 >" with response "" took (4.1µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217488 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/logical/e4977d9c-9daf-6397-fef1-8c985643721f/casesensitivity, err is auth: revision in header is old
2022-03-24 02:15:14.217500 W | etcdserver: failed to apply request "header:<ID:10203326865992843244 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/logical/e4977d9c-9daf-6397-fef1-8c985643721f/casesensitivity\" value_size:69 >" with response "" took (4.273µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217509 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/logical/94a702ce-a10f-2931-94fd-e62d565eee60/packer/buckets/4, err is auth: revision in header is old
2022-03-24 02:15:14.217522 W | etcdserver: failed to apply request "header:<ID:10203326865992843251 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/logical/94a702ce-a10f-2931-94fd-e62d565eee60/packer/buckets/4\" value_size:480 >" with response "" took (4.214µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217530 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/accessor/0ad5472062556c01aece0f5a735b0e243c83504e, err is auth: revision in header is old
2022-03-24 02:15:14.217543 W | etcdserver: failed to apply request "header:<ID:10203326865992843257 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/token/accessor/0ad5472062556c01aece0f5a735b0e243c83504e\" value_size:138 >" with response "" took (4.351µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217550 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/id/hb6c0289ac4c999cf88e1c4d9cea317e5892d17fec06a5dbe599aefe7dc280d6e, err is auth: revision in header is old
2022-03-24 02:15:14.217568 W | etcdserver: failed to apply request "header:<ID:10203326865992843260 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/token/id/hb6c0289ac4c999cf88e1c4d9cea317e5892d17fec06a5dbe599aefe7dc280d6e\" value_size:828 >" with response "" took (6.893µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217574 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/expire/id/auth/kubernetes/login/hb6c0289ac4c999cf88e1c4d9cea317e5892d17fec06a5dbe599aefe7dc280d6e, err is auth: revision in header is old
2022-03-24 02:15:14.217588 W | etcdserver: failed to apply request "header:<ID:10203326865992843261 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/expire/id/auth/kubernetes/login/hb6c0289ac4c999cf88e1c4d9cea317e5892d17fec06a5dbe599aefe7dc280d6e\" value_size:1644 >" with response "" took (4.058µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217598 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/id/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6, err is auth: revision in header is old
2022-03-24 02:15:14.217624 W | etcdserver: failed to apply request "header:<ID:10203326865992843280 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/token/id/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6\" value_size:871 >" with response "" took (21.234µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217630 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/expire/id/auth/kubernetes/login/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6, err is auth: revision in header is old
2022-03-24 02:15:14.217645 W | etcdserver: failed to apply request "header:<ID:10203326865992843284 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/sys/expire/id/auth/kubernetes/login/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6\" > " with response "" took (4.364µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217651 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/accessor/a64aa78d1e78507b67e15a55fe466d7de086ed9f, err is auth: revision in header is old
2022-03-24 02:15:14.217665 W | etcdserver: failed to apply request "header:<ID:10203326865992843285 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/sys/token/accessor/a64aa78d1e78507b67e15a55fe466d7de086ed9f\" > " with response "" took (4.522µs) to execute, err is auth: revision in header is old
2022-03-24 02:15:14.217670 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/token/id/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6, err is auth: revision in header is old
2022-03-24 02:15:14.217684 W | etcdserver: failed to apply request "header:<ID:10203326865992843286 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/sys/token/id/hd8f8461e4534812c6d951a00aca52f0865944f8d8f8feed028d5640ae34a92d6\" > " with response "" took (4.133µs) to execute, err is auth: revision in header is old
@ahrtr ahrtr mentioned this issue Apr 23, 2022
Closed
@rahulbapumore
Copy link
Contributor

rahulbapumore commented Apr 26, 2022 •
edited by serathius
Loading

Hi @ahrtr ,
We also found some error logs where we saw that failed to apply request for role_revoke_permission, and after this error messages we saw all put requests were failing -

2022-01-24 14:51:04.629814 I | rafthttp: started streaming with peer af7df48dbdda0849 (stream Message reader)
2022-01-24 14:51:04.630696 W | etcdserver: failed to apply request "header:<ID:597147566431101301 username:\"root\" auth_revision:11 > auth_role_revoke_permission:<role:\"kms_role\" key:\"/kms/\" range_end:\"\\000\" > " with response "" took (9.208µs) to execute, err is auth: permission is not granted to the role
2022-01-24 14:51:04.630738 W | etcdserver: failed to apply request "header:<ID:597147566431101302 username:\"root\" auth_revision:12 > auth_role_revoke_permission:<role:\"kms_role\" key:\"/shelter/\" range_end:\"\\000\" > " with response "" took (3.796µs) to execute, err is auth: permission is not granted to the role
2022-01-24 14:51:04.861354 W | etcdserver: failed to apply request "header:<ID:17279887961093681272 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/kms/sys/counters/requests/2022/01\" value_size:47 >" with response "" took (24.924µs) to execute, err is auth: revision in header is old
2022-01-24 14:51:04.861363 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/sys/counters/requests/2022/01, err is auth: revision in header is old
2022-01-24 14:51:04.861378 W | etcdserver: failed to apply request "header:<ID:17279887961093681273 username:\"eric-sec-key-management\" auth_revision:15 > put:<key:\"/shelter/sys/counters/requests/2022/01\" value_size:47 >" with response "" took (5.403µs) to execute, err is auth: revision in header is old
2022-01-24 14:51:04.861512 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/lock/6fce7e8c3f89b083, err is auth: revision in header is old
2022-01-24 14:51:04.861556 W | etcdserver: failed to apply request "header:<ID:17279887961093681286 username:\"eric-sec-key-management\" auth_revision:15 > txn:<compare:<target:CREATE key:\"/kms/core/lock/6fce7e8c3f89b083\" create_revision:0 > success:<request_put:<key:\"/kms/core/lock/6fce7e8c3f89b083\" lease:8056515924238905475 >> failure:<>>" with response "" took (10.574µs) to execute, err is auth: revision in header is old
2022-01-24 14:51:04.861620 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/lock/8497e8c3f9d6182, err is auth: revision in header is old
2022-01-24 14:51:04.861641 W | etcdserver: failed to apply request "header:<ID:597147566431101317 username:\"eric-sec-key-management\" auth_revision:15 > txn:<compare:<target:CREATE key:\"/shelter/core/lock/8497e8c3f9d6182\" create_revision:0 > success:<request_put:<key:\"/shelter/core/lock/8497e8c3f9d6182\" lease:597147566431101314 >> failure:<>>" with response "" took (4.867µs) to execute, err is auth: revision in header is old
2022-01-24 14:51:04.861650 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /shelter/core/leader/b717d213-abc9-a616-a868-947232dca460, err is auth: revision in header is old
2022-01-24 14:51:04.861673 W | etcdserver: failed to apply request "header:<ID:17279887961093681296 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/shelter/core/leader/b717d213-abc9-a616-a868-947232dca460\" > " with response "" took (6.748µs) to execute, err is auth: revision in header is old
2022-01-24 14:51:04.861678 W | auth: request auth revision is less than current node auth revision,current node auth revision is 17,request auth revision is 15,request key is /kms/core/leader/3c8e88e3-f0a8-6171-77ee-299e54292d41, err is auth: revision in header is old
2022-01-24 14:51:04.861692 W | etcdserver: failed to apply request "header:<ID:17279887961093681297 username:\"eric-sec-key-management\" auth_revision:15 > delete_range:<key:\"/kms/core/leader/3c8e88e3-f0a8-6171-77ee-299e54292d41\" > " with response "" took (4.406µs) to execute, err is auth: revision in header is old

So we suspect that RoleRevokePermission method in store.go file of etcd code base is having some problem which is failing to apply request.
Would you please help in debugging in that?

Thanks,
Rahul

@ahrtr
Copy link
Member Author

ahrtr commented Apr 26, 2022

cc @mitake

Please format the log using code @rahulbapumore I said this multiple times!

@mitake
Copy link
Contributor

mitake commented Apr 26, 2022

@rahulbapumore I guess this issue was fixed in #13308 Can you reproduce it in the latest main branch?

@serathius
Copy link
Member

@ahrtr sorry for confusion, I was the one who formatted the code before :P

@ahrtr
Copy link
Member Author

ahrtr commented Apr 26, 2022

@rahulbapumore I guess this issue was fixed in #13308 Can you reproduce it in the latest main branch?

Thanks @mitake , probably it makes sense to backport the fix to 3.4 as well?

@ahrtr ahrtr mentioned this issue Apr 27, 2022
Closed
@mitake
Copy link
Contributor

mitake commented Apr 27, 2022

@ahrtr Yeah, probably there are some requirements. Let me open a PR for backporting.

@mitake mitake mentioned this issue Apr 29, 2022
Merged
1 task
@mitake
Copy link
Contributor

mitake commented Apr 29, 2022

@ahrtr opened #13999 for backporitng, but it's still WIP (Currently I have m1 mac and release-3.4 cannot be built on it. I'll be able to build and test on amd64 tomorrow). I'll let you know when it's ready to be reviewed.

@rahulbapumore
Copy link
Contributor

Hi @ahrtr ,
We have one question, like etcd does use any flag or method internally to identify data inconsistency and tell user that data inconsistency is there in cluster inside logs?
Because its hard to know data inconsistency issue, we had to compare keys and values from different nodes.
Could you please confirm on this?

Thanks,
Rahul

@rahulbapumore
Copy link
Contributor

Hi @mitake ,
Could you please confirm that when the etcd 3.4 would be available with the backport fix for auth_revision?

Thanks,
Rahul

@mitake
Copy link
Contributor

mitake commented May 7, 2022

@rahulbapumore the change will be included in the next stable release, v3.4.19 (#14017). I think @hexfusion is maintaining stable releases of v3.4, do you have a plan for the next release?

@ahrtr
Copy link
Member Author

ahrtr commented May 7, 2022

Hi @ahrtr , We have one question, like etcd does use any flag or method internally to identify data inconsistency and tell user that data inconsistency is there in cluster inside logs? Because its hard to know data inconsistency issue, we had to compare keys and values from different nodes. Could you please confirm on this?

Thanks, Rahul

Please read v3.5-data-inconsistency.md#detection

@rahulbapumore
Copy link
Contributor

Hi @ahrtr ,
Could you please confirm when v3.4.19 will be released and available?

Thanks,
Rahul

@rahulbapumore
Copy link
Contributor

Hi @ahrtr ,
Could you please confirm when v3.4.19 will be released and available?

Thanks,
Rahul

@rahulbapumore
Copy link
Contributor

Hi @ahrtr ,
Could you please confirm when v3.4.19 will be released and available?

Thanks,
Rahul

@didihongsheng
Copy link

didihongsheng commented May 31, 2022 •
edited
Loading

I meet data inconsistency in 3.4.9. the leader's revsion is a lot behind, and memory is much smaller than followers.
key total number is almost the same，but revision and events is far less than followers.

the revision diff show as below:
(max(etcd_debugging_mvcc_current_revision)-min(etcd_debugging_mvcc_current_revision))

image

in use db size:
image

the request inbound rate：

image

peer network:

image

mvcc metrics:

image

the log of leader show as below:

May 27 02:32:32  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:32  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:32:34  etcd[302992]: request "header:<ID:665547241150858105 > txn:<compare:<result:NOT_EQUAL target:CREATE key:\"/service/price.business.sg.live.master.default.c3c7b617b56d41e0e5a56bc242bf2946\" create_revision:0  result:NOT_EQUAL target:VALUE key:\"/service/price.business.sg.live.master.default.c3c7b617b56d41e0e5a56bc242bf2946\" value_size:36 > success:<> failure:<request_put:<key:\"/spex/spex.sg.live.master.sg.10_70_147_156/price.business.sg.live.master.default.c3c7b617b56d41e0e5a56bc242bf2946\" value_size:638 lease:665547241150858020 > request_put:<key:\"/service/price.business.sg.live.master.default.c3c7b617b56d41e0e5a56bc242bf2946\" value_size:36 lease:665547241150858020 >>>" with result "size:32" took too long (1.367591513s) to execute
May 27 02:32:35  etcd[302992]: read-only range request "key:\"/spex\" range_end:\"/spey\" sort_order:ASCEND sort_target:MOD serializable:true " with result "range_response_count:89879 size:98795656" took too long (2.260380202s) to execute
May 27 02:32:37  etcd[302992]: segmented wal file /data/etcd/member/wal/0000000000000e82-0000000015cdc29b.wal is created
May 27 02:32:38  etcd[302992]: start to snapshot (applied: 365803669, lastsnap: 365703667)
May 27 02:32:38  etcd[302992]: saved snapshot at index 365803669
May 27 02:32:38  etcd[302992]: compacted raft log at 365798669
May 27 02:32:38  etcd[302992]: WARNING: 2022/05/27 02:32:38 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:32:38  etcd[302992]: WARNING: 2022/05/27 02:32:38 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:32:40  etcd[302992]: WARNING: 2022/05/27 02:32:40 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:32:41  etcd[302992]: WARNING: 2022/05/27 02:32:41 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:32:43  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:43  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:32:44  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:44  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzQsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.n70FWD7zXe-QvVYq2sLPCHF9tSdd_p2_6ZRAFmIcOlMvwc4h3FsvOTtdM1ouUQWhg0QNJ43908p0oWZVNu5814Tw7CjtRMpD2Q77mNCEj3ADPWFGdHWW4tktUxA2OFXoJZ5G9HswZppJaL0XZ_70zxdGOBkTpkoFIXHonbmbYmajbwWhSWJj-b2k6zajW9jl-IgJdRWnmcfkhMm0utdBHPfGl-WWQTZIUjWguIqFOmBZMBs7xMKRlosiSxpjVZ4YH33y1yzB6Kb1q3TIxnSRTMg3FHvEud5SGZMbKLdBbm3qCd64MiOoKMr8ADWv2pAUV-FYdwbZ5QE2olayON85Qw
May 27 02:32:44  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:44  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:32:46  etcd[302992]: purged file /data/etcd/member/wal/0000000000000e7d-0000000015c6d548.wal successfully
May 27 02:32:54  etcd[302992]: purged file /data/etcd/member/snap/0000000000000002-0000000015c6176f.snap successfully
May 27 02:32:55  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:55  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:32:56  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:56  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzAsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.Zzm_Q6gdMvACQm4jgD_P5MFAHJhHjRlqKYAKE2Q0f2gOKvVqSVaF3VeR8WOIVE0OzSFr5v9X1FhNZFgAx0o74iCdj1BnLCUwgAScG6wwQTH6NvTIGnyUv5fbf2KNW1_oV9FdoqbWJk8CkllXvQlyH95oZol4hitPyQn0ER8fiP0K_9SR6wB44t3P59iIXC08qjjseoMd8OBowUcvhVrJBDE3n3MXwC_skFAVVlR2clR2Hi3fQDCayrXXALsnMPrZeJagm5ycl5WKIoVWHaLBZmVoCbcIQPxq6N74TGJaxd6hzrTrw5p8YBXasW-PSImR1JHYHXRF97C9fTaQI1dBzg
May 27 02:32:56  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:56  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:32:56  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:56  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzAsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.Zzm_Q6gdMvACQm4jgD_P5MFAHJhHjRlqKYAKE2Q0f2gOKvVqSVaF3VeR8WOIVE0OzSFr5v9X1FhNZFgAx0o74iCdj1BnLCUwgAScG6wwQTH6NvTIGnyUv5fbf2KNW1_oV9FdoqbWJk8CkllXvQlyH95oZol4hitPyQn0ER8fiP0K_9SR6wB44t3P59iIXC08qjjseoMd8OBowUcvhVrJBDE3n3MXwC_skFAVVlR2clR2Hi3fQDCayrXXALsnMPrZeJagm5ycl5WKIoVWHaLBZmVoCbcIQPxq6N74TGJaxd6hzrTrw5p8YBXasW-PSImR1JHYHXRF97C9fTaQI1dBzg
May 27 02:32:56  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:56  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:32:56  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzQsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.n70FWD7zXe-QvVYq2sLPCHF9tSdd_p2_6ZRAFmIcOlMvwc4h3FsvOTtdM1ouUQWhg0QNJ43908p0oWZVNu5814Tw7CjtRMpD2Q77mNCEj3ADPWFGdHWW4tktUxA2OFXoJZ5G9HswZppJaL0XZ_70zxdGOBkTpkoFIXHonbmbYmajbwWhSWJj-b2k6zajW9jl-IgJdRWnmcfkhMm0utdBHPfGl-WWQTZIUjWguIqFOmBZMBs7xMKRlosiSxpjVZ4YH33y1yzB6Kb1q3TIxnSRTMg3FHvEud5SGZMbKLdBbm3qCd64MiOoKMr8ADWv2pAUV-FYdwbZ5QE2olayON85Qw
May 27 02:32:56  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:33:26  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:26  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzAsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.Zzm_Q6gdMvACQm4jgD_P5MFAHJhHjRlqKYAKE2Q0f2gOKvVqSVaF3VeR8WOIVE0OzSFr5v9X1FhNZFgAx0o74iCdj1BnLCUwgAScG6wwQTH6NvTIGnyUv5fbf2KNW1_oV9FdoqbWJk8CkllXvQlyH95oZol4hitPyQn0ER8fiP0K_9SR6wB44t3P59iIXC08qjjseoMd8OBowUcvhVrJBDE3n3MXwC_skFAVVlR2clR2Hi3fQDCayrXXALsnMPrZeJagm5ycl5WKIoVWHaLBZmVoCbcIQPxq6N74TGJaxd6hzrTrw5p8YBXasW-PSImR1JHYHXRF97C9fTaQI1dBzg
May 27 02:33:36  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:36  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzAsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.Zzm_Q6gdMvACQm4jgD_P5MFAHJhHjRlqKYAKE2Q0f2gOKvVqSVaF3VeR8WOIVE0OzSFr5v9X1FhNZFgAx0o74iCdj1BnLCUwgAScG6wwQTH6NvTIGnyUv5fbf2KNW1_oV9FdoqbWJk8CkllXvQlyH95oZol4hitPyQn0ER8fiP0K_9SR6wB44t3P59iIXC08qjjseoMd8OBowUcvhVrJBDE3n3MXwC_skFAVVlR2clR2Hi3fQDCayrXXALsnMPrZeJagm5ycl5WKIoVWHaLBZmVoCbcIQPxq6N74TGJaxd6hzrTrw5p8YBXasW-PSImR1JHYHXRF97C9fTaQI1dBzg
May 27 02:33:36  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:36  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:33:36  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:36  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzQsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.n70FWD7zXe-QvVYq2sLPCHF9tSdd_p2_6ZRAFmIcOlMvwc4h3FsvOTtdM1ouUQWhg0QNJ43908p0oWZVNu5814Tw7CjtRMpD2Q77mNCEj3ADPWFGdHWW4tktUxA2OFXoJZ5G9HswZppJaL0XZ_70zxdGOBkTpkoFIXHonbmbYmajbwWhSWJj-b2k6zajW9jl-IgJdRWnmcfkhMm0utdBHPfGl-WWQTZIUjWguIqFOmBZMBs7xMKRlosiSxpjVZ4YH33y1yzB6Kb1q3TIxnSRTMg3FHvEud5SGZMbKLdBbm3qCd64MiOoKMr8ADWv2pAUV-FYdwbZ5QE2olayON85Qw
May 27 02:33:38  etcd[302992]: WARNING: 2022/05/27 02:33:38 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:38  etcd[302992]: WARNING: 2022/05/27 02:33:38 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:38  etcd[302992]: WARNING: 2022/05/27 02:33:38 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:41  etcd[302992]: request "header:<ID:665547241150877827 > txn:<compare:<result:NOT_EQUAL target:CREATE key:\"/service/cart.pipeline.db_sync.global.live.master.default.78982891bb51d3f26b382f00a09c7d98\" create_revision:0  result:NOT_EQUAL target:VALUE key:\"/service/cart.pipeline.db_sync.global.live.master.default.78982891bb51d3f26b382f00a09c7d98\" value_size:36 > success:<> failure:<request_put:<key:\"/spex/spex.sg.live.master.sg.10_65_236_176/cart.pipeline.db_sync.global.live.master.default.78982891bb51d3f26b382f00a09c7d98\" value_size:266 lease:665547241150877721 > request_put:<key:\"/service/cart.pipeline.db_sync.global.live.master.default.78982891bb51d3f26b382f00a09c7d98\" value_size:36 lease:665547241150877721 >>>" with result "size:32" took too long (1.332261891s) to execute
May 27 02:33:42  etcd[302992]: read-only range request "key:\"/spex\" range_end:\"/spey\" sort_order:ASCEND sort_target:MOD serializable:true " with result "range_response_count:89891 size:98803717" took too long (2.219724443s) to execute
May 27 02:33:45  etcd[302992]: request "header:<ID:665547241150880558 > txn:<compare:<> success:<request_put:<key:\"/spex/spex.sg.live.master.sg.10_65_234_186\" value_size:139 lease:665547241150880538 > request_put:<key:\"/admin/spex/spex.sg.live.master.sg.10_65_234_186\" value_size:103 lease:665547241150880538 >> failure:<>>" with result "size:34" took too long (1.311931673s) to execute
May 27 02:33:46  etcd[302992]: read-only range request "key:\"/spex\" range_end:\"/spey\" sort_order:ASCEND sort_target:MOD serializable:true " with result "range_response_count:89894 size:98805864" took too long (2.387294615s) to execute
May 27 02:33:46  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:46  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:33:48  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:48  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzAsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.Zzm_Q6gdMvACQm4jgD_P5MFAHJhHjRlqKYAKE2Q0f2gOKvVqSVaF3VeR8WOIVE0OzSFr5v9X1FhNZFgAx0o74iCdj1BnLCUwgAScG6wwQTH6NvTIGnyUv5fbf2KNW1_oV9FdoqbWJk8CkllXvQlyH95oZol4hitPyQn0ER8fiP0K_9SR6wB44t3P59iIXC08qjjseoMd8OBowUcvhVrJBDE3n3MXwC_skFAVVlR2clR2Hi3fQDCayrXXALsnMPrZeJagm5ycl5WKIoVWHaLBZmVoCbcIQPxq6N74TGJaxd6hzrTrw5p8YBXasW-PSImR1JHYHXRF97C9fTaQI1dBzg
May 27 02:33:48  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:48  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzAsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.Zzm_Q6gdMvACQm4jgD_P5MFAHJhHjRlqKYAKE2Q0f2gOKvVqSVaF3VeR8WOIVE0OzSFr5v9X1FhNZFgAx0o74iCdj1BnLCUwgAScG6wwQTH6NvTIGnyUv5fbf2KNW1_oV9FdoqbWJk8CkllXvQlyH95oZol4hitPyQn0ER8fiP0K_9SR6wB44t3P59iIXC08qjjseoMd8OBowUcvhVrJBDE3n3MXwC_skFAVVlR2clR2Hi3fQDCayrXXALsnMPrZeJagm5ycl5WKIoVWHaLBZmVoCbcIQPxq6N74TGJaxd6hzrTrw5p8YBXasW-PSImR1JHYHXRF97C9fTaQI1dBzg
May 27 02:33:48  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:48  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:33:48  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:48  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzUsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.hypTHIwK5pOesfsvZVwgQ2mp90L_HFJ3r5d5drpdWdUISl2Ncd6OQ-lNQmZnnShlZUzxaOR1lvDxR-ilwaxGQ8cT32V6x75I-IceMVQp-I1OwJCVZcHIV59jL6KRiLcKmtheMmcFgm-qYNwXpny4lrMjwilCMd_EzX8TE_BOzWT9yVKJjM4AMBmLwQkflUqAud9Vvqwjqk-BPocF81kLVXl87rxe-5hb_0892QnAc7ph-L6fTEm-xan3k5AkAPMyooMcOeWO4GUiOEoMVwQ0bCpnpCxq7aUCpSh7ZDZcERQCff_92CS9wkQfcDBT71w_Hn1j3Sgs-7UxG33Xc1BaQg
May 27 02:33:48  etcd[302992]: failed to parse jwt token: Token is expired
May 27 02:33:48  etcd[302992]: invalid auth token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc5MjI4MzQsInJldmlzaW9uIjo0LCJ1c2VybmFtZSI6InJvb3QifQ.n70FWD7zXe-QvVYq2sLPCHF9tSdd_p2_6ZRAFmIcOlMvwc4h3FsvOTtdM1ouUQWhg0QNJ43908p0oWZVNu5814Tw7CjtRMpD2Q77mNCEj3ADPWFGdHWW4tktUxA2OFXoJZ5G9HswZppJaL0XZ_70zxdGOBkTpkoFIXHonbmbYmajbwWhSWJj-b2k6zajW9jl-IgJdRWnmcfkhMm0utdBHPfGl-WWQTZIUjWguIqFOmBZMBs7xMKRlosiSxpjVZ4YH33y1yzB6Kb1q3TIxnSRTMg3FHvEud5SGZMbKLdBbm3qCd64MiOoKMr8ADWv2pAUV-FYdwbZ5QE2olayON85Qw
May 27 02:33:50  etcd[302992]: WARNING: 2022/05/27 02:33:50 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:50  etcd[302992]: WARNING: 2022/05/27 02:33:50 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:50  etcd[302992]: WARNING: 2022/05/27 02:33:50 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:50  etcd[302992]: WARNING: 2022/05/27 02:33:50 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:50  etcd[302992]: WARNING: 2022/05/27 02:33:50 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:50  etcd[302992]: WARNING: 2022/05/27 02:33:50 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:33:55  etcd[302992]: failed to apply request "header:<ID:16633622669987855994 > lease_revoke:<id:093c7fb08a09eab6>" with response "size:31" took (1.738464ms) to execute, err is lease not found
May 27 02:33:55  etcd[302992]: failed to apply request "header:<ID:16633622669987855996 > lease_revoke:<id:093c7fb08a09a062>" with response "size:31" took (37.861µs) to execute, err is lease not found
May 27 02:33:55  etcd[302992]: failed to revoke 093c7fb08a09eab6 ("lease not found")
May 27 02:33:55  etcd[302992]: failed to revoke 093c7fb08a09a062 ("lease not found")
May 27 02:33:55  etcd[302992]: failed to apply request "header:<ID:16633622669987855995 > lease_revoke:<id:093c7fb08a09a0a9>" with response "size:31" took (39.343µs) to execute, err is lease not found
May 27 02:33:55  etcd[302992]: failed to apply request "header:<ID:16633622669987855997 > lease_revoke:<id:093c7fb08a09ea80>" with response "size:31" took (38.702µs) to execute, err is lease not found
May 27 02:33:55  etcd[302992]: failed to revoke 093c7fb08a09a0a9 ("lease not found")
May 27 02:33:55  etcd[302992]: failed to revoke 093c7fb08a09ea80 ("lease not found")
May 27 02:33:55  etcd[302992]: failed to apply request "header:<ID:16633622669987855998 > lease_revoke:<id:093c7fb08a099344>" with response "size:31" took (41.485µs) to execute, err is lease not found
May 27 02:33:55  etcd[302992]: failed to revoke 093c7fb08a099344 ("lease not found")
May 27 02:34:02  etcd[302992]: WARNING: 2022/05/27 02:34:02 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:34:02  etcd[302992]: WARNING: 2022/05/27 02:34:02 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:34:02  etcd[302992]: WARNING: 2022/05/27 02:34:02 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:34:03  etcd[302992]: failed to revoke 18c67fb08cf38e36 ("etcdserver: too many requests")
May 27 02:34:10  etcd[302992]: failed to revoke 18c67fb08cf38e36 ("etcdserver: too many requests")
May 27 02:34:10  etcd[302992]: segmented wal file /data/etcd/member/wal/0000000000000e83-0000000015cea523.wal is created
May 27 02:34:16  etcd[302992]: purged file /data/etcd/member/wal/0000000000000e7e-0000000015c8537a.wal successfully
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb087645317 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 18c67fb08885dd16 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb087645331 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb08764541f ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb087645422 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 18c67fb08885b777 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb0876455ad ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb087645429 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb087645560 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb08764542b ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb087645306 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb0876455b2 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb087645590 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 093c7fb087645580 ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 18c67fb08885b72b ("etcdserver: too many requests")
May 27 02:34:23  etcd[302992]: failed to revoke 18c67fb08885b72e ("etcdserver: too many requests")
May 27 02:34:25  etcd[302992]: failed to revoke 093c7fb08763b71f ("etcdserver: too many requests")
May 27 02:34:25  etcd[302992]: failed to revoke 093c7fb08763b71c ("etcdserver: too many requests")
May 27 02:34:25  etcd[302992]: failed to revoke 093c7fb08763b74c ("etcdserver: too many requests")
May 27 02:34:25  etcd[302992]: failed to revoke 093c7fb08763b74b ("etcdserver: too many requests")
May 27 02:34:25  etcd[302992]: failed to revoke 18c67fb08885be5b ("etcdserver: too many requests")
May 27 02:34:25  etcd[302992]: failed to revoke 093c7fb08763b771 ("etcdserver: too many requests")
May 27 02:34:25  etcd[302992]: failed to revoke 18c67fb08885be5f ("etcdserver: too many requests")
May 27 02:34:26  etcd[302992]: WARNING: 2022/05/27 02:34:26 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:34:26  etcd[302992]: WARNING: 2022/05/27 02:34:26 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:34:26  etcd[302992]: WARNING: 2022/05/27 02:34:26 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
May 27 02:34:26  etcd[302992]: failed to revoke 093c7fb08763ba04 ("etcdserver: too many requests")
May 27 02:34:26  etcd[302992]: failed to revoke 18c67fb08885d23c ("etcdserver: too many requests")

@rahulbapumore
Copy link
Contributor

Hi @ahrtr ,
Could you please confirm when v3.4.19 will be released and available?

Thanks,
Rahul

@ahrtr
Copy link
Member Author

ahrtr commented Jun 7, 2022

Hi @ahrtr , Could you please confirm when v3.4.19 will be released and available?

Thanks, Rahul

We are still working on it. See issues/14074 and issues/13912. Instead of passively waiting on the release, please feel free to proactively help to resolve the test issue in 3.4.

@ahrtr
Copy link
Member Author

ahrtr commented Jun 7, 2022

Since the PR have already been backported to 3.4, so closing this issue.

@ahrtr ahrtr closed this as completed Jun 7, 2022
@rahulbapumore
Copy link
Contributor

Hi @ahrtr ,
We will try to see whatever help we can do, Thanks for reply .

Thanks,
Rahul

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mitake @serathius @didihongsheng @rahulbapumore @ahrtr