docker login failed: login attempt to https://xxx/v2/ failed with status: 401 Unauthorized on v2.10.2

[sicko583]

Hi team,

I upgrade Harbor from v 2.8.4 to v2.10.2 today, and I got some errors when testing:

1. docker login failed with 401 Unauthorized
2. OIDC user login failed with 401 Unauthorized
3. robot account login failed with 401 Unauthorized
4. fail to pull images with 401 Unauthorized

Some logs shown:

docker login errors:

docker login https://harbor.xxx
Username: robot$harborupgradetest
Password:
Error response from daemon: login attempt to https://harbor.xxx/v2/ failed with status: 401 Unauthorized

harbor-core logs:

2024-06-19T15:15:03Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="03e33ec8faa7014fed6be743a91ce4a0" traceID="b83be82217786d316bb066b1966d9e17"]: failed to verify secret, username: Jinshuai_Ni, error: failed to refresh token, username: Jinshuai_Ni, error: oauth2: "invalid_grant" "Offline user session not found"
2024-06-19T15:15:03Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.9.62.130" requestID="03e33ec8faa7014fed6be743a91ce4a0" traceID="b83be82217786d316bb066b1966d9e17" user agent="containerd/1.7.11"]: failed to authenticate user:Jinshuai_Ni, error:not supported
2024-06-19T15:15:04Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="e86dfdd32ed638e89d3250033bad21fd" traceID="b83be82217786d316bb066b1966d9e17"]: failed to verify secret, username: Jinshuai_Ni, error: failed to refresh token, username: Jinshuai_Ni, error: oauth2: "invalid_grant" "Offline user session not found"
2024-06-19T15:15:04Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.9.62.130" requestID="e86dfdd32ed638e89d3250033bad21fd" traceID="b83be82217786d316bb066b1966d9e17" user agent="containerd/1.7.11"]: failed to authenticate user:Jinshuai_Ni, error:not supported
2024-06-19T15:15:17Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="b47acdf59c6eb432f6cab4fa08237a3f"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found
2024-06-19T15:15:17Z [ERROR] [/pkg/reg/adapter/native/adapter.go:126]: failed to ping registry https://xxx: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
2024-06-19T15:15:58Z [INFO] [/server/middleware/security/robot.go:71][requestID="9a3e1c823caf91df5520dcdb28ba4d21" traceID="66af0e823e57929c948cfbb46e4bb10e"]: a robot security context generated for request HEAD /v2/iescapital-cloud/capital_server_2024/manifests/dev.2408_599
2024-06-19T15:15:58Z [INFO] [/server/middleware/security/robot.go:71][requestID="f84402ec8942dc719818a1599b9cfa80" traceID="66af0e823e57929c948cfbb46e4bb10e"]: a robot security context generated for request HEAD /v2/iescapital-cloud/capital_server_2024/manifests/dev.2408_599
2024-06-19T15:16:35Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="62ff0e8632a44b44ffc13fa79b21c165"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found

2024-06-20T03:07:55Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="334fcd11abfd96644d485b9ab971f268"]: failed to verify secret, username: robot, error: failed to get oidc user info, error: <QuerySeter> no row found
2024-06-20T03:07:55Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.27.63.71" requestID="334fcd11abfd96644d485b9ab971f268" user agent="docker/1.13.1 go/go1.10.3 kernel/5.10.16.3-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \(linux\))"]: failed to authenticate user:robot, error:not supported

pod logs:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  79s                default-scheduler  Successfully assigned https://xxx/cis-worker-capx2408-5555c985bf-s28jb to ip-xxx.ap-northeast-1.compute.internal
  Normal   Pulling    35s (x3 over 79s)  kubelet            Pulling image "https://xxx/xxx/supporttools:capitalboto3"
  Warning  Failed     34s (x3 over 78s)  kubelet            Failed to pull image "https://xxx/xxx/supporttools:capitalboto3": failed to pull and unpack image "https://xxx/xxx/supporttools:capitalboto3": failed to resolve reference "https://xxx/xxx/supporttools:capitalboto3": unexpected status from HEAD request to https://xxx/v2/xxx/supporttools/manifests/capitalboto3: 401 Unauthorized
  Warning  Failed     34s (x3 over 78s)  kubelet            Error: ErrImagePull
  Normal   BackOff    4s (x4 over 78s)   kubelet            Back-off pulling image "https://xxx/xxx/supporttools:capitalboto3"
  Warning  Failed     4s (x4 over 78s)   kubelet            Error: ImagePullBackOff

In my situation I can login Harbor UI with OIDC user, and I can see/search images in projects as expected. Robot account can be created/removed/edited, but seems all operations with API calls would fail. I have searched quite a lot in issues but can not find the root cause. Can anyone Senior can help me on this? Thanks quite a lot for your help and time here.

[sicko583]

I just saw one similar issue #15253 which was fix in v.28.0, is it related?

[sicko583]

another error logs found when I tried to re-deploy harbor-helm with the same version:

2024-06-20T07:24:11Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:11Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="6bc792915459cba2cd8fee690c2dca39" traceID="438f2ecd2454ca03a45ae9e79b73871c"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:12Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:12Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="5e7147da3b401d907a82be7e654adffd" traceID="efac1860eea7b97998e55fbc45946f26"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:12Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:12Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="804fd458a0bb6d308526f7bffb21f27a" traceID="f07fdda0a4520138d704016416195806"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:13Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:13Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="b69126e2fbb8c21d398572afdcb80ee7" traceID="3c4a6c04af270d0370fda5598b60077c"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:13Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:13Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="e7d978e868b433cdcdef6ce8233b9d88" traceID="36949227711ca55fcae0d3cae6d4d267"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:13Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:13Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="1675bb80358656d63ce0225309f49147" traceID="688832c9b852594f388280d3c5f155d9"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:14Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:14Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="4bf97483eaee610d81cecdbc7376ea4e" traceID="5c866b21a984d4832f261376f17d6a58"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:16Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:16Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="3a19ce11c1ebdbe8ea4d55610270785f" traceID="18749ff015ec8f82e9874b7a3c9bd9d1"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:17Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:17Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="760fa9065279cc901b16a623b79f7b92" traceID="3acf97a45e8725565d550025d32e449d"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:18Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:18Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="42b249555f5c1cba7e50c1ae371ede57" traceID="68a9d1d09b6f9a59410146027433bb0a"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:24:18Z [ERROR] [/pkg/token/token.go:82]: parse token error, crypto/rsa: verification error
2024-06-20T07:24:18Z [WARNING] [/server/middleware/security/v2_token.go:67][requestID="ec1cdc29e47df0f0088c550a65e48362" traceID="3acf97a45e8725565d550025d32e449d"]: failed to decode bearer token: crypto/rsa: verification error
2024-06-20T07:25:45Z [INFO] [/lib/encrypt/encrypt.go:60]: the path of key used by key provider: /etc/core/key

And no errors when docker login with robot account:

2024-06-20T07:27:23Z [INFO] [/server/middleware/security/robot.go:71][requestID="9930899d4c802ed0ad94be0f6a9a9308"]: a robot security context generated for request GET /service/token
2024-06-20T07:28:02Z [INFO] [/server/middleware/security/robot.go:71][requestID="58ef3c80168928bb8a6f35787f4e1da2"]: a robot security context generated for request GET /service/token

[sicko583]

I removed the configured credentials in value file and leave it blank, and it fixed. I will close this issue.