[ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error

[rajatrj16]

I am occasionally getting the below error and due to this not able to push the image unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

Harbor version: Version v2.11.0-70255684
harbor-helm: v1.15.0

Harbor core is constantly giving below errors. There is no error in the istio-proxy

Appending internal tls trust CA to ca-bundle ...
find: '/etc/harbor/ssl': No such file or directory
Internal tls trust CA appending is Done.
2024/08/05 06:24:43.594 �[1;44m[D]�[0m  init global config instance failed. If you do not use this, just ignore it.  open conf/app.conf: no such file or directory
2024-08-05T06:24:43Z [ERROR] [/lib/cache/cache.go:124]: failed to ping redis://harbor-redis.management.eks.region.aws.client.cloud:6379/0?idle_timeout_seconds=30, retry after 500ms : dial tcp 10.39.xx.xx:6379: connect: connection refused
2024-08-05T06:24:44Z [ERROR] [/pkg/audit/forward.go:44]: failed to create audit log, error dial tcp: missing address
2024/08/05 06:24:44.494 �[1;34m[I]�[0m [server.go:281]  http server Running on http://:8080
Appending internal tls trust CA to ca-bundle ...
find: '/etc/harbor/ssl': No such file or directory
Internal tls trust CA appending is Done.
2024/08/05 06:25:01.515 �[1;44m[D]�[0m  init global config instance failed. If you do not use this, just ignore it.  open conf/app.conf: no such file or directory
2024-08-05T06:25:01Z [ERROR] [/lib/cache/cache.go:124]: failed to ping  #redis://harbor-redis.management.eks.region.aws.client.cloud:6379/0?idle_timeout_seconds=30, retry after 500ms : dial tcp 10.39.xx.xx:6379: connect: connection refused
2024-08-05T06:25:02Z [ERROR] [/pkg/audit/forward.go:44]: failed to create audit log, error dial tcp: missing address
2024/08/05 06:25:02.341 �[1;34m[I]�[0m [server.go:281]  http server Running on http://:8080
2024-08-05T06:25:21Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T06:25:23Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T06:25:23Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T06:25:23Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T06:25:30Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:08:42Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:08:43Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:43Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:43Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:43Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:44Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:44Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:54Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:41Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:41Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024/08/05 07:14:41 http: proxy error: context canceled
2024-08-05T07:14:47Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:47Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:47Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:47Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:58Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:58Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024/08/05 07:14:58 http: proxy error: context canceled
2024/08/05 07:14:58 http: proxy error: context canceled
2024/08/05 07:14:58 http: proxy error: context canceled
2024-08-05T07:15:08Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:15:08Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:23:25Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:23:25Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error

Jobservice logs:

2024-08-05T06:25:43Z [ERROR] [/pkg/config/rest/rest.go:50]: Failed on load rest config err:http error: code 401, message {"errors":[{"code":"UNAUTHORIZED","message":"only internal service is allowed to call this API"}]}, url:http://harbor-registry-core:80/api/v2.0/internalconfig
2024-08-05T08:00:00Z [ERROR] [/pkg/config/manager.go:81]: AllConfigs failed, error failed to load rest config
2024-08-05T08:00:10Z [ERROR] [/jobservice/runtime/bootstrap.go:168]: trigger hook event error: Post "https://harbor-registry-core:443/service/notifications/tasks/2": read tcp 100.xx.xx.37:48090->172.20.xx.xx:443: read: connection reset by peer
2024-08-05T06:25:43Z [ERROR] [/jobservice/job/impl/context.go:76]: Job context initialization error: failed to load rest config
2024-08-05T08:02:26Z [ERROR] [/jobservice/hook/hook_agent.go:155]: Retry: sending hook event error: Post "https://harbor-registry-core:443/service/notifications/tasks/2": read tcp 100.64.xx.xx:39476->172.xx.xx.255:443: read: connection reset by peer, evt=status change: job=a7f29fa10abe9e99b13bd6f3@1722844800, status=Running, revision=1722844704, check_in=checkin->https://harbor-registry-core:443/service/notifications/tasks/2, duration=1m18.436330113s

[zyyw]

Is this a fresh install or upgrade from an older harbor? Also what is your authentication method, basic auth / oidc / ldap?

Found this issue might be related. Maybe you want to take a look at it:

• docker login failed: login attempt to https://xxx/v2/ failed with status: 401 Unauthorized on v2.10.2 #20629

[rajatrj16]

Is this a fresh install or upgrade from an older harbor? Also what is your authentication method, basic auth / oidc / ldap?

Found this issue might be related. Maybe you want to take a look at it:

• docker login failed: login attempt to https://xxx/v2/ failed with status: 401 Unauthorized on v2.10.2 #20629

This was an upgrade from 2.10 to 2.11.0.
The auth method is OIDC.

I have not configured any credentials except existingSecretAdminPassword and existingSecretAdminPasswordKey in values.yaml but I get unauthorized errors occasionally for robot accounts.

[NikolaiBessonov]

We have the same issue. It's floating problem.
The problem reproduced in May of 2024, 09 of September and today again.
Harbor version - 2.11.0 with s3 storage.

UPD: I have analyzed logs and metrics and got new information. When, core component runs at few replicas(e.g. 3), and controller deletes one of that replicas, new pod starts faster, then old goes delete. After new replica says in logs "Server running at 0.0.0.0:8080" errors appear.