Harbor not work after remove chartmuseum component.

[chlins]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:
A clear and concise description of what you expected to happen and what's the actual behavior. If applicable, add screenshots to help explain your problem.

Deploy harbor cluster by full_stack, and when it all work, update the cr of harborcluster to remove the chartmuseum component. Although the pod of Harbor chartmuseum has been removed, but user can not enter to project view in UI and API got 500 error.

Steps to reproduce the problem:
Please provide the steps to reproduce this problem.

1. deploy full_stack
2. update harbor cluster(remove chartmuseum)
3. check harbor project in ui and core log

Versions:
Please specify the versions of following systems.

• harbor operator version: [x.y.z]
• harbor version: [x.y.z]
• kubernetes version: [x.y.z]
• Any additional relevant versions such as CertManager

Additional context:

ui screenshot

core.log

 [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"get chart count of project 1 failed: get content failed: send request GET /library/index.yaml failed: Get \"https://harborcluster-sample-harbor-harbor-chartmuseum/library/index.yaml\": dial tcp: lookup harborcluster-sample-harbor-harbor-chartmuseum on 10.96.0.10:53: server misbehaving"}]}
/harbor/src/server/v2.0/handler/project.go:721, github.com/goharbor/harbor/src/server/v2.0/handler.(*projectAPI).populateProperties
/harbor/src/server/v2.0/handler/project.go:660, github.com/goharbor/harbor/src/server/v2.0/handler.(*projectAPI).getProject
/harbor/src/server/v2.0/handler/project.go:328, github.com/goharbor/harbor/src/server/v2.0/handler.(*projectAPI).GetProject
/harbor/src/server/v2.0/restapi/configure_harbor.go:2129, github.com/goharbor/harbor/src/server/v2.0/restapi.HandlerAPI.func92
/harbor/src/server/v2.0/restapi/operations/project/get_project.go:19, github.com/goharbor/harbor/src/server/v2.0/restapi/operations/project.GetProjectHandlerFunc.Handle
/harbor/src/server/v2.0/restapi/operations/project/get_project.go:69, github.com/goharbor/harbor/src/server/v2.0/restapi/operations/project.(*GetProject).ServeHTTP
/harbor/src/server/v2.0/restapi/operations/harbor_api.go:1746, github.com/goharbor/harbor/src/server/v2.0/restapi/operations.(*HarborAPI).handler.func1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/apiversion/api_version.go:28, github.com/goharbor/harbor/src/server/middleware/apiversion.Middleware.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/router/router.go:90, github.com/goharbor/harbor/src/server/router.(*Route).Handler.func1
/harbor/src/vendor/github.com/astaxie/beego/router.go:769, github.com/astaxie/beego.(*ControllerRegister).ServeHTTP
/harbor/src/server/middleware/middleware.go:52, github.com/goharbor/harbor/src/server/middleware.New.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/security/security.go:75, github.com/goharbor/harbor/src/server/middleware/security.UnauthorizedMiddleware.func1
/harbor/src/server/middleware/middleware.go:57, github.com/goharbor/harbor/src/server/middleware.New.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/security/security.go:62, github.com/goharbor/harbor/src/server/middleware/security.Middleware.func1
/harbor/src/server/middleware/middleware.go:57, github.com/goharbor/harbor/src/server/middleware.New.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/artifactinfo/artifact_info.go:60, github.com/goharbor/harbor/src/server/middleware/artifactinfo.Middleware.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/middleware.go:52, github.com/goharbor/harbor/src/server/middleware.New.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/notification/notification.go:31, github.com/goharbor/harbor/src/server/middleware/notification.Middleware.func1
/harbor/src/server/middleware/middleware.go:57, github.com/goharbor/harbor/src/server/middleware.New.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/orm/orm.go:53, github.com/goharbor/harbor/src/server/middleware/orm.MiddlewareWithConfig.func1
/harbor/src/server/middleware/middleware.go:57, github.com/goharbor/harbor/src/server/middleware.New.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/csrf/csrf.go:48, github.com/goharbor/harbor/src/server/middleware/csrf.attach.func1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/vendor/github.com/gorilla/csrf/csrf.go:297, github.com/gorilla/csrf.(*csrf).ServeHTTP
/harbor/src/server/middleware/csrf/csrf.go:68, github.com/goharbor/harbor/src/server/middleware/csrf.Middleware.func2
/harbor/src/server/middleware/middleware.go:57, github.com/goharbor/harbor/src/server/middleware.New.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/session/session.go:34, github.com/goharbor/harbor/src/server/middleware/session.Middleware.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/log/log.go:33, github.com/goharbor/harbor/src/server/middleware/log.Middleware.func1
/harbor/src/server/middleware/middleware.go:57, github.com/goharbor/harbor/src/server/middleware.New.func1.1
/usr/local/go/src/net/http/server.go:2046, net/http.HandlerFunc.ServeHTTP
/harbor/src/server/middleware/requestid/requestid.go:43, github.com/goharbor/harbor/src/server/middleware/requestid.Middleware.func1

• Harbor dependent services:
  • Context info of postgreSQL
  • Context info of Redis
  • Context info of storage
• Log files: Collect logs and attach them here if have.
• Kubernetes: How Kubernetes access was provided (what cloud provider, service-account configuration, ...).

[GeneralFox]

Does a workaround exists for this issue ?
Because I've noticed some performance issue on project loading and those are related to the chartmuseum call for index.yaml

[....]
2021-12-30T07:35:02Z [ERROR] [/server/v2.0/handler/project.go:435][requestID="815ee84a56c792fc7740c63b2df2fac5"]: failed to populate propertites for project XXX-thirdparty, error: get chart count of project 9 failed: get content failed: send request GET /XXX-thirdparty/index.yaml failed: Get "https://XXX-prod-cluster-harbor-harbor-chartmuseum/XXX-thirdparty/index.yaml": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
--
Thu, Dec 30 2021 8:35:02 am | 2021-12-30T07:35:02Z [ERROR] [/server/v2.0/handler/project.go:435][requestID="815ee84a56c792fc7740c63b2df2fac5"]: failed to populate propertites for project gian, error: get chart count of project 10 failed: get content failed: send request GET /gian/index.yaml failed: Get "https://XXX-prod-cluster-harbor-harbor-chartmuseum/gian/index.yaml": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Thu, Dec 30 2021 8:35:02 am | 2021-12-30T07:35:02Z [ERROR] [/server/v2.0/handler/project.go:435]
[....]

I was able to reproduce the problem with a fresh install of Harbor 2.4 with Helm and chartmuseum enabled, disabling chartmuseum and upgrading via helm has fixed the performance issue and there is no problem on harbor side.
Unfortunately with the operator I was not able to do that and I'm in the same situation as @chlins

[steven-zou]

@GeneralFox

A possible workaround is you can find the core deployment and edit it to remove the WITH_CHARTMUSEUM env. This may fix the problem.