Impact of Not Using External Redis/PostgreSQL for Harbor Proxy Cache

[varshabhatnagar]

We're considering using Harbor as a proxy cache with JFrog Artifactory as the source registry. We have 24 AWS EKS clusters, each with ~100 nodes, running on spot instances. Since these nodes recycle weekly, images are pulled frequently, leading to a high load on the registry.

To improve availability and redundancy, we plan to run a Harbor instance in each cluster. However, we're concerned whether to deploy an external Redis and PostgreSQL or rely on the built-in database and cache.

Key concerns:

• Will Harbor’s built-in database and caching mechanisms be sufficient to handle the frequent image pulls, or will we see performance
  degradation?
• What kind of impact (latency, failures) should we expect if we don’t use external Redis and PostgreSQL?
• Are there any benchmarks or guidelines on when external Redis/PostgreSQL becomes necessary for a Harbor proxy cache?

Any insights or best practices would be greatly appreciated!

[MinerYang]

Hi @varshabhatnagar ,

Here is what we've tested , please refer to https://github.com/goharbor/perf/wiki/Harbor-2.10.0.

If anything you found that worth to be discussed, please feel free to communicate with us.

Best,
Miner

[stonezdj]

total nodes > 2400, the postgres and redis should be externanized to get better scalability. the embedded postgres and redis are single stateful pod, can not be scaled up.

[varshabhatnagar]

thanks for your response.
We're planning to run harbor on each cluster independently to eliminate single point of failure and have redundancy.
In that case, we are wondering if we do need to externalize Redis and postgrewsql

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.