Skip to content

harbor-registry timeouts when connecting to Redis with TLS enabled #21913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
z0rc opened this issue Apr 17, 2025 · 1 comment
Closed

harbor-registry timeouts when connecting to Redis with TLS enabled #21913

z0rc opened this issue Apr 17, 2025 · 1 comment
Assignees

Comments

@z0rc
Copy link

z0rc commented Apr 17, 2025

I'm running Harbor on AWS EKS, deployed via helm chart. External redis on AWS Elasticache.

I tried recently released support of Redis TLS. So far it's working on all helm components except harbor-registry. Example log line:

time="2025-04-16T20:54:28.581348803Z" level=error msg="redis: error connecting: read tcp 10.103.94.152:51858->10.103.130.14:6379: i/o timeout" go.version=go1.23.8 instance.id=046ea105-712a-44d1-aca6-74153ca4c969 redis.connect.duration=10.00192435s service=registry version=v2.8.3-21-gea065fef 

In the same time other harbor components were connecting just fine to Redis via TLS.

I have one suspicion why it's so. You added tls support for redis at goharbor/distribution#5, yaml key is enabletls. But in chart yaml key is enableTLS. As yaml is case sensitive, the keys don't match, resulting in harbor-registry ignoring this option and trying to connect to redis without TLS enabled, which results in i/o timeout. i/o timeout isn't a connection timeout, connection is established but communication isn't working as protocols don't match.

PS It's strange to see you forking distribution/distribution to add redis tls support. You actively resisted forking distribution when you were asked about adding modern AWS auth support (#12888), because it would be available in next major upstream release, distribution 3.0. Same story goes with Redis TLS, it isn't supported by distribution 2.8, but is supported by distribution 3.0. But instead of waiting for 3.0, you forked and patched 2.8.

@MinerYang MinerYang self-assigned this Apr 21, 2025
@MinerYang MinerYang transferred this issue from goharbor/harbor-helm Apr 22, 2025
@MinerYang
Copy link
Contributor

Hi @z0rc ,

Thanks for reporting this issue. We will fix this issue in the next patch release which will be v2.13.1.

For current Harbor v2.13.0 users that using external redis with TLS, please follow the workaround to modify the config of registry.

Install via harbor-offline installer

Step1 stop the current harbor instance

sudo docker compose down -v

Step2 modify config.yml fro registry

cd  /path/harbor/install/diretcory

sudo vi common/config/registry/config.yml

### edit the redis config block 
### change the "enableTLS: true" to "enabletls: true"
### save the changes

Step3 restart harbor instance check the functionality works as normal

sudo docker compose up -d

### check all harbor pods are running and healthy
### push a new image to harbor
### no error logs of redis connection in the registry pod
### blob/repository data wrote to redis db 1

Install via harbor-helm

Step 1 edit the registry config map

 kubectl edit cm harbor-registry -n <namespace>
### edit the redis config block 
### change the "enableTLS: true" to "enabletls: true"
### save the changes

Step2 rollout the registry pod

kubectl rollout restart deployment harbor-registry -n <namespace>

Step3 Verify the new config been mounted correctly with redis.enabletls: true

$ kubectl exec -it harbor-registry-5c947d7f78-986f9 -c registry -n <namespace> -- cat /etc/registry/config.yml  | grep -A6 redis
  ...
--
  redis:
  addr: xxx.xxx.xx.xxx:6379
  db: 2
  readtimeout: 10s
  writetimeout: 10s
  dialtimeout: 10s
  enabletls: true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants