502 Bad gateway when harbor-core starts up

[waddles]

When harbor-core pods start up, the first attempt to fetch an image proxied to ECR results in a 502 Bad gateway. Subsequent requests succeed without error but this occurs fairly regularly with busy harbor deployments when autoscaling is configured.

Kubernetes event logs for the client look like this

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image \
"123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/eks/pause:3.5": failed to pull image \
"123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/eks/pause:3.5": failed to pull and unpack image \
"123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/eks/pause:3.5": failed to resolve reference \
"123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/eks/pause:3.5": unexpected status from HEAD request \
to https://harbor.example.com/v2/1234566789012.dkr.ecr.ap-southeast-2.amazonaws.com/eks/pause/manifests/3.5?ns=123456789012.dkr.ecr.ap-southeast-2.amazonaws.com: 502 Bad Gateway

There are no relevant logs from the harbor-core pods but I think it is fetching a token before querying ECR.

Steps to reproduce the problem:

1. Configure a registry mirror for ECR in harbor
2. Configure containerd on clients to request ECR images via harbor proxy
3. Restart or scale up harbor-core pods

Versions:

• harbor version: v2.12.0-9da38ae0
• containerd version: 1.7.23

Additional context:
I have tried changing the startupProbe to use /api/v2.0/health instead of /api/v2.0/ping but that made no difference.

[Vad1mo]

There are quite a few issues on the internet regarding AWS ECR 502 Bad Gateway, I suspect this is more related outside harbor and rather your setup or AWS infrastructure..

[MinerYang]

Could you verify that a manually push/pull could works as expected? And please also check the harbor-registry and the proxy logs in the meantime.

[waddles]

I went searching for logs from all harbor components during a period where my clients got a couple of 502s but all I can find are 200s and 307s which I think makes sense as why would a component log anything if it didn't handle the request?

There were 2 different client nodes that logged a 502 when attempting to pull the pause image for 4 different pods - all daemonset pods which implies they were new nodes starting up at the same time the harbor-core pods were rolled. That also makes sense as a node that's already in service would already have the pause image downloaded. Unfortunately we churn nodes often and I can't access the containerd logs from those ones. When it happens again I'll try and get those logs from containerd.

Is there some debug flag I can add to harbor-core to test this further?
Also is there some document that outlines the request flow through harbor's components when containerd tries to pull an image?

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.