Pulls from proxy cache not updating DB, S3 or Redis

[Lep3188]

We are running harbor v2.11.1-6b7ecba1on a Kubernetes cluster, Using Postgres DB 13.15 and for Storage AWS S3.
In our registry we had a situation which made us notice some discrepancies between the index manifests digest in DB, S3 and external registry.

We would like to know if what we are seeing is normal behavior on Proxy Cache images that have changed over time in the external repo.

Below you will see different scenarios we have seen. These are all for images that have OCI Index. On all of these scenarios we expect Harbor to validate if there is a new digest in the Proxied Repo, update our DB and S3 storage with it and provide the user with the new image. By not seeing a change locally we think that we are just passing through what the proxied registry has.

How we assume Harbor should work:

1. User does a docker pull for Proxy cache image
2. Registry compares with external repo for the newest digest.
3. Registry Updates the value in DB due to difference.
4. S3 digest changes to the new one.
5. User machine receives the external digest not what we had on DB previously.

First Scenario: Pull by tag sometimes updates DB and doesn't update S3.
Assume we already have a cached image that hasn’t been pulled in one month. We can see in the external registry that a update was done to the specific tag and the Manifest digest changed. When a user does a pull sometimes the DB changes it to another value that does not match, but in the S3 bucket nothing changes at all.
Also, if the DB changes we have an example like the one below that it changed to a digest that does not match the external one. That pull and every other after that one will serve the external registry digest only.

Digests we see:
External registry: sha256:42124d7a0f4d3fcb82524a0fee72a513a8e575e7398e7ddca8f380a57c76146f
Our DB before pull: sha256:6ef06e35dee686ff88448aba45cfc8076457a138eea0413b4e38c45227f3ad4c
S3 storage: Has nothing in the manifests directory.
DB after pull: sha256:b2dddc9d80bf40665f452fd5b42a942ad9e83c7a651c0d5b991d9b3a0c0de930
When pulling Users get: sha256:42124d7a0f4d3fcb82524a0fee72a513a8e575e7398e7ddca8f380a57c76146f

Second Scenario: Pull doesn't update DB or S3 on constantly used images but we have matching data locally between DB and S3

On the second scenario we use an image that is constantly pulled (we have pipelines that can run these every 15 minutes every day). For this scenario since the images are pulled often we can see that there is a S3 manifest directory with digest that matches what we have in the DB.

On this second scenario we believed this could be caused due to the redis cache. But since the registry is serving the external registry digest instead of the local, I think we are just having issues. In some very rare cases sometimes it can pull the local digest but after a repull it goes back to the external and not what is cached.

Digests we see:
External Registry: sha256:7a5342b7662db8de99e045a2b47b889c5701b8dde0ce5ae3f1577bf57a15ed40
Our DB: sha256:605644ccbf77342e408186a4c4f1a97cdbae8623f712382ad632690b9d01f482
S3 Storage: sha256:605644ccbf77342e408186a4c4f1a97cdbae8623f712382ad632690b9d01f482
When pulling Users get: sha256:7a5342b7662db8de99e045a2b47b889c5701b8dde0ce5ae3f1577bf57a15ed40

There is a 3rd scenario which is the same as 2nd just that there is no matching between s3, DB and external registry and nothing gets updated after a pull.

[Lep3188]

Looking further into the DB I noticed that in some scenarios the Index digest in harbor wouldn't match a Proxy Cache Repo's but the image within would have the proper digest. I believe this would be due to the other architectures having their own index in hub.docker.com for example.

But the interesting part is that when I pull a image like hub.docker.com/library/docker:stable, the digests are the following:

hub.docker: sha256:8f71deccd0856d8a36db659a8c82894be97546b47c1817de27d5ee7eea860162
My DB on the Index: sha256:ad23a2ac3bbf789ca412564f6d7f3d91db6645d52915638eb2d73093d6788048
My DB and UI on the amd64 image: sha256:8f71deccd0856d8a36db659a8c82894be97546b47c1817de27d5ee7eea860162
S3: sha256:fd4d028713fd05a1fb896412805daed82c4a0cc84331d8dad00cb596d7ce3e3a
What I get when I pull: sha256:fd4d028713fd05a1fb896412805daed82c4a0cc84331d8dad00cb596d7ce3e3a

For that fd4d0 digest I don't have anything on my Database, not as an artifact, blob, etc... But I do have it on the S3 and is where log below is fetching it. So this is another behavior different to the scenarios above that would provide the digest of external registry.

Log I am seeing:
2025-03-18T20:32:05Z [ERROR] [/server/middleware/repoproxy/proxy.go:122]: failed to proxy manifest, fallback to local, request uri: /v2/ext.hub.docker.com/library/docker/manifests/sha256:fd4d028713fd05a1fb896412805daed82c4a0cc84331d8dad00cb596d7ce3e3a, error: login to dockerhub error: Post "https://hub.docker.com/v2/users/login/": EOF

Similar to the scenarios I shared on the initial thread I have this image which appears in our S3 storage (with different digest) and there is nothing on our DB. It pulls with the external repo due to not being found (log below).

2025-03-05T09:11:07Z [WARNING] [/server/middleware/repoproxy/proxy.go:208]: Artifact: ext.hub.docker.com/library/golang:, digest:sha256:43c094ad24b6ac0546c62193baeb3e6e49ce14d3250845d166c77c25f64b0386 is not found in proxy cache, fetch it from remote repo

Would the reason a new record or the image is not updated is due to conflict with S3? is the only thing that comes to mind but I don't want to alter anything.

[stonezdj]

You cannot use the digest in the database or s3 to compare, for image index, it is cached in the redis, it is not exposed to the end user. and the manifest content/digest in the database is different with the manifest content/digest in the docker hub, because the local one is partial of the content in dockerhub.

[Lep3188]

@stonezdj thanks for the reply.

After your comment I did a test with an image that has not been pulled in a few months and that hub.docker had updated a few days ago. After pulling I could see that Index digest switched to another one that did not match external repo like you said and the image digest had the new one (all in the DB). But I never had saw a change in the S3 bucket, Should I expect a change to occur instantly or does it stays cache in Redis first and then moved to S3?

We are trying to learn how this works because we use a CDN (Content Delivery Network) to cache locally the blobs and in certain proxy cache images received a "manifest unknown" (it validates with s3 manifest). When I looked at all the images that had the issue they all shared the same manifest digest (this is on different images from different repos). We had another image which had a old digest and when pulling from CDN it would grabbed the s3 content while regular pulls would get the current digest of hub.docker.

Basically we just want to know that the backend is working as it should and there is no problem with our harbor storage and that we are just doing something wrong when caching the blobs on other regions.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.