Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker.io proxy cache failing after V2.2 migration #14791

Closed
dioguerra opened this issue Apr 30, 2021 · 23 comments
Closed

docker.io proxy cache failing after V2.2 migration #14791

dioguerra opened this issue Apr 30, 2021 · 23 comments
Assignees

Comments

@dioguerra
Copy link

docker pulling some images through the proxy cache does not persist them.

Expected behavior and actual behavior:
Expected the images if they do not exist in proxy cache to be downloaded and persisted locally until deleted (by retain policy for example)

Steps to reproduce the problem:
Setup docker.io proxy cache on harbor registry and pull through it:
docker pull registry.foo.bar/docker.io/osixia/openldap:1.3.0

Versions:
Installed with harbor-helm

dependencies:
- name: harbor
  version: 1.6.1
  repository: https://helm.goharbor.io

Additional context:

  • Behind OIDC setup
  • Can only log up to INFO, nothing else is shown related to this
2021-04-30T09:20:40Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:22:23Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:23:15Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:24:23Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/webhippie/redis:latest, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:27:51Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/webhippie/redis:latest, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:28:04Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:29:26Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:30:09Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:53:18Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:53:18Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
@dioguerra
Copy link
Author

Just confirmed that this actually also affects repositories/images that worked before.

Everything that is moving through docker.io proxy is getting always re-fetched
Can you please confirm @ninjadq? This is important

Previous to finding this, we where using Version 2.1 (installed from respective helm chart) and it worked flawlessly

@dioguerra dioguerra changed the title docker.io proxy cache failing for select repos/images docker.io proxy cache failing after V2.2 migration Apr 30, 2021
@dioguerra
Copy link
Author

Another problem i found, testing the pull of python:2.6 from the docker.io proxy
docker pull registry.foo.bar/docker.io/library/python:2.6

i get:

2021-04-30T15:33:04Z [WARNING] [/controller/proxy/local.go:132]: failed to remove artifact, error http status code: 401, body: 
2021-04-30T15:33:05Z [WARNING] [/controller/proxy/local.go:132]: failed to remove artifact, error http status code: 401, body: 
Error response from daemon: unknown: resource not found: repo docker.io/library/python, tag 2.6 not found

This is a proxy cache. The artifact indeed does not exist visually on the Portal. I can also not see it in the DB (am i searching in the correct place?):

SELECT ar.repository_name, ar.digest, ar.pull_time, ar.media_type, tag.artifact_id, tag.name
	FROM public.artifact AS ar
	INNER JOIN public.tag AS tag
		ON ar.repository_id = tag.repository_id
	WHERE repository_name LIKE 'docker.io/library/python%'
	ORDER BY tag.name ASC;
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"

This is a proxy-cache that is public. If tag needs to be deleted it should be handled by the system. So, error returned is also misleading...

@ywk253100
Copy link
Contributor

@stonezdj Please have a look

@dioguerra
Copy link
Author

dioguerra commented Jul 13, 2021

Related:

2021-07-13T16:40:18Z [DEBUG] [/server/middleware/repoproxy/proxy.go:252]: Failed to ensure tag {Repository:docker.io/webhippie/redis Reference:latest ProjectName:docker.io Digest: Tag:latest BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to latest

This seems like a concurrency problem?

@vzaitsev77ionos
Copy link

For me looks like caching works if I use not library image, but do not work (not cached, not shown in UI) if I use library variant.
harbor.domain.name/dockerhub/library/redis:6.2 <- not cached or shown in UI
vs
harbor.domain.name/dockerhub/bitnami/redis:5.0.13-debian-10-r5 <- looks like cached and shown in UI

@Morriz
Copy link

Morriz commented Nov 1, 2021

using 2.4.0 and this is still an issue...

@Steuf
Copy link

Steuf commented Dec 6, 2021

Same problem here on version 2.4.0 sometime images was cached, sometime never (on public or private repo)...

@tim-sendible
Copy link

Same issue on v 2.4.1

@tabnul
Copy link

tabnul commented Feb 1, 2022

Same issue here, using 2.4.1. For me not limited to Library images.
Behaviour seems to differ when i pull it from K8s/k3s (imagepullpolicy: always) versus when pulled via manual docker pull:

K8s Pull:
2022-02-01T20:06:38Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: gcr_cache/external-dns/external-dns:v0.10.2, digest: is not found in proxy cache, fetch it from remote repo
--> nothing cached

Manual docker pull:

2022-02-01T20:08:49Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: gcr_cache/external-dns/external-dns:v0.10.2, digest: is not found in proxy cache, fetch it from remote repo
2022-02-01T20:08:49Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: gcr_cache/external-dns/external-dns:, digest:sha256:1fe47236861e3046b1d7b64225f60e6267d23b91a3bd24980911d4c97e641d2d is not found in proxy cache, fetch it from remote repo
2022-02-01T20:08:50Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: gcr_cache/external-dns/external-dns:, digest:sha256:4cf7d828f7c26f62bce71fa69a53c327d1d036af3ab45d712342041b4bec3884 is not found in proxy cache, fetch it from remote repo
2022-02-01T20:09:10Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'InternalArtifact' on topic 'PUSH_ARTIFACT': ID-11, Repository-gcr_cache/external-dns/external-dns Tags-[] Digest-sha256:4cf7d828f7c26f62bce71fa69a53c327d1d036af3ab45d712342041b4bec3884 Operator-harbor#proxy-cache-service OccurAt-2022-02-01 20:09:10
2022-02-01T20:09:10Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'P2PPreheat' on topic 'PUSH_ARTIFACT': ID-11, Repository-gcr_cache/external-dns/external-dns Tags-[] Digest-sha256:4cf7d828f7c26f62bce71fa69a53c327d1d036af3ab45d712342041b4bec3884 Operator-harbor#proxy-cache-service OccurAt-2022-02-01 20:09:10
2022-02-01T20:09:10Z [INFO] [/controller/event/handler/webhook/artifact/artifact.go:75]: []
.........

--> It is cached.

For some reason some other images did get cached, i dont see the logic.
In this case i use a 'Docker Registry' , configured to gcr.io. Maybe it has something to do with this.

@fmalykh
Copy link

fmalykh commented Jun 1, 2022

Same issue on v2.5.0-98e1b82f

1 similar comment
@sacrificeme
Copy link

Same issue on v2.5.0-98e1b82f

@github-actions
Copy link

github-actions bot commented Jul 6, 2022

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label Jul 6, 2022
@tim-sendible
Copy link

This is still an issue.

@github-actions github-actions bot removed the Stale label Jul 7, 2022
@RonanQuigley
Copy link

RonanQuigley commented Aug 18, 2022

Same issue for me on v2.5.1-b0506782.

@shovelend
Copy link

Any news on this one? It's still an issue on v2.5.

@github-actions
Copy link

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label Oct 24, 2022
@msotoperdomo
Copy link

still an issue in v2.6.1

@github-actions github-actions bot removed the Stale label Nov 18, 2022
@shyamjos
Copy link

We also facing this issue in v2.6.1

@stonezdj
Copy link
Contributor

We also facing this issue in v2.6.1

Please specify the container images that fails to pull by proxy cache?

@github-actions
Copy link

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label Jan 28, 2023
@sfxworks
Copy link

sfxworks commented Feb 9, 2023

Having this issue too

2023-02-09T20:42:37Z [ERROR] [/server/middleware/repoproxy/proxy.go:103]: failed to proxy manifest, fallback to local, request uri: /v2/docker/itgz/minecraft-server/manifests/latest, error: http status code: 401, body: 

Confirmed the creds are correct. Even remade new creds.

Same issue on three separate harbor registries.

@sfxworks
Copy link

sfxworks commented Feb 9, 2023

Ah it was a typo in the image. I wish the error was more clear.

@github-actions github-actions bot removed the Stale label Feb 10, 2023
@stonezdj
Copy link
Contributor

stonezdj commented Mar 6, 2023

Summary of this issue: the image tag is not guaranteed to be cached in Harbor, because it depends on the docker client's behavior, if the tag/digest information is cached local, the docker client just sends a pull by digest request to the server, Harbor could not cache this image by tag, but cache it with digest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests