CNCF position on new Docker policy limiting image retention

[caniszczyk]

Starting Nov 1, 2020, Docker is planning to limit retention of images on Docker Hub for free accounts to 6m 1. This will likely affect many CNCF projects that distribute binaries. For example, while the Jaeger projects makes multiple releases per year, it does not mean that all users are upgrading more frequently than 6m. We also have a number of build and CI related images that are not updated that often, and if they start TTL-ing out it will introduce additional maintenance burden.

What is the TOC recommendation on this front? Should CNCF upgrade some of Docker Hub accounts (e.g. starting with graduated projects) to paid plans?

[estesp]

All it takes is a single pull in 6 mo. to reset the counter. My reading of the FAQ makes me think only truly unused images will be affected by the new image retention policy.

If an image has not either been pulled or pushed in the amount of time specified in your subscription plan, the image will be tagged "inactive."

[billimek]

From their FAQ it looks as if the 6m timeline is only for images that haven't been pulled in 6 months as well,

Image retention is based on the activity of each individual image stored within a user account. If an image has not either been pulled or pushed in the amount of time specified in your subscription plan, the image will be tagged “inactive.” Any images that are tagged as “inactive” will be scheduled for deletion. Only accounts that are on the Free individual or organization plans will be subject to image retention limits. A new dashboard will also be available in Docker Hub that offers the ability to view the status of all of your container images.

Which may result in a situation not quite as dire as expected when first reading the updated terms.

[caniszczyk]

@estesp If that is indeed the case, then I am less concerned about this having wide impact

[yurishkuro]

Agreed, if pulls extend the TTL then it should not be an issue.

[kmova]

There also seems to be a rate-limiting on the number of pulls per user.
https://hub.docker.com/pricing

Limited to 100-200 pulls in 6 hours.

This may impact automated pipelines.

[stefanprodan]

Limited to 100-200 pulls in 6 hours.

I think this will have a grave impact for people contributing to OSS. For example, In FluxCD org we use containers in CI to pull base images such as Alpine from Docker Hub. We will have to buy a Docker Hub account and set a pull secret in GH Actions. When people will open PRs, running "FROM alpine" in CI will either hit the 100 pulls limit or fail because the GitHub pull secret doesn't work for GitHub forks.

[estesp]

@stefanprodan what gave you the impression Docker would limit pulls of [their own] official images? (otherwise known as the library/ repo on DockerHub). Who would they charge to remove the limit? Themselves? That would make no sense at all. If there is any confusion around official images, I sure hope Docker will clarify.

[stefanprodan]

@estesp that's a question Docker can answer, on their website there is no exception listed. Unless you pay for a Docker Hub account, you can't pull more than 200 images/6h, this limit is per Docker account.

[tomkerkhove]

What about anonymous pulls? 🤔

[estesp]

@stefanprodan I think you nailed the confusion point. You are talking about pulls/user, and Docker seems to be talking about pulls/image. How can you limit an unauthenticated user on how many pulls they can perform across a time period? You don't know who they are? The pull limits are per image, and what you are paying for is removing the pull limits on your images--that's my understanding.

[stefanprodan]

You are talking about pulls/user, and Docker seems to be talking about pulls/image.

Anonymous pulls are limited to 100/6h based on source/ip.

[stefanprodan]

What about anonymous pulls?

@tomkerkhove my understanding is that there are no anonymous pulls for images owned by a paying customer. Anonymous pulls work for images owned by a free account and they are limited to 100 pulls/6h based on source/ip.

[tomkerkhove]

Hm from the image above it looks like anonymous pulls are still allowed, but just throttled on 100 pulls/6h. Damn this is not ideal for OSS projects. We'll need to move KEDA to something else now as we only get 3.6 pulls per minute.

[justincormack]

@estesp yes we are limiting pulls per image, including official images, and the charge is for the pulling user not the image owner. Anonymous pulls will be rate limited per IP address if the user is unauthenticated, by userid if authenticated.

We understand that this is going to be somewhat inconvenient, and we will have some free plans for open source projects coming before this goes live, but the costs of providing this are not sustainable.

[estesp]

Got it; thanks for the clarification @justincormack. @stefanprodan: seems like your reading comprehension was better than mine in this case 😅

A minor point about IP address tracking: this tends to bite large enterprises in very surprising ways with egress NAT for 1000s of employees ending up only using a handful of IPs.

I assume the same will be true for those using shared services like Actions, Travis, etc. but maybe with a larger range of potential IPs.

I would assume a given IBM location would use up "anonymous" pull rates on a single image (like alpine or ubuntu or nginx) in a matter of minutes or even seconds depend on the geographical site size. We won't know (for that use case) until post-COVID though--it's a good time to generalize 1 IP = 1 user at least for now 😁

[justincormack]

Large organizations should consider running a pull through cache, most CI providers do already. One of the issues of free bandwidth is that people have not felt any need to run local caches for clusters or CI.

Also don't forget the Dragonfly CNCF project for P2P image distribution https://github.com/dragonflyoss/Dragonfly

[bboreham]

@justincormack do you have any reference for running a pull-through cache? I know you can use the Docker registry but not for quay.io, k8s.gcr.io, etc., and alternatives I've come across have disadvantages such as not enforcing authentication. (Maybe that's ok for the 'large organization' case under discussion)

[BretFisher]

more info on technicals of pull limits, with more blog/faq coming next week https://docs.docker.com/docker-hub/download-rate-limit/

[idvoretskyi]

For the record: https://github.blog/2020-09-01-introducing-github-container-registry/

[j-s-3]

Proxying Docker images via an OSS instance of Nexus Repository Manager would be one way to reduce concerns around rate limiting from DockerHub.

[bacongobbler]

As a heads up to other CNCF projects: everyone should review their own CI pipelines and make sure they log in to DockerHub before pulling any images - even base images required to run the CI job itself. Docker Hub imposes rate limits based on the originating IP. Since CircleCI (along with many other CI providers) run jobs from a shared pool of IPs, it is highly recommended to use authenticated Docker pulls with Docker Hub to avoid rate limit problems.

This comes in effect in two week's time.

For reference:

• Add Docker authentication to CircleCI jobs helm/helm#8848
• https://circleci.com/docs/2.0/private-images/

[stefanprodan]

@bacongobbler when using GitHub Actions adding the Docker auth will break CI for PRs, the GitHub secrets are not shared in forks and for good reasons.

[estesp]

If anyone is interested in a discussion on long-term ideas around this issue with representation from Docker and most cloud providers/registries, the OCI weekly call today (14 Oct @ 2pm US Pacific) will be covering this topic: https://hackmd.io/El8Dd2xrTlCaCG59ns5cwg#October-14-2020

[caniszczyk]

https://github.com/cncf/servicedesk#my-project-is-affected-by-the-docker-hub-rate-limits-policy-changes-what-can-i-do