DockerHub rate limiting

[timja]

Service(s)

ci.jenkins.io

Summary

Still getting rate limited on https://github.com/jenkinsci/docker

https://ci.jenkins.io/blue/organizations/jenkins/Packaging%2Fdocker/detail/PR-1383/3/pipeline/413

[2022-05-30T19:43:41.951Z] error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/debian/manifests/sha256:3f1d6c17773a45c97bd8f158d665c9709d7b29ed7917ac934086ad96f92e4510: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[2022-05-30T19:43:41.951Z] make: *** [Makefile:53: build-debian_jdk11] Error 1

dependabot PR swarm never works first time.

follow on to #2837

Reproduction steps

No response

[dduportal]

For context, the issue #2842 describe what the Docker Open Source membership gives us and what it does NOT give us.

The issue raised by @timja here is unrelated to the Open Source program: it is related to the fact that our organizations are only "free team" accounts, with still the rate limit at 200 req / 6h and 3 user seats (ref. https://www.docker.com/pricing/)

Since all of the jobs on ci.jenkins.io for the jenkinsci/docker* projects are building Docker images from the official images such as alpine or ubuntu, they are rate limited at 200 req / 6h.

[dduportal]

We are in discussion with Docker, they told us they we OK to grant us a "Team plan" with at least 5000 req / day, maybe more (not sure yet).
That should make, in the worst case, a threshold increase from 200 to 1250 per 6hours range. That might be enough.

If we can't be upgraded to such a plan, or if the new threshold is not enough, we have the following solutions available:

• Stop authenticating requests during builds and tests AND use public IPs for our agents

  • As per https://docs.docker.com/docker-hub/download-rate-limit/#what-is-the-download-rate-limit-on-docker-hub, each public IP is limited to 100 req / 6h. Easy to apply (1 pipeline change to stop authenticating until deploy stage + 1 ci.jenkins.io config change per type of VM agent)
  • Challenge: have impact on the agents spin-up: VM can be slower to be available + additional costs associated to this setup

• Use a collection of free "dummy" account for pulling and spread the load

  • Easy to apply (programmatic change in the withDockerPullCredentials() pipeline library, using a modulo could do it)
  • Challenge: maintaining this list of accounts (and associated credentials) cannot be automated (missing feature from DockerHub API) so will be painful

• Use a docker "caching" registry that would be authenticated to the DockerHub, and our VMs would be configured to use it.

  • Generic solution that would allow us to be "good citizen" of the DockerHub
  • Challenge: cost to build and maintain (host the service, clean it up, restrict its access, maybe duplicate per cloud or per region, configuration of the agents, etc.)

• Use another registry for base images

  • No more DockerHub API rate limit
  • Challenge: this new registry MUST be available for our users if they want to build the images themselves, this registry must be closely mirroring the "official" DockerHub images, and we usually have to pay for such service (storage, cloud fees, or another form of API rate limit)

[dduportal]

In short-short term (until we have news from Docke Inc.) , we have to be patient and "sequentialize" the builds on ci.jenkins.io

[timja]

Challenge: this new registry MUST be available for our users if they want to build the images themselves, this registry must be closely mirroring the "official" DockerHub images, and we usually have to pay for such service (storage, cloud fees, or another form of API rate limit)

They wouldn't need to access it: https://docs.docker.com/registry/recipes/mirror/#configure-the-docker-daemon

---

Best and cheapest solution I think is an upgraded account, either sponsored by docker hub or CDF / linux foundation

[dduportal]

They wouldn't need to access it: https://docs.docker.com/registry/recipes/mirror/#configure-the-docker-daemon

This documentation is for the "docker caching" solution. In this case, I agree end-user don't need (and forward: they should NOT be allowed) to access the caching registry.

I might have mis-written the scenario "use another registry": this case means changing the value of the FROM instruction of all of our Dockerfiles to the URL of another registry (for instance: FROM alpine:3.15 should be changed to ghcr.io/jenkinsci/alpine:3.15).

[timja]

This is getting rediculous, PRs barely pass at all on the repo, taking many many builds.

jenkinsci/docker#1399
jenkinsci/docker#1398
jenkinsci/docker#1356

In short-short term (until we have news from Docke Inc.) , we have to be patient and "sequentialize" the builds on ci.jenkins.io

I have tried that twice in a row now, still getting failures.

[dduportal]

fair, let's prioritize that today.

Short term goal to unblock these PRs:

• enabling public IP on all the VM agents for ci.jenkins.io (AWS and Azure)
• Modify the shared library to stop logging in to DockerHub

[dduportal]

• fix(ci.jenkins.io) use public IP for Azure VM agents jenkins-infra#2227 => ensure that Azure VM agents have a public Ip (currently applied manually on ci.j). EC2 VM agents already use a public IP.
• chore(Jenkinsfile) stop logging in Docker Engine to DockerHub jenkinsci/docker#1400 to update the build pipeline, in order to avoid authenticating on ci.j

[dduportal]

Update: we received notifications (and were able to confirm) that the DockerHub organizations jenkinsciinfra and jenkins4eval are now under a Team plan.

Let's try to authenticate again the agent for pulls and see the behavior (e.g. do we reach the new limit when dependenbot starts a lot of PRs)

[dduportal]

Sounds like we're doing better! Gotta close this issue unless you start seeing HTTP/429 about rate limit.