Dockerhub pull request limits - additional registry/docs

[planetf1]

See https://docs.docker.com/docker-hub/download-rate-limit/ & some useful analysis at https://www.openshift.com/blog/mitigate-impact-of-docker-hub-pull-request-limits

Need to consider

• effect on build process
• effect on consumers
• whether we need to save base images (as those images will have restrictions)
• use of other registries (github, azure, quay.io) and/or premium subscriptions

tbc....

[planetf1]

Dockerhub is now offering exemption for qualifying open source projects -> see https://www.docker.com/blog/expanded-support-for-open-source-software-projects/
Will see if we can get this in place cc: @jmertic

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

During an attempt to test the 2.10 release #5211 , the following was noted when trying to setup egeria:

 Warning  Failed          19s               kubelet            Failed to pull image "docker.io/odpi/egeria:2.10": rpc error: code = Unknown desc = Error reading manifest 2.10 in docker.io/odpi/egeria: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Following https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/ we see:

TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:odpi/egeria:pull" | jq -r .token)

and decoding at jwt.io :

{
  "access": [
    {
      "type": "repository",
      "name": "odpi/egeria",
      "actions": [
        "pull"
      ],
      "parameters": {
        "pull_limit": "100",
        "pull_limit_interval": "21600"
      }
    }
  ],
  "aud": "registry.docker.io",
  "exp": 1622154518,
  "iat": 1622154218,
  "iss": "auth.docker.io",
  "jti": "fTZgVmovxlHuKuoo9K1Y",
  "nbf": 1622153918,
  "sub": ""
}

curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/odpi/egeria/manifests/2.10 2>&1
HTTP/1.1 200 OK
content-length: 19568
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:e3c61ab251ec404b05d0dc92b136a7823b245690989e1b20badf30c7c6d0d792
docker-distribution-api-version: registry/2.0
etag: "sha256:e3c61ab251ec404b05d0dc92b136a7823b245690989e1b20badf30c7c6d0d792"
date: Thu, 27 May 2021 22:25:35 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 100;w=21600

This doesn't seem to indicate an issue..... as our limit is 100, with a limit remaining of 100

[planetf1]

It looks as if the image limits are now controlled by IP address - so this limit may only apply to my k8s cluster - hence the odd results above (initiated from laptop)

Or the entire cloud provider...

If so we may have less of a problem. See https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ . However at this time quay.io appears to remain free/unlimited for public repos

[planetf1]

dockerhub is useful for 'visibility', so we may be able to publish to both, and have our charts pointing to quay.io.

[planetf1]

Even with a free account, these limits are slightly higher if authenticated when doing a pull.
Additionally, rather than have the potential for sharing any free quotas with other users appearing at same IP (which may depend on network topology/cloud provider) this uses a unique user id for 'me'.

Additionally if an org is making extensive use of images they can either build locally, or pull the image into their local container registry.

As an example, for openshift by following https://docs.openshift.com/container-platform/4.7/openshift_images/managing_images/using-image-pull-secrets.html

oc create secret docker-registry dockerhub --docker-server=docker.io --docker-username=MYUSERID --docker-password='MYPASSWORD --docker-email=MYEMAIL
oc secrets link default  dockerhub --for=pull

(For odpi-egeria-lab we currently use the 'default' serviceaccount - the egeria-lab chart is different)

With this done images would then load & containers start

[planetf1]

May not be request-limiting, but noticing more occurances of outages connecting to dockerhub

• it would be prudent to start looking at publishing to an additional repository soon.....

ie

  Warning  Failed          17m                  kubelet            Failed to pull image "docker.io/odpi/egeria-configure:2.11-SNAPSHOT": rpc error: code = Unknown desc = error pinging docker registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 161.26.0.8:53: read udp 10.242.128.44:39047->161.26.0.8:53: i/o timeout

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

This is affecting operator dev at times, AND this will impact the Dojo as we will be asking many users to retrieve the containers.
It needs fixing ASAP....
cc: @davidradl

[planetf1]

Build process for egeria now being updated to publish to both dockerhub and quay.io
We know we have users pulling from docker, plus it's the most well known registry, so for now we will continue to publish to both

References in the helm charts in egeria-charts will be updated as the container images are available on quay.io

• egeria
• egeria-configure
• jupyter
• egeria-ui
• egeria-react-ui
• egeria-database-connectors
• egeria-connector-sas-viya
• egeria-k8s-operator (scripts updated, no full ci/cd publish in place yet)

[planetf1]

Just waiting on sas-viya fix to be merged (approval required), other images done.
Charts updated

[planetf1]

Verified our charts.

However of note is that 'nginx' (which we use for the egeria UI, and in future may consolidate more UI access under) fails ie:

  Warning  Failed          56s                kubelet            Failed to pull image "docker.io/nginx": rpc error: code = Unknown desc = Error reading manifest latest in docker.io/library/nginx: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Not yet clear which nginx image on quay.io is appropriate.

[planetf1]

Changes to our images are now almost done,

However the same issue affects any third party containers we use. WIll close the main issue and track these on a case by case basis:

• see Helm charts - pull request limits on nginx egeria-charts#79 specifically for nginx

[planetf1]

Response from nginx team - their images are also published to the non-limited container registry:
https://gallery.ecr.aws/nginx/nginx

Therefore I'm going to update our references to go here instead of dockerhub

[planetf1]

Changes made in charts (merged) and egeria-ui (PR open: cc: @lpalashevski @sarbull )

Note public amazon registry is public.ecr.aws