New DockerHub Image retention policies will delete unused images after 6 months #121

scottyhq · 2020-08-31T19:00:56Z

https://www.docker.com/pricing/resource-consumption-updates

Starting November 2020, Images untouched for 6 months will be scrubbed.

On one hand, this isn't an issue because this repository stores the complete configuration needed to build any previously tagged image. But eventually someone might want to reproduce a study from a year ago and hit an 'image not found' error.

There are some options... 1) Pangeo has it's own pro Docker account. 2) Start pushing copies of the image to GitHub Packages via an Action.

cc @jhamman @rabernat @salvis2 @TomAugspurger @tjcrone

scottyhq · 2020-12-02T17:53:43Z

noting some alternatives to dockerhub:

AWS ECR as of 12/1/2020 has a public hosting option
https://aws.amazon.com/blogs/aws/amazon-ecr-public-a-new-public-container-registry/
Quay.io, below is an article about limits
https://www.openshift.com/blog/mitigate-impact-of-docker-hub-pull-request-limits

scottyhq · 2021-03-19T16:10:28Z

also now github 'packages' is now 'github container registry' and seems to be more like the other available options, so would be nice to try that out! https://docs.github.com/en/packages/guides/about-github-container-registry

@yuvipanda, @consideRatio what are you using for 2i2c these days?

yuvipanda · 2021-03-20T10:29:34Z

I am trying out sending people to quay.io, run by RedHat (so IBM) - see jupyterhub/repo2docker-action#60. But I think the way to do it now is to just push it to everything - dockerhub, quay.io, github registry, etc. If we match tags and names, should offer some protection against future issues.

cisaacstern · 2022-10-04T00:03:26Z

Following #379, I was unable to pull the latest pangeo/forge:7c87e6c tag from within a Google Dataflow job. Specifically, I hit the following error (xref pangeo-forge/pangeo-forge-orchestrator#156 (comment)):

2022-10-03 10:30:38.775 PDTA worker was unable to start up. Error: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: Get "https://registry-1.docker.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers). This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image.

However, I was:

Able to pull pangeo/forge:7c87e6c with a local docker pull
Able to pull apache/beam_python3.9_sdk:2.41.0 from Docker Hub from within a Dataflow Job

@yuvipanda suggested this second point indicates that egress limits places on the free pangeo/ repo are likely to blame here, and that the sponsored oss status of the apache/ repo may explain why it is pullable in settings where the pangeo/ tags are not. Noting this here because Yuvi asked me to. Our short term workaround, added in pangeo-forge/pangeo-forge-orchestrator#157, is to mirror the specific image tags we need for Dataflow onto gcr.io.

scottyhq · 2022-10-06T03:06:41Z

Just created https://quay.io/organization/pangeo and added @yuvipanda as a member. We can easily push images there as well, but I probably wouldn't get around to it until next week...

yuvipanda · 2022-10-07T16:20:08Z

Thanks a lot, @scottyhq :D Once you push that through we'll start using those as the default.

scottyhq · 2022-10-14T18:23:00Z

Ran into trouble pushing the larger ML images to quay.io ("no space left on device" since images are ~10GB and in theory available disk space is just 14GB)...

2022-10-14T18:02:13.2488555Z [command]/usr/bin/podman --root /tmp/podman-from-docker-Vwqc3N --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs pull docker-daemon:pangeo/pytorch-notebook:f02e5e7
2022-10-14T18:05:33.5241254Z Getting image source signatures
2022-10-14T18:05:33.5288871Z Copying blob sha256:92f5277a940d1ed8a5ae63ef31e93e5ceb3d0a2a444ccac98f8e6e7d1a6e3dcc
2022-10-14T18:05:33.5291615Z Copying blob sha256:4c3b471254bbd61428b9ab0d8b53767b33a658b5dd000058e86a7f1d2a90dc39
2022-10-14T18:05:33.5292233Z Copying blob sha256:17f623af01e277c5ffe6779af8164907de02d9af7a0e161662fc735dd64f117b
2022-10-14T18:05:33.5292767Z Copying blob sha256:26ee70d7754029bb5ee40352e8aa38d8e848140e4795a03e8bc19cdeda4945e4
2022-10-14T18:05:33.5293286Z Copying blob sha256:6dda489a86ed92f56e13b2063b9c7a21f68803975429ee1092a43f9d15543e9d
2022-10-14T18:05:33.5294081Z Copying blob sha256:50777fa7dc21df1d8f7239b6ff7907a0d2e86ec051cb1e796614a1151533ee83
2022-10-14T18:05:33.5294617Z Copying blob sha256:189f855ec36cc1ca4b4322300f77cbd7fe350ead512ef6b78ce652969abf88c8
2022-10-14T18:05:33.5295143Z Copying blob sha256:c96a72b172dd0dfe1f50ca2967a316621a65a744b29d2edc542e64598391884b
2022-10-14T18:05:33.5295650Z Copying blob sha256:51f7226c230dd74d35b977550b50b75449625c2269536364eefa4e5b09540072
2022-10-14T18:05:33.5296178Z Copying blob sha256:ab3700cd20db601257bf2fdb8deca4164ebc4befebeebb7137db85e900bb95eb
2022-10-14T18:08:17.6257230Z Error: writing blob: adding layer with blob "sha256:51f7226c230dd74d35b977550b50b75449625c2269536364eefa4e5b09540072": Error processing tar file(exit status 1): write /srv/conda/envs/notebook/lib/python3.9/site-packages/botocore/data/ec2/2015-04-15/service-2.json: no space left on device
2022-10-14T18:08:17.6366603Z ##[endgroup]

https://github.com/pangeo-data/pangeo-docker-images/actions/runs/3251827256/jobs/5337356216

weiji14 · 2022-10-14T18:41:20Z

Ran into trouble pushing the larger ML images to quay.io ("no space left on device" since images are ~10GB and in theory available disk space is just 14GB)...

Try putting this in the GitHub Actions workflow, see https://github.com/2i2c-org/hub-user-image-template/blob/d6f7d4f42ba4ce9275321f5b3b3d084d8b1fae1a/.github/workflows/build.yaml#L13-L21 and actions/runner-images#2606 (comment)

    - name: cleanup disk space
      run: |
        sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc
        df -h

scottyhq · 2022-10-14T19:41:09Z

Add the following which frees up ~14GB of additional space

pangeo-docker-images/.github/workflows/Build.yml

Lines 78 to 84 in 3b4c8fe

    
               - name: Free up disk space 
        
                 run: | 
        
                   df -h 
        
                   docker image ls 
        
                   sudo apt clean 
        
                   sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc 
        
                   df -h

Pushing to quay.io appears to be quite slow. Building and pushing ml-notebook to DockerHub takes 19min, and then pushing the built image to quay.io takes 18min. Not too concerned with time since we're not building images all the time, but there probably is some way to speed that up...
https://github.com/pangeo-data/pangeo-docker-images/actions/runs/3252252263/jobs/5338272965

After more closely reading https://github.com/redhat-actions/push-to-registry It seems podman is quite different from Docker and a better strategy is probably not to use podman at all and simply use the docker github action to push to multiple recipes https://github.com/docker/build-push-action/blob/master/docs/advanced/push-multi-registries.md !

scottyhq · 2022-10-14T19:49:53Z

Even the Publish workflow of retagging to 'latest' and calver and pushing takes 5+min versus seconds on DockerHub so it's probably not recognizing that the layers already exist...
https://github.com/pangeo-data/pangeo-docker-images/actions/runs/3252503512/jobs/5338784571

But, leaving it for now! @yuvipanda @cisaacstern you can now pull quay.io/pangeo/forge:2022.10.14 !

yuvipanda · 2022-10-18T00:37:05Z

\o/ thanks a lot, @scottyhq!

scottyhq mentioned this issue Aug 31, 2020

New DockerHub Image retention policies will delete unused images after 6 months pangeo-data/pangeo-binder#157

Open

scottyhq mentioned this issue Dec 15, 2020

dockerhub expiration policies and alternative image storage UW-GDA/uwgda-image#12

Open

cisaacstern mentioned this issue Oct 4, 2022

Set default container_image & mirror it to gcr.io pangeo-forge/pangeo-forge-orchestrator#157

Merged

scottyhq mentioned this issue Oct 14, 2022

bump github actions, add quay.io #391

Merged

scottyhq mentioned this issue Oct 17, 2022

Switch to buildx for github actions #394

Merged

scottyhq closed this as completed in #394 Oct 17, 2022

yuvipanda mentioned this issue Mar 20, 2023

Audit 2i2c + community image use for Docker Hub risk due to removal of Free Team orgs. 2i2c-org/infrastructure#2361

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New DockerHub Image retention policies will delete unused images after 6 months #121

New DockerHub Image retention policies will delete unused images after 6 months #121

scottyhq commented Aug 31, 2020 •

edited

Loading

scottyhq commented Dec 2, 2020

scottyhq commented Mar 19, 2021 •

edited

Loading

yuvipanda commented Mar 20, 2021

cisaacstern commented Oct 4, 2022

scottyhq commented Oct 6, 2022 •

edited

Loading

yuvipanda commented Oct 7, 2022

scottyhq commented Oct 14, 2022

weiji14 commented Oct 14, 2022 •

edited

Loading

scottyhq commented Oct 14, 2022 •

edited

Loading

scottyhq commented Oct 14, 2022

yuvipanda commented Oct 18, 2022

New DockerHub Image retention policies will delete unused images after 6 months #121

New DockerHub Image retention policies will delete unused images after 6 months #121

Comments

scottyhq commented Aug 31, 2020 • edited Loading

scottyhq commented Dec 2, 2020

scottyhq commented Mar 19, 2021 • edited Loading

yuvipanda commented Mar 20, 2021

cisaacstern commented Oct 4, 2022

scottyhq commented Oct 6, 2022 • edited Loading

yuvipanda commented Oct 7, 2022

scottyhq commented Oct 14, 2022

weiji14 commented Oct 14, 2022 • edited Loading

scottyhq commented Oct 14, 2022 • edited Loading

scottyhq commented Oct 14, 2022

yuvipanda commented Oct 18, 2022

scottyhq commented Aug 31, 2020 •

edited

Loading

scottyhq commented Mar 19, 2021 •

edited

Loading

scottyhq commented Oct 6, 2022 •

edited

Loading

weiji14 commented Oct 14, 2022 •

edited

Loading

scottyhq commented Oct 14, 2022 •

edited

Loading