CI: Set up a container cache

[leofang]

As we continue optimizing over the CI performance (ex: #1156), pulling a container, however "vanilla" it is (such as bare Ubuntu or manylinux), becomes a bottleneck. For example, cibuildwheel takes >30 s to pull a manylinux2.28 container.

Unless someone is willing to overhaul and maintain a new CI design, I'd like to insist that we stay with vanilla images, and set up a container cache to accelerate pulling.

[leofang]

It seems this is blocked by https://github.com/nv-gha-runners/roadmap/issues/242.

[cryos]

There is already a docker pull through cache in place for the standard NV GHA runners referenced in the document you linked to @leofang.

EDIT: Adding relevant quote from https://github.com/nv-gha-runners/roadmap/issues/242:

While our environments do have DockerHub pull-through caches configured by default, they only cache compressed image layers. The compressed image layers will be fetched quickly from the local network cache, but still need to be decompressed on each ephemeral virtual machine. For large images, this decompression can take a lot of time.

Switching to zstd compression has been mentioned as a promising improvement to improve decompression times.

[vyasr]

RAPIDS switched to zstd compression (see rapidsai/ci-imgs#281) and saw very good results. If you control your images that's a good way to make this cheaper. If you're using upstream manylinux images then you can't do that much yourself though.

And yes, the upstream pull-through cache should already be helping you here.

[leofang]

Based on the above (and internal) conversations it sounds like we already have a cache in place. Since there is currently no plan to maintain our own containers, there is nothing to do. Thanks, guys! Closing.