Devcontainer Cache w/GitHub Actions Not Working?

[metaskills]

I have an open source project (https://github.com/customink/crypteia) I use Codespaces with. Prebuilds are tuned on (https://github.com/customink/crypteia/pkgs/container/crypteia-devcontainer) and I have the GitHub Action set (https://github.com/customink/crypteia/blob/main/.github/workflows/test.yml#L30) to use the same devcontainer. No matter what, the Actions will take many minutes vs use the devcontainer. From the logs I can see it rebuilding again and trying to push afterward and finding out the image is already present. What could keep this from working or is it expected that it will not pull the image and it always builds it?

[Chuxel]

This may be due to some sort of Docker caching issue when features are used. This is described in devcontainers/cli#153 . A PR was merged for a workaround that should be getting released with the next version of the CLI and then be picked up by the Action.

[Chuxel]

A few notes:

1. This action does not currently use Codespace prebuilds if that is what you are referring to - an image pre-built image can use cache from. I know this has been discussed but is not there now.

2. Just to be clear, the imageName is the name of the output image from the build, it does not override the input. So looking at your workflow, it would try to rebuild each time. Setting cacheFrom: ghcr.io/customink/crypteia-devcontainer would treat the published image as a cache instead.

3. To avoid pushing, set push:never

Generally what I've been doing personally is that I have a dev container config that is for pre-creating an image, then another one that just references the image that I use to run.

You can use cacheFrom instead to speed up the build process, but that was impacted by devcontainers/cli#153 in some cases per my original reply.

@stuartleeks may have other tips.

[metaskills]

Thanks, so I can use cacheFrom and push: never described above when this project picks up the updated CLI?

[Chuxel]

Yep - you can add them now, and they should help. The only thing that might happen is that some of the layers aren't cached near term - but should happens less (if at all) on Linux where Actions workflows run. So things may work as expected already.

[metaskills]

So I did a few changes and things got slower. Could it be that a commit to main which triggers a rebuild of the dev container blows it away before it could be used. Other thoughts? rails-lambda/crypteia@482fa89

[Chuxel]

Yeah, there's a few things that could be going on:

1. cacheFrom should be pulling down the image for a cache check, so I wouldn't expect an update of ghcr.io/customink/crypteia-devcontainer to have an effect unless:

   1. The Dockerfile changed since the image was published
   2. The upstream image (looks like mcr.microsoft.com/devcontainers/rust) updated since the image was published. You can pin to a specific version of the rust image (or a SHA) to make sure that doesn't happen. Version 0.202.9 was released 6 days ago.
   3. Right now your features are using "latest" for the feature version. Since Universe is coming up, there's lots of fixes coming in, so you may be getting slightly different contents at different points in time. Depends a bit on timing. You can specify a feature version explicitly to prevent that from happening. If you click on the "package" for the feature you'll see all the available versions for it:. e.g. node 1.1.1 updated 4 days ago. https://github.com/devcontainers/features/pkgs/container/features%2Fnode

2. Codespace pre-builds do not create a pre-built image. That stores an entire working container, while the Action currently supports only pre-built images. (FWIW I think "pre-cached Codespace" is probably a more accurate description for Codespace pre-builds.) I mention this one because I don't see the workflow for pre-building or publishing ghcr.io/customink/crypteia-devcontainer to know for sure (and if the image is published, it's private so I can't pull it). Either way, I'd recommending pre-building your image since it will also make pre-building/caching your Codespace much faster... and you can reuse the image in a local dev container too.

   FWIW - You can also have "cacheFrom" and "imageName" set to the exact same image along with "push" as true. This would cause it to publish the image each time, but when you get a cache hit, the push will do nothing since the image is already present in the registry.... assuming cacheFrom is working per the above.

Any of that help? If you point me to the workflow for the ghcr.io/customink/crypteia-devcontainer build and publish I can take a look if that exists.

[Chuxel]

FYI - the cache fix is in the CI action, the dev container CLI, and the VS Code Dev Containers extension. It should also land in Codespaces soon.

[metaskills]

Cool, I'm seeing it working now with something like this. I realize the push was not needed there, but this works.

push: always
imageName: ghcr.io/customink/crypteia-devcontainer
cacheFrom: ghcr.io/customink/crypteia-devcontainer

Want to close this issue?

[Chuxel]

Sure! Glad you were unblocked! We updated docs a bit here as well given what you hit.

[metaskills]

I finally put this to use (https://github.com/customink/dnd-demo/blob/main/.github/workflows/test.yml) and I love it. I have a step that does that push always with imageName and cacheFrom:

        with:
          push: always
          imageName: ghcr.io/customink/dnd-demo-ci
          cacheFrom: ghcr.io/customink/dnd-demo-ci
          runCmd: echo DONE!

Then each step afterward does this.

        with:
          push: never
          cacheFrom: ghcr.io/customink/dnd-demo-ci

And I get the desired output using all the building blocks GitHub Actions provides. Thank you!

[cjam]

I'm trying to get this to work on my project as well. We have a docker-compose devcontainer setup and it unfortunately seems to rebuild the features every time. I've been trying to shave down our build time as the devcontainer build accounts for a pretty good chunk of it right now.