Cannot access the image in subsequent workflow steps

[YevheniiSemendiak]

Behaviour

Steps to reproduce this issue

Easier will be just to put here a part of my workflow in which (1) build image with docker/build-push-action and load=true and then (2) run docker container using this image.

      - name: Get image tags
        id: get-image-type
        run: |
          if [[ ${{ matrix.base-image }} =~ runtime ]]; then
            export BASE_IMAGE_TYPE="runtime";
          elif [[ ${{ matrix.base-image }} =~ devel ]]; then
            export BASE_IMAGE_TYPE="devel";
          fi
          echo "::set-output name=BASE_IMAGE_TYPE::$BASE_IMAGE_TYPE"
          echo "::set-output name=platform_image_tag::neuromation/base:built-$BASE_IMAGE_TYPE"
      - name: Build image
        uses: docker/build-push-action@v2
        with:
          context: .
          builder: ${{ steps.buildx.outputs.name }}
          build-args: |
            BASE_IMAGE=${{ matrix.base-image }}
          load: true
          tags: ${{ steps.get-image-type.outputs.platform_image_tag }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: List pip packages
        run: |
            docker run --tty --rm ${{ steps.get-image-type.outputs.platform_image_tag }} pip list

Expected behaviour

Docker container is running with built image.

Actual behaviour

First flow run fails with Unable to find image 'neuromation/base:built-devel' locally
docker: Error response from daemon: manifest for neuromation/base:built-devel not found: manifest unknown: manifest unknown.

However, if I rerun the jobs it succeeds.

Configuration

• Repository URL (if public): https://github.com/neuro-inc/neuro-base-environment/blob/2e11da9ee7eae7912908e5b7941573b7312b99ce/.github/workflows/ci.yaml#L93-L107
• Build URL (if public): https://github.com/neuro-inc/neuro-base-environment/runs/3984118340?check_suite_focus=true

Logs

logs_549.zip

[crazy-max]

@YevheniiSemendiak Looking at the size of your image, I think you hit #321. Can you remove pre-installed components as suggested in #321 (comment) and see if it's ok?

[YevheniiSemendiak]

@crazy-max thanks a lot for the hint!
It did work for me (at least the first run after adding a work-around).

Sometimes I'm really blind and cannot find the info (needed issue, you mentioned) on the surface.

[jacksongoode]

@crazy-max I know this is a closed issue but it was similar enough to my issue that I wanted to ask you - how would you access an image build with cache-from: type=gha across subsequent jobs? As in:

jobs:
  build:
    ...
    steps:
      ...
      - name: Build and push
         uses: docker/build-push-action@v3
         with:
           context: .
           push: true
           tags: ${{ env.REPO }}:${{ env.COMMIT }}
           cache-from: type=gha
           cache-to: type=gha,mode=max
      - name: Likely this would not be able to find it
         run: docker image inspect ${{ env.REPO }}:${{ env.COMMIT }}
  load:
    ...
    steps:
      ...
      - name: Load image
         uses: docker/build-push-action@v3
         with:
           context: .
           load: true
           tags: ${{ inputs.REPO }}:${{ inputs.COMMIT }}
           cache-from: type=gha
      - name: But this step should, no?
         run: docker image inspect ${{ env.REPO }}:${{ env.COMMIT }}

I would expect both inspect statements to find the image with the right flags (?). But I really only would need the second job being able to retrieve the image from the cache into the local registry. I would imagine this "Load image" step would be very fast, but maybe I am misinterpreting the flexibility of the gha cache option.

I should also say I've looked through all of the examples and would not want to use the upload/download artifact method as it's quite slow and the images don't need to be exposed since they are pushed to a remote registry.