Shouldn't default-repo be applied to cacheFrom directives?

[lucasmpr]

I'm trying very hard to optimize my DotNet multi-stage build process.

To do that I'm creating three profiles that I must execute in sequence.

Profile 1 - Base images
Profile 2 - Dependency Builder
Profile 3 - Final builder

The idea is to use caching and don't rebuild the whole system every time, and build in parallel things that are not dependent on others.

For each step, I want to use a previous image in cache or the image build in the latest profile. So was trying to use cacheFrom when I stumbled into that.

I use the --default-repo to set the image according to the branch, in runtime, and I expected the cache image to be pulled from my default repo, but It doesn't seem to be.

Additional

• I would like to be able to use templating in the cacheFrom directive, so I can use the previous image and not necessarly the one tagged as latest.

Expected behavior

If default-repo is specified, cacheFrom images without full path should be rewriten if not fully qualified.

Actual behavior

Skaffold says the image could not be found.

Information

• Skaffold version: 1.2.0
• Operating system: Windows
• Contents of skaffold.yaml:

profile:
- name: restore-stage
  build:
    local: 
      useBuildkit: false
    artifacts:
    - image: restore
      context: .
      docker:
        dockerfile: Dockerfile.default
        cacheFrom:
        - restore:latest
        target: pre_publish

[tejal29]

@lucasmpr, thank you filing this issue.

IIUC, restore:latest is one of the images built by skaffold in another profile?
You used the --default-repo flag when building restore:latest in previous skaffold build command.
e.g.
skaffold build --default-repo=gcr.io/test -p build-restore
The built image was gcr.io/test/restore:latest

Now you want to use the built image as cacheFrom value?
Is that correct?

That would mean, skaffold have to see if images in docker.cacheFrom list is one of the build artifacts. Currently skaffold does not evaluate all possible artifacts (based on non active profiles).
Is it possible for you to share share entire your skaffold.yaml or other profiles?

[lucasmpr]

Hello tejal, let me explain a little bit more what I'm trying to do.

a) I have one big repo with multiple services. I have a global dockerfile that does the following:

1. Builds the base image (base)
2. Builds the remote dependencies image (restore) (it is the same for all images)
3. Copy the local dependencies, if any of them have changed (dependency_builder)
4. Builds the project

What I'm doing is a workaround for the "Dependent images" from #889
I want to build the base image without concurrency and fetch the remote dependencies after.

Then I want to build all the projects in parallel.

So the idea is the following. I'm running skaffold 3 times.

1. Calls skaffold to Build base:latest image and restore:latest image and also use base:latest and restore:latest as cache.
   (If the image is the same, nothing is built and the image doesnt' change)
2. Build the cache stage, for each project build an image called ${project}-cache:latest with the ${project}-cache:latest (so if one of the projects is cached and didn't changed, nothing is done)
3. Builds the final stage and pushs it

piVersion: skaffold/v2alpha2
kind: Config
build:
  local: 
    useBuildkit: false
    concurrency:  0
  artifacts:
  - image: project1
    context: .
    docker:
      dockerfile: Dockerfile.default
      buildArgs:
        PROJECT: Project1
      cacheFrom:
      - default-repo/base:latest
      - default-repo/restore:latest
      - default-repo/project1-cache:latest
  - image: Project2
    context: .
    docker:
      dockerfile: Dockerfile.default
      buildArgs:
        PROJECT: Project2
      cacheFrom:
      - default-repo/base:latest
      - default-repo/restore:latest
      - default-repo/project2-cache:latest
  tagPolicy:
    gitCommit:
      variant: AbbrevCommitSha    
profiles:
- name: restore-stage
  build:
    local: 
      useBuildkit: false
      concurrency: 0
    artifacts:
    - image: restore
      context: .
      docker:
        dockerfile: Dockerfile.default
        cacheFrom:
        - default-repo/restore:latest
        target: pre_publish
        buildArgs:
          BUILDKIT_INLINE_CACHE: 1
    - image: base
      context: .
      docker:
        dockerfile: Dockerfile.default
        cacheFrom:
        - default-repo/base:latest
        target: base
        buildArgs:
          BUILDKIT_INLINE_CACHE: 1

- name: cache-stage
  build:
    local: 
      useBuildkit: false
      concurrency:  0
    artifacts:
    - image: project1-cache
      context: .
      docker:
        dockerfile: Dockerfile.default
        target: dependency_builder
        buildArgs:
          PROJECT: 'Project1'
        cacheFrom:
        - default-repo/base:latest
        - default-repo/restore:latest
        - default-repo/web-cache:latest
    - image: project2-cache
      context: .
      docker:
        dockerfile: Dockerfile.default
        target: dependency_builder
        buildArgs:
          PROJECT: 'Project2'
        cacheFrom:
        - default-repo/base:latest
        - default-repo/restore:latest
        - default-repo/api-cache:latest

I'm running it on google cloud with the following command:

skaffold build -p restore-stage --default-repo=gcr.io/$PROJECT_ID/$BRANCH_NAME -t latest --cache-artifacts=false  && skaffold build -p cache-stage --default-repo=gcr.io/$PROJECT_ID/$BRANCH_NAME -t latest --cache-artifacts=false && skaffold build -f=skaffold.yaml --default-repo=gcr.io/$PROJECT_ID/$BRANCH_NAME;'

What I expected was that if I dont type default-repo in the skaffold.yaml the cache from gets rewritten with my gcr.io/$PROJECT_ID/$BRANCH_NAME so my master branch always get the cache from my latest master branch, and when I'm building in another branch, it uses a previous branch.

[dgageot]

I think this makes sense. I'm going to bump the priority a little

[nkubala]

bumping the priority again, I think we should try and get to this in the next few milestones. if anyone is interested in helping out here, it would be appreciated :)

[briandealwis]

I think default-repo should only be applied to the cacheFrom if the image repository matches that of a defined artifact. After all, there may be a reason to have the cache be pulled from some other repository.

[tejal29]

This is nearing SLO and was wondering if anyone one is interested in taking this.
@GoogleContainerTools/skaffold-team

[fredleger]

We also need this one. Let me describe the use case here.
We are building a front and backend nodejs app in a single repo style with lerna.
So it make sense to define 3 docker images in a multi-stage way:

• base image (where all the yarn install is done)
• front (which basically copy from base and build static pages)
• back (which also copy from base and build dynamic pages)

So it make perfectly sense to build base and have the 2 others cacheFrom the base (and node image as well). For now it's failing because of this issue.

Another way to do this could also tu support variables expansion in skaffold.yaml which could have numerous other use cases but if skaffold checks for cacheFrom image in default repo before checking docker hub this is also fine to me.

[earlybard]

Any updates on this?

I've got a use-case where we use the same skaffold file in different environments with their own registry. We can't realistically use caching without the default repo passing through.

(As above... variable expansion would solve it too and my last resort is pre-processing my skaffold.yaml - please no!)

[ericzzzzzzz]

bumping this up