Multi stages build cache work only with last stage

[kwladyka]

Dockerfile

FROM clojure:lein-alpine as dependencies
WORKDIR /app
COPY project.clj .
RUN lein deps

FROM dependencies
WORKDIR /app
RUN lein test

FROM dependencies as builder
WORKDIR /app
RUN lein uberjar

FROM anapsix/alpine-java
EXPOSE 8080
WORKDIR /app
COPY --from=builder /app/target/*-standalone.jar ./app.jar
CMD ["java", "-cp", "/app/app.jar", "clojure.main", "-m", "etingroup.core"]

It looks like only this part (last step) is cached:

FROM anapsix/alpine-java
EXPOSE 8080
WORKDIR /app
COPY --from=builder /app/target/*-standalone.jar ./app.jar
CMD ["java", "-cp", "/app/app.jar", "clojure.main", "-m", "etingroup.core"]

Is it possible to improve?

[chrishiestand]

This should now be possible with cache_from. The complication is that you have to save each intermediate image (everything up until the new FROM) as its own image, and include that image in the cache_from array. This is a docker limitation afaik.

[zivori]

I see the base image cached locally as < none > : < none >
but how can I tag it so it can be pushed to a remote registery (so it can be used afterwards for caching)

REPOSITORY TAG IMAGE ID CREATED SIZE

<none> <none> 7dbdf83dd6a0 8 seconds ago 690MB

[kwladyka]

BTW There is connected topic to this issue
docker/cli#666

[chrishiestand]

@zivori something like this should work:

docker tag 7dbdf83dd6a0 my-intermediate-image:latest
docker push my-intermediate-image:latest

[dhinus]

This is a full example of how I am doing multi-stage builds using cache_from, hope it helps. Please note I am also using load_bases to make sure the base image (node:8-alpine in my case) is cached.

resources:
  - name: node
    type: docker-image
    source:    
      repository: node
      tag: "8-alpine"

  - name: docker-app
    type: docker-image
    source:
      repository: my/app
  
  - name: docker-app-stage1
    type: docker-image
    source:
      repository: my/app
      tag: stage1-latest
    
jobs:
- name: build
  plan:
    - get: docker-app
      params:
        save: true
    - get: docker-app-stage1
      params:
        save: true

    - put: docker-app-stage1
      params:
        build: app
        target_name: stage1
        load_bases:
          - node
        cache_from:
          - docker-app-stage1
      get_params:
        save: true

    - put: docker-app
      params:
        build: app
        load_bases:
          - node
        cache_from:
          - docker-app
          - docker-app-stage1
      get_params:
        skip_download: true

[evelant]

I am trying to use the technique laid out here by @dhinus but it appears my stage1 image isn't getting saved for some reason. When I get to the app build stage I get open app-image-stage1/image: no such file or directory as if the stage1 image were never saved.
#237

[dhinus]

@AndrewMorsillo that's weird, would you mind posting the full output of the job? Does the node image load correctly? You should see lines like this one at the beginning of your app-image-stage1 job:

Loaded image ID: sha256:df48..........

You can also use fly intercept to inspect the filesystem where the image should be saved.

[xtremerui]

Hi, is @dhinu 's solution works for your case @kwladyka ?

I don't see activities on docker/cli#666 either so just wondering.

[kwladyka]

To be honest I don't remember. But as I remember I didn't find anything what satisfied me. I was loosing too much time to maintenance concurse. After all I moved to github actions.

But you can always make additional image in the middle. So download all dependencies etc. and make a image:deps and use it later to build final app. Files which generate dependencies don't change too often. Not exactly what I was asking here, but some kind of solution.

[xtremerui]

@kwladyka thx for the response. Since the solution is presented and we are not going to add more features to this resource I will close this issue now.

Bon voyage with Github actions!