Docker BuildKit with --mount=type=cache and mounted yarn cache seemingly empty always? #1283

samesfahani-tuplehealth · 2022-08-18T17:03:26Z

samesfahani-tuplehealth
Aug 18, 2022

I am trying to speed up my yarn install types when a package dependency changes. I've tried to repro the issue down to its core.

package.json (random packages, sample)

{
  "name": "docker-cache",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "colors": "^1.4.0",
    "deep-equal": "2.0.3",
    "dotenv": "8.2.0",
    "eslint": "^7.14.0",
    "eslint-config-prettier": "^6.15.0",
    "eslint-config-react-app": "^6.0.0",
    "eslint-loader": "^4.0.2",
    "eslint-plugin-filenames": "^1.3.2",
    "eslint-plugin-flowtype": "^5.2.0",
    "eslint-plugin-import": "^2.22.1",
    "eslint-plugin-jsx-a11y": "^6.4.1",
    "eslint-plugin-react": "^7.21.5",
    "eslint-plugin-react-hooks": "^4.2.0",
    "exceljs": "^4.2.1",
    "fast-check": "^1.24.2",
    "file-loader": "6.0.0",
    "find": "0.3.0",
    "fs-extra": "9.0.1",
    "html-webpack-plugin": "4.3.0",
    "jest": "26.2.2",
    "jest-canvas-mock": "2.2.0",
    "jest-pnp-resolver": "^1.2.1",
    "json-diff": "0.5.4",
    "lerna": "^3.22.1",
    "madge": "^5.0.1",
    "mini-css-extract-plugin": "0.9.0",
    "mobx-logger": "^0.7.1",
    "mutationobserver-shim": "^0.3.7",
    "node-fetch": "^2.2.1",
    "nodemon": "^2.0.4",
    "object-hash": "2.0.3",
    "optimize-css-assets-webpack-plugin": "5.0.3",
    "pdfjs-dist": "^2.5.207",
    "pdfreader": "^1.2.6",
    "pnp-webpack-plugin": "^1.6.4",
    "postcss": "^8.3.11",
    "postcss-flexbugs-fixes": "^5.0.2",
    "postcss-loader": "^4.0.0",
    "postcss-normalize": "^10.0.1",
    "postcss-preset-env": "^6.7.0",
    "postcss-safe-parser": "^6.0.0",
    "prettier": "2.0.5",
    "q": "^1.5.1",
    "react-dev-utils": "^11.0.4",
    "react-error-overlay": "6.0.7",
    "react-test-renderer": "16.12.0",
    "rimraf": "^3.0.2"
  }
}

Dockerfile

FROM node:16.13.0
WORKDIR /tmp/
COPY package.json yarn.lock ./

ENV YARN_CACHE_FOLDER=/root/.yarn
RUN yarn config set cache-folder $YARN_CACHE_FOLDER # just to be explicit
RUN --mount=type=cache,mode=0777,target=$YARN_CACHE_FOLDER yarn cache list
RUN --mount=type=cache,mode=0777,target=$YARN_CACHE_FOLDER YARN_CACHE_FOLDER=$YARN_CACHE_FOLDER yarn install --network-timeout=600000 --frozen-lockfile

CMD ["echo ", "Done"]

Now I run docker buildx build --progress=plain --no-cache -t attempt-1 .

Output of first pass

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 32B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/node:16.13.0
#3 DONE 0.2s

#4 [stage-0 1/6] FROM docker.io/library/node:16.13.0@sha256:580a0850049c59a48f06090edd48c9f966c5e6572bbbabc369ba3ecbc4855dba
#4 DONE 0.0s

#5 [internal] settings cache mount permissions
#5 CACHED

#6 [stage-0 2/6] WORKDIR /tmp/
#6 CACHED

#7 [internal] load build context
#7 transferring context: 509.56kB 0.0s done
#7 DONE 0.0s

#8 [stage-0 3/6] COPY package.json yarn.lock ./
#8 DONE 0.1s

#9 [stage-0 4/6] RUN yarn config set cache-folder /root/.yarn
#9 0.598 yarn config v1.22.15
#9 0.620 success Set "cache-folder" to "/root/.yarn".
#9 0.620 Done in 0.03s.
#9 DONE 0.8s

#10 [stage-0 5/6] RUN --mount=type=cache,mode=0777,target=/root/.yarn yarn cache list
#10 0.370 yarn cache v1.22.15
#10 0.403 Name Version Registry Resolved
#10 0.403 Done in 0.04s.
#10 DONE 0.7s

#11 [stage-0 6/6] RUN --mount=type=cache,mode=0777,target=/root/.yarn YARN_CACHE_FOLDER=/root/.yarn yarn install --network-timeout=600000 --frozen-lockfile
#11 0.579 yarn install v1.22.15
#11 0.673 [1/4] Resolving packages...
#11 1.081 [2/4] Fetching packages...
#11 28.26 info fsevents@2.3.2: The platform "linux" is incompatible with this module.
#11 28.26 info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
#11 28.28 [3/4] Linking dependencies...
#11 28.28 warning " > eslint-config-react-app@6.0.0" has unmet peer dependency "@typescript-eslint/eslint-plugin@^4.0.0".
#11 28.28 warning " > eslint-config-react-app@6.0.0" has unmet peer dependency "@typescript-eslint/parser@^4.0.0".
#11 28.28 warning " > eslint-config-react-app@6.0.0" has unmet peer dependency "babel-eslint@^10.0.0".
#11 28.28 warning " > eslint-loader@4.0.2" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
#11 28.29 warning " > file-loader@6.0.0" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
#11 28.29 warning " > html-webpack-plugin@4.3.0" has unmet peer dependency "webpack@>=4.0.0 < 6.0.0".
#11 28.29 warning "lerna > @lerna/version > @lerna/github-client > @octokit/rest > @octokit/plugin-request-log@1.0.4" has unmet peer dependency "@octokit/core@>=3".
#11 28.29 warning " > mini-css-extract-plugin@0.9.0" has unmet peer dependency "webpack@^4.4.0".
#11 28.29 warning " > mobx-logger@0.7.1" has unmet peer dependency "mobx@>= 4.0.0".
#11 28.29 warning " > optimize-css-assets-webpack-plugin@5.0.3" has unmet peer dependency "webpack@^4.0.0".
#11 28.29 warning " > postcss-loader@4.3.0" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
#11 28.29 warning " > postcss-normalize@10.0.1" has unmet peer dependency "browserslist@>= 4".
#11 28.29 warning "postcss-normalize > postcss-browser-comments@4.0.0" has unmet peer dependency "browserslist@>=4".
#11 28.29 warning " > react-test-renderer@16.12.0" has unmet peer dependency "react@^16.0.0".
#11 44.69 [4/4] Building fresh packages...
#11 45.52 Done in 44.95s.
#11 DONE 46.0s

#12 exporting to image
#12 exporting layers
#12 exporting layers 10.5s done
#12 writing image sha256:5866fab165e759bd4c57c48f85856d98dc2bfa778e1f549a39b36289c8749bbb done
#12 naming to docker.io/library/attempt-1 done
#12 DONE 10.5s

Next I removed the colors package, ran yarn (to update yarn.lock) and re-ran the docker buildx.

Output of second pass

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 32B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/node:16.13.0
#3 DONE 0.5s

#4 [stage-0 1/6] FROM docker.io/library/node:16.13.0@sha256:580a0850049c59a48f06090edd48c9f966c5e6572bbbabc369ba3ecbc4855dba
#4 DONE 0.0s

#5 [internal] settings cache mount permissions
#5 CACHED

#6 [stage-0 2/6] WORKDIR /tmp/
#6 CACHED

#7 [internal] load build context
#7 transferring context: 511.02kB 0.0s done
#7 DONE 0.0s

#8 [stage-0 3/6] COPY package.json yarn.lock ./
#8 DONE 0.0s

#9 [stage-0 4/6] RUN yarn config set cache-folder /root/.yarn
#9 0.341 yarn config v1.22.15
#9 0.372 success Set "cache-folder" to "/root/.yarn".
#9 0.372 Done in 0.04s.
#9 DONE 0.4s

#10 [stage-0 5/6] RUN --mount=type=cache,mode=0777,target=/root/.yarn yarn cache list
#10 0.338 yarn cache v1.22.15
#10 0.368 Name Version Registry Resolved
#10 0.369 Done in 0.03s.
#10 DONE 0.4s

#11 [stage-0 6/6] RUN --mount=type=cache,mode=0777,target=/root/.yarn YARN_CACHE_FOLDER=/root/.yarn yarn install --network-timeout=600000 --frozen-lockfile
#11 0.473 yarn install v1.22.15
#11 0.544 [1/4] Resolving packages...
#11 0.949 [2/4] Fetching packages...
#11 27.43 info fsevents@2.3.2: The platform "linux" is incompatible with this module.
#11 27.43 info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
#11 27.45 [3/4] Linking dependencies...
#11 27.46 warning " > eslint-config-react-app@6.0.0" has unmet peer dependency "@typescript-eslint/eslint-plugin@^4.0.0".
#11 27.46 warning " > eslint-config-react-app@6.0.0" has unmet peer dependency "@typescript-eslint/parser@^4.0.0".
#11 27.46 warning " > eslint-config-react-app@6.0.0" has unmet peer dependency "babel-eslint@^10.0.0".
#11 27.46 warning " > eslint-loader@4.0.2" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
#11 27.46 warning " > file-loader@6.0.0" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
#11 27.46 warning " > html-webpack-plugin@4.3.0" has unmet peer dependency "webpack@>=4.0.0 < 6.0.0".
#11 27.46 warning "lerna > @lerna/version > @lerna/github-client > @octokit/rest > @octokit/plugin-request-log@1.0.4" has unmet peer dependency "@octokit/core@>=3".
#11 27.46 warning " > mini-css-extract-plugin@0.9.0" has unmet peer dependency "webpack@^4.4.0".
#11 27.46 warning " > mobx-logger@0.7.1" has unmet peer dependency "mobx@>= 4.0.0".
#11 27.46 warning " > optimize-css-assets-webpack-plugin@5.0.3" has unmet peer dependency "webpack@^4.0.0".
#11 27.46 warning " > postcss-loader@4.3.0" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
#11 27.46 warning " > postcss-normalize@10.0.1" has unmet peer dependency "browserslist@>= 4".
#11 27.46 warning "postcss-normalize > postcss-browser-comments@4.0.0" has unmet peer dependency "browserslist@>=4".
#11 27.46 warning " > react-test-renderer@16.12.0" has unmet peer dependency "react@^16.0.0".
#11 40.10 [4/4] Building fresh packages...
#11 40.82 Done in 40.35s.
#11 DONE 41.1s

#12 exporting to image
#12 exporting layers
#12 exporting layers 11.8s done
#12 writing image sha256:668320ce0930ff1174cdc14ec5cdc35f4de94e328b04a745ba41ec48643b074c done
#12 naming to docker.io/library/attempt-1 done
#12 DONE 11.8s

As you can see, the second pass look pretty identical to the first. The result of yarn cache list is still empty and it took roughly ~40-46s to complete, indicating that no caching was done.

I am on macOS.

Answered by samesfahani-tuplehealth

Aug 29, 2022

It seems like the cache was empty because the key wasn't properly evaluating $YARN_CACHE_FOLDER. Instead best to just hardcode the paths and not from an ENV variable.

View full answer

samesfahani-tuplehealth · 2022-08-18T21:10:58Z

samesfahani-tuplehealth
Aug 18, 2022
Author

I even tried adding explicit ID, sharing, and mode. Here is an output from a run:

#9 [stage-0 4/4] RUN --mount=type=cache,id=yarn,target=/yarn/.cache,sharing=shared,mode=0777 YARN_CACHE_FOLDER=/yarn/.cache yarn install
#9 0.339 yarn install v1.22.15
#9 0.396 [1/4] Resolving packages...
#9 0.592 [2/4] Fetching packages...
#9 11.64 [3/4] Linking dependencies...
#9 11.64 warning " > optimize-css-assets-webpack-plugin@5.0.3" has unmet peer dependency "webpack@^4.0.0".
#9 11.64 warning " > postcss-loader@4.3.0" has unmet peer dependency "webpack@^4.0.0 || ^5.0.0".
#9 11.64 warning " > postcss-normalize@10.0.1" has unmet peer dependency "browserslist@>= 4".
#9 11.64 warning "postcss-normalize > postcss-browser-comments@4.0.0" has unmet peer dependency "browserslist@>=4".
#9 11.64 warning " > react-test-renderer@16.12.0" has unmet peer dependency "react@^16.0.0".
#9 16.48 [4/4] Building fresh packages...
#9 16.50 Done in 16.17s.
#9 DONE 16.8s

#10 exporting to image
#10 exporting layers
#10 exporting layers 3.0s done
#10 writing image sha256:e5d1cb7a3b4bbae527d3d3b36c82490ec9a53d251d44f696ccdc8e910cd713a9 done
#10 naming to docker.io/library/attempt-1 done
#10 DONE 3.0s

Output of docker buildx du --verbose | grep -B 9 exec.cachemount:

ID:             q0o4n6xtdr63syhrr0xx5fazb
Created at:     2022-08-18 21:08:31.414073768 +0000 UTC
Mutable:        true
Reclaimable:    true
Shared:         false
Size:           0B
Description:    cached mount /$YARN_CACHE_FOLDER from exec /bin/sh -c YARN_CACHE_FOLDER=$YARN_CACHE_FOLDER yarn install
Usage count:    1
Last used:      7 seconds ago
Type:           exec.cachemount

A few questions:

Why is the size 0B?
Why is shared still false even when the default should be sharing and I've explicitly set to shared?

0 replies

samesfahani-tuplehealth · 2022-08-29T19:14:13Z

samesfahani-tuplehealth
Aug 29, 2022
Author

It seems like the cache was empty because the key wasn't properly evaluating $YARN_CACHE_FOLDER. Instead best to just hardcode the paths and not from an ENV variable.

0 replies

blole · 2022-12-27T19:08:02Z

blole
Dec 27, 2022

I encountered the same problem, and adding a

# syntax = docker/dockerfile:1.4

to the top of my Dockerfile made the RUN --mount=type=cache,target=$YARN_CACHE_FOLDER syntax work

0 replies

armenr · 2023-02-06T06:29:01Z

armenr
Feb 6, 2023

A follow-up question here:

If we use the S3 cache backend (which is awesome, btw!!!!!), do these buildx cache-mounts get cached and uploaded to S3 as well? (see code below for an example of the build command)

The reason I ask is because we are stuck using BitBucket pipelines...and there are stupid limits and requirements there, including having to use docker dind on self-hosted runners, and I think that the docker buildx instance gets destroyed/created on each run of the pipeline --> Which means that we don't get the benefit of the cache-mounts, since they get recreated on each run.

Example docker buildx command:

  docker buildx build "." \
    --pull \
    --progress plain \
    --platform linux/amd64 \
    --cache-to type=s3,region=us-west-2,bucket="${S3_BUILDKIT_CACHE}",mode=max,name="${image_name}",access_key_id="${s3_cache_upload_aws_key_id}",secret_access_key="${s3_cache_upload_aws_secret_access_key}" \
    --cache-from type=s3,region=us-west-2,bucket="${S3_BUILDKIT_CACHE}",name="${image_name}",access_key_id="${s3_cache_upload_aws_key_id}",secret_access_key="${s3_cache_upload_aws_secret_access_key}" \

2 replies

polarathene Mar 28, 2023

AFAIK --cache-to and --cache-from is only regarding image build cache, not the cache mounts.

So typically on CI platforms this cache mount in RUN won't benefit you if you're not re-using within the same build. It's helpful for local build iterations though.

Some users have worked around that with a separate cache image that uses RUN cp to transfer the cache mount data into the build cache as part of a RUN layer.

For some usages like package installs, you might not gain much benefit doing so if you already use multi-stage builds. And for caching much more CPU intensive compilation, you might be better served with a build image that does a docker run to store the cache in a bind mount volume, and upload/download that in the CI as a separate cache (agnostic to Docker).

strophy Jun 2, 2023

Correct, --cache-from and --cache-to only applies to the caching of each layer of the docker image, it does not include cache mounts.

There's good progress and a few examples on how to copy and populate cache mounts, including from S3, over here: moby/buildkit#1512 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker BuildKit with --mount=type=cache and mounted yarn cache seemingly empty always? #1283

{{title}}

Replies: 4 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Docker BuildKit with --mount=type=cache and mounted yarn cache seemingly empty always? #1283

samesfahani-tuplehealth Aug 18, 2022

Replies: 4 comments · 2 replies

samesfahani-tuplehealth Aug 18, 2022 Author

samesfahani-tuplehealth Aug 29, 2022 Author

blole Dec 27, 2022

armenr Feb 6, 2023

polarathene Mar 28, 2023

strophy Jun 2, 2023

samesfahani-tuplehealth
Aug 18, 2022

Replies: 4 comments 2 replies

samesfahani-tuplehealth
Aug 18, 2022
Author

samesfahani-tuplehealth
Aug 29, 2022
Author

blole
Dec 27, 2022

armenr
Feb 6, 2023