False cache hits in monorepo projects, leading to use file from different docker build

[mikipoldi]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Hi,
we have have monorepo with multiple spring boot apps. We built them using gradle and spring boot plugin. And we also made some settings to have reproducible jars which helps us make reproducible docker images (same sha for multiple different build times)

And here is the problem. Both apps had same length of starting boot class name (AppYApplication, AppXApplication).

Which in combination with same modification time (reproducible build), same name, same relative path and completely same size of the file MANIFEST.MF lead to false positive cache hit in docker buildx build. MANIFEST.MF from first app was used in the second one, which lead to app not starting.

Reproducible example here:
https://github.com/mikipoldi/docker-false-cache-hit

Expected behaviour

Both app docker containers run smoothly.

Actual behaviour

MANIFEST.MF from appx is also used in appy which leads to:

Exception in thread "main" java.lang.ClassNotFoundException: com.foo.bar.dockercachemixinissue.AppXApplication
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.loadClass(JarUrlClassLoader.java:97)
at org.springframework.boot.loader.launch.LaunchedClassLoader.loadClass(LaunchedClassLoader.java:91)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:534)
at java.base/java.lang.Class.forName(Class.java:513)
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:99)
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64)
at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40)

Buildx version

github.com/docker/buildx v0.20.1-desktop.2 aaf7c2bc7f9ec3afee1cec77d671845a4b57a0c8 (but same issues on newer on jenkins which i could provide later)

Docker info

Client:
 Version:    27.5.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.7.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-ai.exe
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.1-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.38
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.16.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 89
  Running: 20
  Paused: 0
  Stopped: 69
 Images: 133
 Server Version: 27.5.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.1.12-0-g51d5e946
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.87.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 3.822GiB
 Name: docker-desktop
 ID: 10d3b887-9092-42c6-9478-b89041a22c23
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Builders list

docker buildx ls
NAME/NODE             DRIVER/ENDPOINT     STATUS    BUILDKIT   PLATFORMS
multi-platform*       docker-container
 \_ multi-platform0    \_ desktop-linux   running   v0.17.1    linux/amd64 (+3), linux/arm64, linux/arm (+2), linux/ppc64le, (3 more)
default               docker
 \_ default            \_ default         running   v0.18.2    linux/amd64 (+3), linux/arm64, linux/arm (+2), linux/ppc64le, (3 more)
desktop-linux         docker
 \_ desktop-linux      \_ desktop-linux   running   v0.18.2    linux/amd64 (+3), linux/arm64, linux/arm (+2), linux/ppc64le, (3 more)

Configuration

I pushed reproducible example here (we simulate the preserveTimestamps from jar packaging with setting timestamp via linux command)

https://github.com/mikipoldi/docker-false-cache-hit

Build logs

docker buildx build --progress plain --pull -f appy/docker/Dockerfile -t host.docker.internal:5000/appy appy/docker/ --builder=multi-platform --platform=linux/amd64,linux/arm64 --provenance=false --push
#0 building with "multi-platform" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 378B 0.0s done
#1 DONE 0.1s

#2 [linux/amd64 internal] load metadata for host.docker.internal:5000/basejava:latest
#2 DONE 0.0s

#3 [linux/arm64 internal] load metadata for host.docker.internal:5000/basejava:latest
#3 DONE 0.1s

#4 [internal] load .dockerignore
#4 transferring context: 2B 0.0s done
#4 DONE 0.1s

#5 [linux/arm64 1/6] FROM host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af
#5 resolve host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af
#5 resolve host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af 0.1s done
#5 CACHED

#6 [linux/arm64 2/6] WORKDIR /opt/app
#6 DONE 0.1s

#7 [linux/amd64 1/6] FROM host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af
#7 resolve host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af 0.1s done
#7 DONE 0.2s

#8 [linux/amd64 2/6] WORKDIR /opt/app
#8 DONE 0.1s

#9 [internal] load build context
#9 transferring context: 11.28MB 0.7s done
#9 DONE 0.7s

#10 [linux/arm64 3/6] COPY layers/dependencies/ ./
#10 ...

#11 [linux/amd64 3/6] COPY layers/dependencies/ ./
#11 DONE 0.2s

#10 [linux/arm64 3/6] COPY layers/dependencies/ ./
#10 DONE 0.2s

#12 [linux/amd64 4/6] COPY layers/spring-boot-loader/ ./
#12 DONE 0.2s

#13 [linux/arm64 4/6] COPY layers/spring-boot-loader/ ./
#13 DONE 0.2s

#14 [linux/amd64 5/6] COPY layers/snapshot-dependencies/ ./
#14 DONE 0.2s

#15 [linux/arm64 5/6] COPY layers/snapshot-dependencies/ ./
#15 DONE 0.2s

#16 [linux/amd64 6/6] COPY layers/application/ ./
#16 DONE 0.1s

#17 [linux/arm64 6/6] COPY layers/application/ ./
#17 DONE 0.1s

#18 exporting to image
#18 exporting layers
#18 exporting layers 1.0s done
#18 exporting manifest sha256:0e67b90109f0a97d18416161505d95cbd6425d5f586880ea9de425211d2b9676 0.0s done
#18 exporting config sha256:b0e20830c85452c8450042845b4a1352ac17ece631deec3e73ddbf16946b75e9 0.0s done
#18 exporting manifest sha256:66678e1eca2702a1c9b96873f171881117a8d8a7482be62d7614253321402506 0.0s done
#18 exporting config sha256:c1134e817df6dedd3249115a08bd72e2c5bc0dfa527d237f171ab50691770d6a 0.0s done
#18 exporting manifest list sha256:11d5d2d1f5ff59a97d143b19a6d38768aa11009abc37079ae3118c3388a2de20 0.0s done
#18 pushing layers
#18 pushing layers 0.2s done
#18 pushing manifest for host.docker.internal:5000/appx:latest@sha256:11d5d2d1f5ff59a97d143b19a6d38768aa11009abc37079ae3118c3388a2de20
#18 pushing manifest for host.docker.internal:5000/appx:latest@sha256:11d5d2d1f5ff59a97d143b19a6d38768aa11009abc37079ae3118c3388a2de20 0.1s done
#18 DONE 1.4s
#0 building with "multi-platform" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 378B 0.0s done
#1 DONE 0.0s

#2 [linux/amd64 internal] load metadata for host.docker.internal:5000/basejava:latest
#2 DONE 0.0s

#3 [linux/arm64 internal] load metadata for host.docker.internal:5000/basejava:latest
#3 DONE 0.1s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [linux/amd64 1/6] FROM host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af
#5 resolve host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af 0.0s done
#5 DONE 0.1s

#6 [linux/arm64 1/6] FROM host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af
#6 resolve host.docker.internal:5000/basejava:latest@sha256:4b03603319968355ca13dd952e227be8feff9edddbbb7cbf57eaef45d4cb78af 0.0s done
#6 DONE 0.1s

#7 [internal] load build context
#7 transferring context: 14.40kB 1.2s done
#7 DONE 1.2s

#8 [linux/amd64 2/6] WORKDIR /opt/app
#8 CACHED

#9 [linux/amd64 3/6] COPY layers/dependencies/ ./
#9 CACHED

#10 [linux/amd64 4/6] COPY layers/spring-boot-loader/ ./
#10 CACHED

#11 [linux/arm64 3/6] COPY layers/dependencies/ ./
#11 CACHED

#12 [linux/arm64 2/6] WORKDIR /opt/app
#12 CACHED

#13 [linux/arm64 4/6] COPY layers/spring-boot-loader/ ./
#13 CACHED

#14 [linux/amd64 5/6] COPY layers/snapshot-dependencies/ ./
#14 CACHED

#15 [linux/arm64 5/6] COPY layers/snapshot-dependencies/ ./
#15 CACHED

#16 [linux/amd64 6/6] COPY layers/application/ ./
#16 DONE 0.1s

#17 [linux/arm64 6/6] COPY layers/application/ ./
#17 DONE 0.1s

#18 exporting to image
#18 exporting layers
#18 exporting layers 0.2s done
#18 exporting manifest sha256:9277488ce97a806938b1e91ddb1a0bc87826165e8e8d198c9e8345c44f815c13 0.0s done
#18 exporting config sha256:3a794fcb31efceaf18a65b869375cc1c126cb543507d615d1d435330bf12a40c 0.0s done
#18 exporting manifest sha256:42606bec33db95bff47018ded70e76bfeb6306e976d48201f4c34290c0c0bb38 0.0s done
#18 exporting config sha256:a2247a88c1dd4ea513d5dc0ac0a2947b057429955b7521352bd8fb87c86f7f5e 0.0s done
#18 exporting manifest list sha256:39de92b1ef981cc268ad8202952ae6be7cb94fb99bb77b8374f9f6286bd56c61 0.0s done
#18 pushing layers 0.1s done
#18 pushing manifest for host.docker.internal:5000/appy:latest@sha256:39de92b1ef981cc268ad8202952ae6be7cb94fb99bb77b8374f9f6286bd56c61
#18 pushing manifest for host.docker.internal:5000/appy:latest@sha256:39de92b1ef981cc268ad8202952ae6be7cb94fb99bb77b8374f9f6286bd56c61 0.1s done
#18 DONE 0.5s

Additional info

No response

[tonistiigi]

This is expected when manually resetting the file creation times as in your reproducer. File syncing uses the same metadata-based comparison as rsync by default.

We do sync different projects into different locations so that they don't overwrite each other, which would cause inefficient transfers. This is based on the node identifier and the basename of the directory (that we imply as "project name"). I think the reason your repro works is also because you seem to use exactly the same name for the directory in both cases.

We could use longer identifier than basename, but that would mean that changing workdir would reset the incremental file transfers.

[mikipoldi]

ok thanks for clarification, just few extra questions pls:

1. the basename for boths is "docker" when i set build context to appy/docker?
2. if i used build context to appy and appx (instead of appy/docker and appx/docker) this cache hit would not happen?
3. i found something about metadata and hashing them in docs, but i didnt found anything about the "key" for the build cache which is the last folder in path, is it somewhere documented?
4. why am i able to reproduce this with "--no-cache". is some cache still used even if i use this option?

thanks!

[mikipoldi]

also i still think that this could be issue when we have only one project, but one file is somehow changed but remains with the same size (and mtime reseted). this would lead to usage of "old" file when ran on same machine before and after that change.

[tonistiigi]

1. yes
2. I think so. Report back if you see different.
3. this isn't necessarily a documented guarantee. The transfer is based on metadata. Just projects with different basename are transferred to different location. This isn't for correctness of the transfer, but so that if you are switching between projects, they don't overwrite each others files and therefore make it so that same files need to be transferred repeatedly.
4. This doesn't have anything to do with caching layer (if the RUN/COPY steps run again or not). It is the transfer of file updated from the client to daemon. If the file has exactly the same metadata as previous build then it is considered a file that has not changed since previous build (of that project) and is not transferred.