Skip to content

Kubernetes driver not caching a layer that is being cached on the docker builder #1912

Open
@biancarosa

Description

@biancarosa

Contributing guidelines

I've found a bug and checked that ...

  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem

Description

I have a Dockerfile whose last layer, consisted of RUN poetry --no-ansi install --without=dev --no-root && rm -rf $POETRY_CACHE_DIR is not being cached on the Kubernetes driver.

Expected behaviour

This layer should be cached as the dependencies didn't change and take 0s.

Actual behaviour

The layer is not cached on the kubernetes builder while it gets cached on the docker builder. Other layers are cached just fine. It doesn't seem to be a poetry problem as the docker builder hits the cache on the 2nd build here, but when using the Kubernetes driver it always executes the poetry install and downloads the packages from the internet causing builds to be way slower on the Kubernetes driver.

Buildx version

github.com/docker/buildx v0.10.0-rc2-173-g2a83723d 2a83723

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.0-rc2-173-g2a83723d)
  compose: Docker Compose (Docker Inc., v2.15.1)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.18)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.6.0)

Server:
 Containers: 4
  Running: 1
  Paused: 0
  Stopped: 3
 Images: 884
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.764GiB
 Name: docker-desktop
 ID: 266T:FJ4U:KNOV:LM3O:U3GS:6YFI:JPYB:LTAT:VZQO:JIK6:SOOS:JPHS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE           DRIVER/ENDPOINT                                        STATUS   BUILDKIT PLATFORMS
k8s-builder         kubernetes                                                               
  k8s-builder0      kubernetes:///k8s-builder?deployment=&kubeconfig=      running  v0.11.6  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default *           docker                                                                   
  default           default                                                running  20.10.23 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Configuration

Dockerfile

docker buildx use k8s-builder
docker build .

Build logs

➜  example-poetry-docker-project git:(main) docker build .               
WARNING: No output specified with kubernetes driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 82.4s (12/12) FINISHED                                                                                                                                                                                    kubernetes:k8s-builder
 => [internal] load build definition from Dockerfile                                                                                                                                                                                     0.7s
 => => transferring dockerfile: 1.34kB                                                                                                                                                                                                   0.7s
 => [internal] load .dockerignore                                                                                                                                                                                                        0.5s
 => => transferring context: 2B                                                                                                                                                                                                          0.5s
 => [internal] load metadata for docker.io/library/python:3.10.6-slim-buster                                                                                                                                                             2.4s
 => [1/8] FROM docker.io/library/python:3.10.6-slim-buster@sha256:f17c94905c9cd56dce9ef6ce63229045a75f395f7b5b68eb69ef617079c51848                                                                                                       0.0s
 => => resolve docker.io/library/python:3.10.6-slim-buster@sha256:f17c94905c9cd56dce9ef6ce63229045a75f395f7b5b68eb69ef617079c51848                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                                                                        2.0s
 => => transferring context: 123.25kB                                                                                                                                                                                                    1.9s
 => CACHED [2/8] RUN pip install --no-cache-dir "poetry==1.4.0"                                                                                                                                                                          0.0s
 => [3/8] RUN poetry config virtualenvs.create false                                                                                                                                                                                     1.6s
 => [4/8] RUN apt-get update && apt-get install --no-install-recommends -y     cmake     g++     build-essential     libgomp1                                                                                                           21.6s
 => [5/8] RUN ldconfig                                                                                                                                                                                                                   0.2s
 => [6/8] RUN apt-get clean && apt-get autoclean && apt-get autoremove                                                                                                                                                                   2.2s 
 => [7/8] COPY poetry.lock pyproject.toml /                                                                                                                                                                                              0.1s 
 => [8/8] RUN poetry --no-ansi install --without=dev --no-root && rm -rf /tmp/poetry_cache                                                                                                                                              51.8s 
➜  example-poetry-docker-project git:(main) docker build .
[+] Building 82.2s (12/12) FINISHED                                                                                                                                                                                    kubernetes:k8s-builder
 => [internal] load build definition from Dockerfile                                                                                                                                                                                     0.7s
 => => transferring dockerfile: 1.34kB                                                                                                                                                                                                   0.6s
 => [internal] load .dockerignore                                                                                                                                                                                                        0.6s
 => => transferring context: 2B                                                                                                                                                                                                          0.6s
 => [internal] load metadata for docker.io/library/python:3.10.6-slim-buster                                                                                                                                                             0.7s
 => [1/8] FROM docker.io/library/python:3.10.6-slim-buster@sha256:f17c94905c9cd56dce9ef6ce63229045a75f395f7b5b68eb69ef617079c51848                                                                                                       0.0s
 => => resolve docker.io/library/python:3.10.6-slim-buster@sha256:f17c94905c9cd56dce9ef6ce63229045a75f395f7b5b68eb69ef617079c51848                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                                                                        1.1s
 => => transferring context: 123.25kB                                                                                                                                                                                                    1.1s
 => CACHED [2/8] RUN pip install --no-cache-dir "poetry==1.4.0"                                                                                                                                                                          0.0s
 => [3/8] RUN poetry config virtualenvs.create false                                                                                                                                                                                     1.5s
 => [4/8] RUN apt-get update && apt-get install --no-install-recommends -y     cmake     g++     build-essential     libgomp1                                                                                                           21.1s
 => [5/8] RUN ldconfig                                                                                                                                                                                                                   0.2s
 => [6/8] RUN apt-get clean && apt-get autoclean && apt-get autoremove                                                                                                                                                                   2.3s 
 => [7/8] COPY poetry.lock pyproject.toml /                                                                                                                                                                                              0.0s 
 => [8/8] RUN poetry --no-ansi install --without=dev --no-root && rm -rf /tmp/poetry_cache                                                                                                                                              54.1s 
WARNING: No output specified with kubernetes driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load                                                  


### Additional info

_No response_

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions