ResourceExhausted: grpc: received message larger than max (_ vs. 4194304)

[fenollp]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Maybe related: #2208

Expected behaviour

As this Dockerfile is generated in the same way as many others, I'd expect the build to not fail on transport error but maybe on my own logic error.

Actual behaviour

The error:

tame_index-50150de17bcc8948.Dockerfile:1
--------------------
   1 | >>> # syntax=docker.io/docker/dockerfile:1@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
   2 |     # Generated by rustcbuildx@0.2.0
   3 |     FROM scratch AS cratesio-adler-1.0.2-index.crates.io-6f17d22bba15001f
--------------------
ERROR: failed to solve: ResourceExhausted: ResourceExhausted: ResourceExhausted: ResourceExhausted: ResourceExhausted: grpc: received message larger than max (4212583 vs. 4194304)
1 dockerfile/1.7.1 /bin/dockerfile-frontend

Buildx version

github.com/docker/buildx v0.14.0 171fcbe

Docker info

Client: Docker Engine - Community
 Version:    26.1.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /home/pete/.docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 1411
 Server Version: 26.1.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-31-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.33GiB
 Name: solaralchemist
 ID: 4ff1ee7f-a3ff-4df0-ad6e-9d0162ddbda5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: fenollp
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker                                 
 \_ default    \_ default       running   v0.13.2    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Configuration

Command:

mkdir -p /tmp/empty
DOCKER_BUILDKIT=1 docker --debug build \
  --cache-from=type=registry,ref=docker.io/fenollp/myimgcach \
  --tag=docker.io/fenollp/myimgcach:lib-tame-index-0.7.1-50150de17bcc8948 \
  --build-arg=BUILDKIT_INLINE_CACHE=1 \
  --load \
  --network=none \
  --platform=local \
  --pull=false \
  --target=out-50150de17bcc8948 \
  --output=type=local,dest=/tmp/clis-cargo-deny_0-14-3/release/deps \
  --file=/tmp/clis-cargo-deny_0-14-3/release/tame_index-50150de17bcc8948.Dockerfile \
  --build-context=crate_out-1d97b4fdefd0456b=/tmp/empty \
  --build-context=crate_out-817ff5960242e670=/tmp/empty \
  --build-context=crate_out-829a16375a1c16e2=/tmp/empty \
  --build-context=crate_out-9630691b1405392b=/tmp/empty \
  --build-context=crate_out-9e0ba88a97d21589=/tmp/empty \
  --build-context=crate_out-baebf8a1ece7134d=/tmp/empty \
  --build-context=crate_out-efe8d4476a7ff505=/tmp/empty \
  --build-context=rust=docker-image://docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5 \
  /tmp/clis-cargo-deny_0-14-3/release

Huge Dockerfile: huge_Dockerfile.txt

Build logs

#0 building with "default" instance using docker driver

#1 [internal] load build definition from tame_index-50150de17bcc8948.Dockerfile
#1 transferring dockerfile: 1.03MB 0.0s done
#1 DONE 0.0s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#2 DONE 0.0s

#3 docker-image://docker.io/docker/dockerfile:1@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#3 CACHED

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 ...

#5 [context crate_out-829a16375a1c16e2] load .dockerignore
#5 transferring crate_out-829a16375a1c16e2: 2B done
#5 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 ...

#6 [context crate_out-1d97b4fdefd0456b] load .dockerignore
#6 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#6 [context crate_out-1d97b4fdefd0456b] load .dockerignore
#6 transferring crate_out-1d97b4fdefd0456b: 2B done
#6 DONE 0.0s

#7 [context crate_out-9e0ba88a97d21589] load .dockerignore
#7 transferring crate_out-9e0ba88a97d21589: 2B done
#7 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 ...

#8 [context crate_out-817ff5960242e670] load .dockerignore
#8 transferring crate_out-817ff5960242e670: 2B done
#8 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.0s

#9 [context crate_out-9630691b1405392b] load .dockerignore
#9 transferring crate_out-9630691b1405392b: 2B done
#9 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 ...

#10 [context crate_out-efe8d4476a7ff505] load .dockerignore
#10 transferring crate_out-efe8d4476a7ff505: 2B done
#10 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 ...

#11 [context crate_out-baebf8a1ece7134d] load .dockerignore
#11 transferring crate_out-baebf8a1ece7134d: 2B done
#11 DONE 0.0s

#4 [context rust] load metadata for docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5
#4 DONE 0.1s

#12 [internal] load .dockerignore
#12 transferring context: 2B done
#12 DONE 0.0s
tame_index-50150de17bcc8948.Dockerfile:1
--------------------
   1 | >>> # syntax=docker.io/docker/dockerfile:1@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
   2 |     # Generated by rustcbuildx@0.2.0
   3 |     FROM scratch AS cratesio-adler-1.0.2-index.crates.io-6f17d22bba15001f
--------------------
ERROR: failed to solve: ResourceExhausted: ResourceExhausted: ResourceExhausted: ResourceExhausted: ResourceExhausted: grpc: received message larger than max (4212586 vs. 4194304)
1 dockerfile/1.7.1 /bin/dockerfile-frontend
google.golang.org/grpc.(*ClientConn).Invoke
	/src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/frontend/gateway/pb.(*lLBBridgeClient).Solve
	/src/frontend/gateway/pb/gateway.pb.go:3078
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Solve
	/src/frontend/gateway/grpcclient/client.go:415
github.com/moby/buildkit/frontend/dockerfile/builder.Build.func5
	/src/frontend/dockerfile/builder/build.go:135
github.com/moby/buildkit/frontend/dockerui.(*Client).Build.func1
	/src/frontend/dockerui/build.go:39
golang.org/x/sync/errgroup.(*Group).Go.func1
	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

2864  /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Return
	/root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/gateway.go:975
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler.func1
	/root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/pb/gateway.pb.go:3441
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler
	/root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/pb/gateway.pb.go:3443
google.golang.org/grpc.(*Server).processUnaryRPC
	/root/build-deb/engine/vendor/google.golang.org/grpc/server.go:1343
google.golang.org/grpc.(*Server).handleStream
	/root/build-deb/engine/vendor/google.golang.org/grpc/server.go:1737
google.golang.org/grpc.(*Server).serveStreams.func1.1
	/root/build-deb/engine/vendor/google.golang.org/grpc/server.go:986
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

777251 v0.14.0 /usr/libexec/docker/cli-plugins/docker-buildx --debug buildx build --cache-from=type=registry,ref=docker.io/fenollp/myimgcach --tag=docker.io/fenollp/myimgcach:lib-tame-index-0.7.1-50150de17bcc8948 --build-arg=BUILDKIT_INLINE_CACHE=1 --load --network=none --platform=local --pull=false --target=out-50150de17bcc8948 --output=type=local,dest=/tmp/clis-cargo-deny_0-14-3/release/deps --file=/tmp/clis-cargo-deny_0-14-3/release/tame_index-50150de17bcc8948.Dockerfile --build-context=crate_out-1d97b4fdefd0456b=/tmp/empty --build-context=crate_out-817ff5960242e670=/tmp/empty --build-context=crate_out-829a16375a1c16e2=/tmp/empty --build-context=crate_out-9630691b1405392b=/tmp/empty --build-context=crate_out-9e0ba88a97d21589=/tmp/empty --build-context=crate_out-baebf8a1ece7134d=/tmp/empty --build-context=crate_out-efe8d4476a7ff505=/tmp/empty --build-context=rust=docker-image://docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5 /tmp/clis-cargo-deny_0-14-3/release --progress=plain
google.golang.org/grpc.(*ClientConn).Invoke
	google.golang.org/grpc@v1.59.0/call.go:35
github.com/moby/buildkit/frontend/gateway/pb.(*lLBBridgeClient).Solve
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/frontend/gateway/pb/gateway.pb.go:3078
github.com/moby/buildkit/client.(*gatewayClientForBuild).Solve
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/client/build.go:94
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Solve
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/frontend/gateway/grpcclient/client.go:415
github.com/docker/buildx/build.BuildWithResultHandler.func3.1.2
	github.com/docker/buildx/build/build.go:405
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Run
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/frontend/gateway/grpcclient/client.go:218
github.com/moby/buildkit/client.(*Client).Build.func2
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/client/build.go:59
github.com/moby/buildkit/client.(*Client).solve.func3
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/client/solve.go:300
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78
runtime.goexit
	runtime/asm_amd64.s:1650

2864  /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Return
	/root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/gateway.go:975
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Return
	/root/build-deb/engine/vendor/github.com/moby/buildkit/control/gateway/gateway.go:146
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler.func1
	/root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/pb/gateway.pb.go:3441
github.com/docker/docker/api/server/router/grpc.NewRouter.ChainUnaryServer.func1.1.1
	/root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
	/root/build-deb/engine/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:326
github.com/docker/docker/api/server/router/grpc.unaryInterceptor.func1
	/root/build-deb/engine/api/server/router/grpc/grpc.go:58
github.com/docker/docker/api/server/router/grpc.NewRouter.ChainUnaryServer.func1.1.1
	/root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/docker/docker/api/server/router/grpc.NewRouter.ChainUnaryServer.func1
	/root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler
	/root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/pb/gateway.pb.go:3443
google.golang.org/grpc.(*Server).processUnaryRPC
	/root/build-deb/engine/vendor/google.golang.org/grpc/server.go:1343
google.golang.org/grpc.(*Server).handleStream
	/root/build-deb/engine/vendor/google.golang.org/grpc/server.go:1737
google.golang.org/grpc.(*Server).serveStreams.func1.1
	/root/build-deb/engine/vendor/google.golang.org/grpc/server.go:986
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

777251 v0.14.0 /usr/libexec/docker/cli-plugins/docker-buildx --debug buildx build --cache-from=type=registry,ref=docker.io/fenollp/myimgcach --tag=docker.io/fenollp/myimgcach:lib-tame-index-0.7.1-50150de17bcc8948 --build-arg=BUILDKIT_INLINE_CACHE=1 --load --network=none --platform=local --pull=false --target=out-50150de17bcc8948 --output=type=local,dest=/tmp/clis-cargo-deny_0-14-3/release/deps --file=/tmp/clis-cargo-deny_0-14-3/release/tame_index-50150de17bcc8948.Dockerfile --build-context=crate_out-1d97b4fdefd0456b=/tmp/empty --build-context=crate_out-817ff5960242e670=/tmp/empty --build-context=crate_out-829a16375a1c16e2=/tmp/empty --build-context=crate_out-9630691b1405392b=/tmp/empty --build-context=crate_out-9e0ba88a97d21589=/tmp/empty --build-context=crate_out-baebf8a1ece7134d=/tmp/empty --build-context=crate_out-efe8d4476a7ff505=/tmp/empty --build-context=rust=docker-image://docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5 /tmp/clis-cargo-deny_0-14-3/release --progress=plain
google.golang.org/grpc.(*ClientConn).Invoke
	google.golang.org/grpc@v1.59.0/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/api/services/control/control.pb.go:2234
github.com/moby/buildkit/client.(*Client).solve.func2
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/client/solve.go:274
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78
runtime.goexit
	runtime/asm_amd64.s:1650

777251 v0.14.0 /usr/libexec/docker/cli-plugins/docker-buildx --debug buildx build --cache-from=type=registry,ref=docker.io/fenollp/myimgcach --tag=docker.io/fenollp/myimgcach:lib-tame-index-0.7.1-50150de17bcc8948 --build-arg=BUILDKIT_INLINE_CACHE=1 --load --network=none --platform=local --pull=false --target=out-50150de17bcc8948 --output=type=local,dest=/tmp/clis-cargo-deny_0-14-3/release/deps --file=/tmp/clis-cargo-deny_0-14-3/release/tame_index-50150de17bcc8948.Dockerfile --build-context=crate_out-1d97b4fdefd0456b=/tmp/empty --build-context=crate_out-817ff5960242e670=/tmp/empty --build-context=crate_out-829a16375a1c16e2=/tmp/empty --build-context=crate_out-9630691b1405392b=/tmp/empty --build-context=crate_out-9e0ba88a97d21589=/tmp/empty --build-context=crate_out-baebf8a1ece7134d=/tmp/empty --build-context=crate_out-efe8d4476a7ff505=/tmp/empty --build-context=rust=docker-image://docker.io/library/rust:1.78.0-slim@sha256:517c6272b328bc51c87e099ef4adfbc7ab4558af2d757e8d423c7c3f1cbbf9d5 /tmp/clis-cargo-deny_0-14-3/release --progress=plain
github.com/moby/buildkit/client.(*Client).solve.func2
	github.com/moby/buildkit@v0.13.0-rc3.0.20240417151852-71f99c52a669/client/solve.go:290
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78
runtime.goexit
	runtime/asm_amd64.s:1650

Additional info

Note I adapted the call so you can reproduce, but doing so I removed some correct --build-context=.. to refer to some empty directory. This does not have any effect on the reported error, which seems to happen soon enough.

[thompson-shaun]

Transferred to moby/buildkit#4948