runc run failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory

[bruceauyeung]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

os: a customized linux distro, based on centos8 very likely
kernel:Linux LIN-76F1FDC50C2 4.19.112-2.el8.x86_64 #1 SMP Wed Jun 10 09:04:49 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
docker:
├── containerd.io-1.6.28-3.2.el8.x86_64.rpm
├── docker-buildx-plugin-0.13.1-1.el8.x86_64.rpm
├── docker-ce-26.0.0-1.el8.x86_64.rpm
├── docker-ce-cli-26.0.0-1.el8.x86_64.rpm
├── docker-ce-rootless-extras-26.0.0-1.el8.x86_64.rpm
├── docker-compose-plugin-2.25.0-1.el8.x86_64.rpm
└── docker-scan-plugin-0.23.0-3.el8.x86_64.rpm

I'm sure that /bin/sh exists in image golang:1.21

error when run docker build -t controller:latest:

[+] Building 77.4s (15/16)                                                                                                                                                                    
 => [internal] load build definition from Dockerfile                                                                                                                                     0.0s
 => => transferring dockerfile: 1.32kB                                                                                                                                                   0.0s
 => [internal] load metadata for gcr.io/distroless/static:nonroot                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/golang:1.21                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                        0.0s
 => => transferring context: 160B                                                                                                                                                        0.0s
 => [stage-1 1/3] FROM gcr.io/distroless/static:nonroot                                                                                                                                  0.0s
 => [builder 1/9] FROM docker.io/library/golang:1.21                                                                                                                                     0.0s
 => [internal] load build context                                                                                                                                                        0.0s
 => => transferring context: 999B                                                                                                                                                        0.0s
 => [builder 2/9] WORKDIR /workspace                                                                                                                                                     1.1s
 => [builder 3/9] COPY go.mod go.mod                                                                                                                                                     3.1s
 => [builder 4/9] COPY go.sum go.sum                                                                                                                                                     3.1s
 => [builder 5/9] RUN go mod download                                                                                                                                                   48.1s
 => [builder 6/9] COPY cmd/main.go cmd/main.go                                                                                                                                           3.3s
 => [builder 7/9] COPY api/ api/                                                                                                                                                         3.1s
 => [builder 8/9] COPY internal/controller/ internal/controller/                                                                                                                         3.1s
 => ERROR [builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager cmd/main.go                                                                                     6.1s
------                                                                                                                                                                                        
 > [builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager cmd/main.go:
#0 0.548 runc run failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory
------
Dockerfile:24
--------------------
  22 |     # the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore,
  23 |     # by leaving it empty we can ensure that the container and binary shipped on it will have the same platform.
  24 | >>> RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go
  25 |     
  26 |     # Use distroless as minimal base image to package the manager binary
--------------------
ERROR: failed to solve: process "/bin/sh -c CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go" did not complete successfully: exit code: 1

Expected behaviour

docker builds sucessfully even when docker-buildx-plugin installed.

Actual behaviour

if docker-buildx-plugin removed ,docker builds sucessfully
if docker-buildx-plugin installed, docker builds failed

Buildx version

github.com/docker/buildx v0.13.1 7884339 , also tried docker-buildx-plugin-0.10.2-1.el8.x86_64.rpm but with same results.

Docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 24
  Running: 0
  Paused: 0
  Stopped: 24
 Images: 31
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs                                                                                                                                                                      
 Cgroup Version: 1                                                                                                                                                                            
 Plugins:                                                                                                                                                                                     
  Volume: local                                                                                                                                                                               
  Network: bridge host ipvlan macvlan null overlay                                                                                                                                            
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog                                                                                                                    
 Swarm: inactive                                                                                                                                                                              
 Runtimes: io.containerd.runc.v2 runc                                                                                                                                                         
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.19.112-2.el8.x86_64
 Operating System: hided
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.9GiB
 Name: LIN-76F1FDC50C2
 ID: 06862f76-59ee-41b8-bc3e-202307c22dbc
 Docker Root Dir: /media/vdc/data/docker
 Debug Mode: false
 HTTP Proxy: http://hide.hide.hide
 HTTPS Proxy: http://hide.hide.hide
 No Proxy: hide
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker                                 
 \_ default    \_ default       running   v0.13.1    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Configuration

# Build the manager binary
FROM golang:1.21 AS builder
ARG TARGETOS
ARG TARGETARCH

WORKDIR /workspace
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

# Copy the go source
COPY cmd/main.go cmd/main.go
COPY api/ api/
COPY internal/controller/ internal/controller/

# Build
# the GOARCH has not a default value to allow the binary be built according to the host where the command
# was called. For example, if we call make docker-build in a local env which has the Apple Silicon M1 SO
# the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore,
# by leaving it empty we can ensure that the container and binary shipped on it will have the same platform.
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532

ENTRYPOINT ["/manager"]

Build logs

~~~
[+] Building 77.4s (15/16)                                                                                                                                                                    
 => [internal] load build definition from Dockerfile                                                                                                                                     0.0s
 => => transferring dockerfile: 1.32kB                                                                                                                                                   0.0s
 => [internal] load metadata for gcr.io/distroless/static:nonroot                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/golang:1.21                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                        0.0s
 => => transferring context: 160B                                                                                                                                                        0.0s
 => [stage-1 1/3] FROM gcr.io/distroless/static:nonroot                                                                                                                                  0.0s
 => [builder 1/9] FROM docker.io/library/golang:1.21                                                                                                                                     0.0s
 => [internal] load build context                                                                                                                                                        0.0s
 => => transferring context: 999B                                                                                                                                                        0.0s
 => [builder 2/9] WORKDIR /workspace                                                                                                                                                     1.1s
 => [builder 3/9] COPY go.mod go.mod                                                                                                                                                     3.1s
 => [builder 4/9] COPY go.sum go.sum                                                                                                                                                     3.1s
 => [builder 5/9] RUN go mod download                                                                                                                                                   48.1s
 => [builder 6/9] COPY cmd/main.go cmd/main.go                                                                                                                                           3.3s
 => [builder 7/9] COPY api/ api/                                                                                                                                                         3.1s
 => [builder 8/9] COPY internal/controller/ internal/controller/                                                                                                                         3.1s
 => ERROR [builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager cmd/main.go                                                                                     6.1s
------                                                                                                                                                                                        
 > [builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager cmd/main.go:
#0 0.548 runc run failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory
------
Dockerfile:24
--------------------
  22 |     # the docker BUILDPLATFORM arg will be linux/arm64 when for Apple x86 it will be linux/amd64. Therefore,
  23 |     # by leaving it empty we can ensure that the container and binary shipped on it will have the same platform.
  24 | >>> RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go
  25 |     
  26 |     # Use distroless as minimal base image to package the manager binary
--------------------
ERROR: failed to solve: process "/bin/sh -c CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go" did not complete successfully: exit code: 1
~~~

Additional info

this go project is just a skeleton project created by operator-sdk
with docker-buildx-plugin installed, image pulling in docker build is ok, but i can not see that golang:1.21 image in docker images.
with docker-buildx-plugin removed, image pulling in docker build is ok, and i can see that golang:1.21 image in docker images

[tonistiigi]

Looks like most likely problem with your setup, or maybe with the specific to the command you are running. I see there is a previous RUN command that worked so try to figure out what condition or command makes it fail. If you want us to verify the Dockerfile by running it, provide a runnable version.

Additionally, in your go build command, with your current setup passing GOOS/GOARCH like this doesn't do anything.