make image-build takes long to build inference extension docker image

[nirrozenbaum]

When building the inference extension using the make image-build, the docker build takes too long.
I see build time of 5 minutes (minimal time, sometimes a bit longer). some of my colleagues experience even greater build time (10-15 minutes).
for speeding up development and local testing, it would beneficial to solve it and reduce the build time.

[liangyuanpeng]

In which aspect is time mainly consumed? Download the basic container image or compile go? I tried it once on gitpod and it was very fast

gitpod /workspace/gateway-api-inference-extension (main) $ GO_VERSION=1.24 make image-build
docker buildx build -t us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:d935a7c \
        --platform=linux/amd64 \
        --build-arg BASE_IMAGE=gcr.io/distroless/static:nonroot \
        --build-arg BUILDER_IMAGE=golang:1.24 \
         \
         \
         ./
[+] Building 129.7s (18/18) FINISHED                                                                                                                                                                                                                                                                   docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                             0.0s
 => => transferring dockerfile: 679B                                                                                                                                                                                                                                                                             0.0s
 => [internal] load metadata for gcr.io/distroless/static:nonroot                                                                                                                                                                                                                                                0.4s
 => [internal] load metadata for docker.io/library/golang:1.24                                                                                                                                                                                                                                                   0.7s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                0.0s
 => => transferring context: 160B                                                                                                                                                                                                                                                                                0.0s
 => [builder  1/10] FROM docker.io/library/golang:1.24@sha256:d9db32125db0c3a680cfb7a1afcaefb89c898a075ec148fdc2f0f646cc2ed509                                                                                                                                                                                  18.1s
 => => resolve docker.io/library/golang:1.24@sha256:d9db32125db0c3a680cfb7a1afcaefb89c898a075ec148fdc2f0f646cc2ed509                                                                                                                                                                                             0.0s
 => => sha256:86d1854c5e3d0539ccc5eb052ebb60c8d40abf34e3a17833ccb2167931527d22 2.32kB / 2.32kB                                                                                                                                                                                                                   0.0s
 => => sha256:ec829c45babe26254b2b8aa569b45d60af9f2a8c3cc8d94f3240a4ca34ecfeac 2.80kB / 2.80kB                                                                                                                                                                                                                   0.0s
 => => sha256:23b7d26ef1d294256da0d70ce374277b9aab5ca683015073316005cb63d33849 48.49MB / 48.49MB                                                                                                                                                                                                                 0.8s
 => => sha256:d9db32125db0c3a680cfb7a1afcaefb89c898a075ec148fdc2f0f646cc2ed509 10.06kB / 10.06kB                                                                                                                                                                                                                 0.0s
 => => sha256:07d1b5af933d2dfc3d0dd509d6e20534825e4a537f7b006a6cb5b8e5a1f20905 24.01MB / 24.01MB                                                                                                                                                                                                                 0.8s
 => => sha256:1eb98adba0eb44a2e4facf9ca3626a4a66feedd0dd56d159cca90a35205744e7 64.40MB / 64.40MB                                                                                                                                                                                                                 0.9s
 => => extracting sha256:23b7d26ef1d294256da0d70ce374277b9aab5ca683015073316005cb63d33849                                                                                                                                                                                                                        2.8s
 => => sha256:6eb08228d7a5ec326856f57d2a6db89dd7a3ee8773f9adc904269c2f16e4eade 92.33MB / 92.33MB                                                                                                                                                                                                                 1.7s
 => => sha256:f1d296901bdc593d88a0813bb00eef0974b68222cba6add046b831c086a1c68c 78.94MB / 78.94MB                                                                                                                                                                                                                 2.0s
 => => sha256:cc508665364bf890145a9b00cb0af7704946c2051213fcb9f3be3df777619bc0 127B / 127B                                                                                                                                                                                                                       1.0s
 => => sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B                                                                                                                                                                                                                         1.1s
 => => extracting sha256:07d1b5af933d2dfc3d0dd509d6e20534825e4a537f7b006a6cb5b8e5a1f20905                                                                                                                                                                                                                        0.8s
 => => extracting sha256:1eb98adba0eb44a2e4facf9ca3626a4a66feedd0dd56d159cca90a35205744e7                                                                                                                                                                                                                        3.2s
 => => extracting sha256:6eb08228d7a5ec326856f57d2a6db89dd7a3ee8773f9adc904269c2f16e4eade                                                                                                                                                                                                                        3.5s
 => => extracting sha256:f1d296901bdc593d88a0813bb00eef0974b68222cba6add046b831c086a1c68c                                                                                                                                                                                                                        5.8s
 => => extracting sha256:cc508665364bf890145a9b00cb0af7704946c2051213fcb9f3be3df777619bc0                                                                                                                                                                                                                        0.0s
 => => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1                                                                                                                                                                                                                        0.0s
 => CACHED [stage-1 1/3] FROM gcr.io/distroless/static:nonroot@sha256:c0f429e16b13e583da7e5a6ec20dd656d325d88e6819cafe0adb0828976529dc                                                                                                                                                                           0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                0.0s
 => => transferring context: 5.54kB                                                                                                                                                                                                                                                                              0.0s
 => [builder  2/10] WORKDIR /src                                                                                                                                                                                                                                                                                 1.2s
 => [builder  3/10] COPY go.mod go.sum ./                                                                                                                                                                                                                                                                        0.0s
 => [builder  4/10] RUN go mod download                                                                                                                                                                                                                                                                          5.2s
 => [builder  5/10] COPY cmd ./cmd                                                                                                                                                                                                                                                                               0.0s
 => [builder  6/10] COPY pkg ./pkg                                                                                                                                                                                                                                                                               0.0s
 => [builder  7/10] COPY internal ./internal                                                                                                                                                                                                                                                                     0.0s
 => [builder  8/10] COPY api ./api                                                                                                                                                                                                                                                                               0.0s
 => [builder  9/10] WORKDIR /src/cmd/epp                                                                                                                                                                                                                                                                         0.0s
 => [builder 10/10] RUN go build -o /epp                                                                                                                                                                                                                                                                       102.4s
 => [stage-1 2/3] COPY --from=builder /epp /epp                                                                                                                                                                                                                                                                  0.0s
 => exporting to image                                                                                                                                                                                                                                                                                           0.2s
 => => exporting layers                                                                                                                                                                                                                                                                                          0.2s
 => => writing image sha256:94e713297f0765e15ae2114fbcbd80c7994bd396ae38884dccb6def6f280d956                                                                                                                                                                                                                     0.0s
 => => naming to us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:d935a7c                                                                                                                                                                                                       0.0s

[nirrozenbaum]

we see most of the time spent on go build command

[builder 10/10] RUN go build -o /epp

when I build it takes around ~300 seconds

[kfswain]

I just ran it on my local env and it only took ~120s

Looking at our build & push CI:

2025/04/24 15:48:41 Running build jobs...
2025/04/24 15:48:41 No variants.yaml, starting single build job...
2025/04/24 16:04:[12](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/post-inference-extension-push-images/1915432684079812608#1:build-log.txt%3A12) Successfully built image:

timestamps suggest ~12 min

However this CI run, which builds the image as part of our test run,

#16 [builder 10/10] RUN go build -o /epp
#16 DONE 52.9s

Takes less than a minute

[kfswain]

I'm gonna mark this as needing info before we can have someone investigate, unless you see clear actionables @nirrozenbaum ?

[nirrozenbaum]

I see no clear actionables. I was hoping maybe someone from the community experienced this as well and identified where the issue could be.