net/http timeout awaiting response headers when pushing large image layer

[quanvuminh]

I'm trying to push a large image, to my local Harbor registry, after built with
docker run --rm -it --privileged -v $HOME/.docker:/root/.docker -v $(pwd):/build/ --entrypoint=buildctl moby/buildkit:v0.12.0 --allow-insecure-registry --addr tcp://127.0.0.1:1234 build --frontend dockerfile.v0 --local context=/build --local dockerfile=/build --opt attest:provenance=mode=max --output type=image,name=127.0.0.1:8443/facebookincubator/below:test,push=true

and got this error

------
 > exporting to image:
------
error: failed to solve: failed to push 127.0.0.1:8443/facebookincubator/below:test: failed commit on ref "layer-sha256:edba82b3d04bef9ce1bcb76e88dfc28220815a54b8b55f607de2acfc5068dff2": failed to do request: Put "https://127.0.0.1:8443/v2/facebookincubator/below/blobs/uploads/7d0089f1-52bf-45c2-9327-54f28225d68c?_state=tNhz-uB7WCPNxMI_foHWnVAxl5sF1UD1DTy3f4aOiG57Ik5hbWUiOiJhZG1pbnMvYnVpbGQtZmItYmVsb3ciLCJVVUlEIjoiN2QwMDg5ZjEtNTJiZi00NWMyLTkzMjctNTRmMjgyMjVkNjhjIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTA4LTA5VDA5OjQwOjQ2LjY5NjM0MjYwOVoifQ%3D%3D&digest=sha256%3Aedba82b3d04bef9ce1bcb76e88dfc28220815a54b8b55f607de2acfc5068dff2": net/http: timeout awaiting response headers

I can successfully built and push using docker buildx build then docker push

$ docker image history 127.0.0.1:8443/facebookincubator/below:test
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
972a39404256   5 days ago     |2 below_version=0.7.1 build_rev=coccoc1 /bi…   703MB
af5516788767   5 days ago     /bin/sh -c #(nop)  ARG build_rev=coccoc1        0B
4210c5fbb666   5 days ago     /bin/sh -c #(nop)  ARG below_version=0.7.1      0B
b0e7c1d6d508   5 days ago     /bin/sh -c install_packages --no-install-rec…   2.06GB
bdcf99603646   5 days ago     /bin/sh -c #(nop)  ENV PATH=/root/.cargo/bin…   0B
2c68c743eb2d   5 days ago     /bin/sh -c #(nop) COPY file:23e078afa9435463…   154B
7fd9e9fb5277   5 days ago     /bin/sh -c #(nop) COPY file:fe7dfb20c67356c9…   3.39kB
ea6077f407fc   9 months ago                                                   67.5MB    from Bitnami with love

Access log from nginx-proxy in front of my local registry

[09/Aug/2023:09:41:24 +0000] 127.0.0.1 127.0.0.1 "PUT /v2/facebookincubator/below/blobs/uploads/7d0089f1-52bf-45c2-9327-54f28225d68c?_state=tNhz-uB7WCPNxMI_foHWnVAxl5sF1UD1DTy3f4aOiG57Ik5hbWUiOiJhZG1pbnMvYnVpbGQtZmItYmVsb3ciLCJVVUlEIjoiN2QwMDg5ZjEtNTJiZi00NWMyLTkzMjctNTRmMjgyMjVkNjhjIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTA4LTA5VDA5OjQwOjQ2LjY5NjM0MjYwOVoifQ%3D%3D&digest=sha256%3Aedba82b3d04bef9ce1bcb76e88dfc28220815a54b8b55f607de2acfc5068dff2 HTTP/1.1" 499 - "-" "buildkit/v0.0.0+unknown" "-" "-" "-" "127.0.0.1:8080" - 29.963 292377468 0 37.146

It's a HTTP 499 Client Closed Request so I assume that the problem comes from buildctl. Is there any timeout setting we can set somewhere?

[caplod]

I have the same error with large images. Other smaller images are working.

ERROR: failed to push <redacted>: failed to copy: unexpected status from PUT request to https://<redacted>: 499 status code 499