Buildkit should wait for builds to complete before shutting down

[Evesy]

When a SIGTERM is sent to a running buildkit daemon (being used as a remote builder) it immediately stops the server which results in any ongoing builds using that server being abruptly cancelled and failing.

Buildkit should ideally wait for any running builds to complete before shutting down; allowing graceful deployments of buildkit instances

[tonistiigi]

ref #4065

[colinodell]

Related to #3282

[remidebette]

Hi @colinodell,
Did you ever come up with a preStop script for terminating gracefully in a kubernetes context?

Thank you

[colinodell]

@remidebette We did, and we just open-sourced it! https://github.com/seatgeek/buildkit-prestop-script

We would certainly prefer a solution implemented directly in buildkit! Unfortunately, we weren't able to figure that out ourselves, so we put this script together as a temporary stop-gap until this issue is resolved. I'm sure there's room for improvement, but it's been working well enough for our needs so far.

[remidebette]

Awesome @colinodell , that is an instant star, thank you for this

[PidgeyBE]

Nice work @colinodell !
I've created a small PR to make sure the logs of your script end up in the pod logs as well.
Also extended the readme with an alternative approach to install the prestop hook.
seatgeek/buildkit-prestop-script#1

[chaima-belhedi]

Hi all 👋

We recently ran into a situation that appears closely related to this thread. During a few of our CI runs, the build became stuck for over 45 minutes during the export cache step. Upon investigation, we discovered that the BuildKit pod had already been terminated — most likely due to Kubernetes autoscaling — while the client was still waiting for a final confirmation or EOF.

It looks like the daemon completed its export work, but got shut down just before sending the final response. As a result, the client kept hanging indefinitely — it never received the expected completion signal.

We suspect a race condition:

• Export completes
• Connection is still open
• Daemon exits too early (preStop isn’t long enough or missing)
• Client gets stuck waiting forever

Possibly related to #5961, where a missing EOF on the pipe caused the client to hang. The pattern looks similar: the daemon terminates without a clean close, leaving the client waiting indefinitely.

Happy to share more details if helpful!

[remidebette]

Hi, same team here.
Our build github actions show this deadlock randomly after the export cache step.
We have been using the prestop script and are indeed suspecting it is involved.

CI: github action on kubernetes, using the build-push-action
buildkit client version: v0.25.0
buildkit remote version: v0.23.2 (we were not seing this in v0.21.0)

Build logs:

/usr/bin/docker buildx build --cache-from type=registry,image-manifest=true,oci-mediatypes=true,ref=<EDITED>:new-branch-to-test-buildkit --cache-to type=<EDITED>:new-branch-to-test-buildkit,mode=max,compression=zstd,compression-level=6 <More edited: tags, args, file, push>
#0 building with "builder-bcb7ec69-dd59-444c-8fd3-b8961644e695" instance using remote driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 2.32kB done
#1 DONE 0.0s

#2 [linux/amd64 internal] load metadata for docker.io/library/node:18-alpine
#2 ...

#3 [auth] library/node:pull token for registry-1.docker.io
#3 DONE 0.0s

#2 [linux/amd64 internal] load metadata for docker.io/library/node:18-alpine
#2 DONE 1.8s

#4 [internal] load .dockerignore
#4 DONE 0.0s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.1s

#5 [internal] load build context
#5 DONE 0.0s

#6 [linux/amd64 base 1/1] FROM docker.io/library/node:18-alpine@sha256:8d6421d663b4c28fd3ebc498332f249011d118945588d0a35cb9bc4b8ca09d9e
#6 resolve docker.io/library/node:18-alpine@sha256:8d6421d663b4c28fd3ebc498332f249011d118945588d0a35cb9bc4b8ca09d9e 0.0s done
#6 DONE 0.0s

#7 importing cache manifest from <EDITED>:new-branch-to-test-buildkit
#7 ...

#8 [auth] <EDITED>:pull token for <EDITED>
#8 DONE 0.0s

#7 importing cache manifest from <EDITED>:new-branch-to-test-buildkit
#7 ERROR: failed to configure registry cache importer: <EDITED>:new-branch-to-test-buildkit: not found

#5 [internal] load build context
#5 transferring context: 405.04kB 0.0s done
#5 DONE 0.2s

#6 [linux/amd64 base 1/1] FROM docker.io/library/node:18-alpine@sha256:8d6421d663b4c28fd3ebc498332f249011d118945588d0a35cb9bc4b8ca09d9e
#6 sha256:25ff2da83641908f65c3a74d80409d6b1b62ccfaab220b9ea70b80df5a2e0549 446B / 446B 0.2s
#6 sha256:25ff2da83641908f65c3a74d80409d6b1b62ccfaab220b9ea70b80df5a2e0549 446B / 446B 0.2s done
#6 sha256:1e5a4c89cee5c0826c540ab06d4b6b491c96eda01837f430bd47f0d26702d6e3 1.26MB / 1.26MB 0.2s done
#6 sha256:dd71dde834b5c203d162902e6b8994cb2309ae049a0eabc4efea161b2b5a3d0e 6.29MB / 40.01MB 0.2s
#6 sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 3.64MB / 3.64MB 0.2s
#6 sha256:dd71dde834b5c203d162902e6b8994cb2309ae049a0eabc4efea161b2b5a3d0e 40.01MB / 40.01MB 0.4s done
#6 sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 3.64MB / 3.64MB 0.2s done
#6 extracting sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870
#6 extracting sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 0.2s done
#6 DONE 1.1s

#6 [linux/amd64 base 1/1] FROM docker.io/library/node:18-alpine@sha256:8d6421d663b4c28fd3ebc498332f249011d118945588d0a35cb9bc4b8ca09d9e
#6 extracting sha256:dd71dde834b5c203d162902e6b8994cb2309ae049a0eabc4efea161b2b5a3d0e
#6 extracting sha256:dd71dde834b5c203d162902e6b8994cb2309ae049a0eabc4efea161b2b5a3d0e 2.8s done
#6 DONE 3.9s

#6 [linux/amd64 base 1/1] FROM docker.io/library/node:18-alpine@sha256:8d6421d663b4c28fd3ebc498332f249011d118945588d0a35cb9bc4b8ca09d9e
#6 extracting sha256:1e5a4c89cee5c0826c540ab06d4b6b491c96eda01837f430bd47f0d26702d6e3
#6 extracting sha256:1e5a4c89cee5c0826c540ab06d4b6b491c96eda01837f430bd47f0d26702d6e3 0.2s done
#6 DONE 4.1s

#6 [linux/amd64 base 1/1] FROM docker.io/library/node:18-alpine@sha256:8d6421d663b4c28fd3ebc498332f249011d118945588d0a35cb9bc4b8ca09d9e
#6 extracting sha256:25ff2da83641908f65c3a74d80409d6b1b62ccfaab220b9ea70b80df5a2e0549 0.0s done
#6 DONE 4.1s

#9 [linux/amd64 builder 1/5] WORKDIR /app
#9 DONE 0.0s

#10 [linux/amd64 deps 1/6] RUN apk add --no-cache libc6-compat
#10 0.129 fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/x86_64/APKINDEX.tar.gz
#10 ...

#11 [linux/amd64 runner 2/6] RUN addgroup --system --gid 1001 nodejs
#11 DONE 0.2s

#12 [linux/amd64 runner 3/6] RUN adduser --system --uid 1001 nextjs
#12 DONE 0.2s

#10 [linux/amd64 deps 1/6] RUN apk add --no-cache libc6-compat
#10 0.606 fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/x86_64/APKINDEX.tar.gz
#10 1.055 (1/3) Installing musl-obstack (1.2.3-r2)
#10 1.092 (2/3) Installing libucontext (1.3.2-r0)
#10 1.123 (3/3) Installing gcompat (1.1.0-r4)
#10 1.159 OK: 10 MiB in 20 packages
#10 DONE 1.5s

#13 [linux/amd64 deps 2/6] WORKDIR /app
#13 DONE 0.0s

<Misc build>

#25 exporting to image
#25 exporting layers
#25 exporting layers 0.9s done
#25 exporting manifest sha256:57be6bbc71721f07c3baf24d502c5a64146394f886048dc3ffa525596507d81d done
#25 exporting config sha256:0155d449515a4fe845fcb18d3a09ae1a10ee52d2de27b81681891a779d87b775 done
#25 exporting attestation manifest sha256:6c07ce226b5cb9c77cc64e4cbc562def7c04ae4f7ae810beec2eaca99fc2b8a9 done
#25 exporting manifest sha256:a159fe5ad164835cde59a796bb64e21411714f5cb19a67b00c6dba0f087677ff done
#25 exporting config sha256:687b8d323dfd95067454982926ed67d8fb0f8dafb7c79d1865b8b296a46725cb done
#25 exporting attestation manifest sha256:845d793854a36990190d5d7511ae71ff23a4786263b7a81ca8cc25bb4dc27da2 0.0s done
#25 exporting manifest list sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc done
#25 pushing layers
#25 ...

#26 [auth] <EDITED>:pull,push token for <EDITED>
#26 DONE 0.0s

#25 exporting to image
#25 pushing layers 1.5s done
#25 pushing manifest for <EDITED>:539fdc6fc3c882337b43d58d0f907876879cce05@sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc
#25 pushing manifest for <EDITED>:539fdc6fc3c882337b43d58d0f907876879cce05@sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc 0.6s done
#25 pushing layers 0.1s done
#25 pushing manifest for <EDITED>:new-branch-to-test-buildkit@sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc
#25 pushing manifest for <EDITED>:new-branch-to-test-buildkit@sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc 0.2s done
#25 pushing layers 0.2s done
#25 pushing manifest for <EDITED>:539fdc6f@sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc
#25 pushing manifest for <EDITED>:539fdc6f@sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc 0.2s done
#25 pushing layers 0.2s done
#25 pushing manifest for <EDITED>:v1.0@sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc
#25 pushing manifest for <EDITED>:v1.0@sha256:cd2fa6728cf39c25208611877b480e86c20610049337796a95b66c28b4530cbc 0.2s done
#25 DONE 4.2s

#27 exporting cache to registry
#27 preparing build cache for export
#27 writing layer sha256:01e9952aefe129776d4a9e52bbecfc2ecd07a53e324c378c48bacc1136871d0f
#27 ...

#28 [auth] <EDITED>:pull,push token for <EDITED>
#28 DONE 0.0s

#27 exporting cache to registry
#27 writing layer sha256:01e9952aefe129776d4a9e52bbecfc2ecd07a53e324c378c48bacc1136871d0f 300.4s done
#27 writing layer sha256:10c2ee203c2994d97012b977a0d959cedd42478cc287cf7bdf12627ed2f38c00
#27 writing layer sha256:10c2ee203c2994d97012b977a0d959cedd42478cc287cf7bdf12627ed2f38c00 0.0s done
#27 writing layer sha256:1851c8e757f832a794dd15e382f20d6342ca738e6be0e06d491daa368ed666ba 0.0s done
#27 writing layer sha256:1b05ced3804e0c557ec47816384832e2ae88bef67d5d3102f5e2f2dc9bd7c15e 0.0s done
#27 writing layer sha256:1d8bd37d121d501bf768976bbeee6af8e474d35a98d7406305a946943d946bf1 0.0s done
#27 writing layer sha256:1e5a4c89cee5c0826c540ab06d4b6b491c96eda01837f430bd47f0d26702d6e3
#27 writing layer sha256:1e5a4c89cee5c0826c540ab06d4b6b491c96eda01837f430bd47f0d26702d6e3 0.0s done
#27 writing layer sha256:25ff2da83641908f65c3a74d80409d6b1b62ccfaab220b9ea70b80df5a2e0549 0.0s done
#27 writing layer sha256:2f452eb341b5104484e14657d59cb6f37963d1164da102f1bec068324cabab4f 0.0s done
#27 writing layer sha256:38094f68ed576dca07e18327627d0bca982c3b5e5f94e4b59f274af9862977c1
#27 writing layer sha256:38094f68ed576dca07e18327627d0bca982c3b5e5f94e4b59f274af9862977c1 0.0s done
#27 writing layer sha256:67ed8e816cd4ddd3fc6c1652c93f537a801673e96240270caa68d08933c46857 0.0s done
#27 writing layer sha256:72374e8eeb64830e5cbd0915ffbdaec44e61b7641652dd57851e66460817f912 0.0s done
#27 writing layer sha256:807b92947d7e17f2696a51604a5082a3610740d5c929aa6a7c8ed0d693760dfd
#27 writing layer sha256:807b92947d7e17f2696a51604a5082a3610740d5c929aa6a7c8ed0d693760dfd 0.0s done
#27 writing layer sha256:8222c036fd0ddf92899acfffe2690ae1d5d1cb2971589bf187b0d738eecba2dc 0.0s done
#27 writing layer sha256:925ecd16dabc9162b4a352d872260c1563819f1af175cc8384d47924b29a5976 0.0s done
#27 writing layer sha256:9f03d7c0391fb6b216f53b67fbeee66ef9a4f47410c85a6765a6ccbccd85898f 0.0s done
#27 writing layer sha256:9f2ab099d86c0a921bd5ccaf7455618dd0ea083accfad85ccbe7311e39ab104b
#27 writing layer sha256:9f2ab099d86c0a921bd5ccaf7455618dd0ea083accfad85ccbe7311e39ab104b 0.0s done
#27 writing layer sha256:afe8bc454445be27b82d929e15a929166e8212e0b1c4f8ff2d50225447298750 0.0s done
#27 writing layer sha256:c53c1ead4e98bfe214181c4daedb388db6da418b1eb42eabf3f93a83b15b3191 0.0s done
#27 writing layer sha256:dd71dde834b5c203d162902e6b8994cb2309ae049a0eabc4efea161b2b5a3d0e 0.0s done
#27 writing layer sha256:e30f99d2289f4fc13d6e80d9a1036d34d529c982d51af033c28adf9018e37675
#27 writing layer sha256:e30f99d2289f4fc13d6e80d9a1036d34d529c982d51af033c28adf9018e37675 0.0s done
#27 writing layer sha256:f18232174bc91741fdf3da96d85011092101a032a93a388b79e99e69c2d5c870 0.0s done
#27 writing layer sha256:f2bbf31f8a444a49e31e62dde5ca526b6c965a70011127f2acfd1d7b250118d6 0.0s done
#27 writing config sha256:d19cf5241d8510b67343cffbf2ff673b331018a8a4ee72b48ff7e71a68b15cc5
#27 writing config sha256:d19cf5241d8510b67343cffbf2ff673b331018a8a4ee72b48ff7e71a68b15cc5 0.3s done
#27 preparing build cache for export 401.0s done
#27 writing cache image manifest sha256:f185d5b3481a03191174fbb40c00b79366e6b5a1ba7a3ca0453a6b7e7f2cf034 0.1s done
#27 DONE 401.0s

After the last #27 DONE 401.0s, we observe the build-push-action stays hanging for more than 45min, and that the remote buildkit server is already removed.

Kubernetes event logs (the remote buildkit server pod did not stay up for long):

56m         Normal    Scheduled           pod/buildkit-buildkit-service-767649887c-rn6n9      Successfully assigned buildkit/buildkit-buildkit-service-767649887c-rn6n9 to node010
56m         Normal    Pulled              pod/buildkit-buildkit-service-767649887c-rn6n9      Container image "moby/buildkit:v0.23.2-rootless" already present on machine
56m         Normal    Created             pod/buildkit-buildkit-service-767649887c-rn6n9      Created container buildkit-service
56m         Normal    Started             pod/buildkit-buildkit-service-767649887c-rn6n9      Started container buildkit-service
50m         Normal    Killing             pod/buildkit-buildkit-service-767649887c-rn6n9      Stopping container buildkit-service

Remote buildkit logs:
We see the prestop hook waited for 10s, most likely happened during the cache export?

1753276365858	2025-07-23T13:12:45.858Z	[rootlesskit:parent] error: child exited: exit status 1
1753276365852	2025-07-23T13:12:45.852Z	[rootlesskit:child ] error: command [buildkitd --addr unix:///run/user/1000/buildkit/buildkitd.sock --addr tcp://0.0.0.0:1234 --oci-worker-no-process-sandbox] exited: exit status 1
1753276365804	2025-07-23T13:12:45.804Z		/usr/local/go/src/runtime/asm_amd64.s:1700
1753276365804	2025-07-23T13:12:45.804Z	runtime.goexit
1753276365804	2025-07-23T13:12:45.804Z		/src/util/appcontext/appcontext.go:38
1753276365804	2025-07-23T13:12:45.804Z	github.com/moby/buildkit/util/appcontext.Context.func1.1
1753276365804	2025-07-23T13:12:45.804Z	buildkitd: got 1 SIGTERM/SIGINTs, forcing shutdown
1753276365803	2025-07-23T13:12:45.803Z	time="2025-07-23T13:12:45Z" level=info msg="stopping server"
1753276365771	2025-07-23T13:12:45.771Z	{"message": "[PreStop Hook] All build processes seem to have stopped. Exiting now.", "timestamp": "2025-07-23T13:12:45Z", "level": "info"}
1753276355517	2025-07-23T13:12:35.517Z	{"message": "[PreStop Hook] Waiting for build processes to finish...", "timestamp": "2025-07-23T13:12:35Z", "level": "info"}
1753276235094	2025-07-23T13:10:35.094Z	time="2025-07-23T13:10:35Z" level=warning msg="reference for unknown type: application/vnd.buildkit.cacheconfig.v0" spanID=127362e47afb72b3 traceID=112f706f128d200bc4449d1f34f8d197
1753276220969	2025-07-23T13:10:20.969Z	time="2025-07-23T13:10:20Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations" span="exporting to image" spanID=56b5f816c5f03cf2 traceID=112f706f128d200bc4449d1f34f8d197
1753275996133	2025-07-23T13:06:36.133Z	time="2025-07-23T13:06:36Z" level=info msg="running server on /run/user/1000/buildkit/buildkitd.sock"
1753275996133	2025-07-23T13:06:36.133Z	time="2025-07-23T13:06:36Z" level=info msg="running server on [::]:1234"
1753275996130	2025-07-23T13:06:36.130Z	time="2025-07-23T13:06:36Z" level=warning msg="currently, only the default worker can be used."
1753275996130	2025-07-23T13:06:36.130Z	time="2025-07-23T13:06:36Z" level=info msg="found 1 workers, default=\"pzo5zl4l4mnlak1j31mo50iui\""
1753275996130	2025-07-23T13:06:36.130Z	time="2025-07-23T13:06:36Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
1753275996129	2025-07-23T13:06:36.129Z	time="2025-07-23T13:06:36Z" level=info msg="found worker \"pzo5zl4l4mnlak1j31mo50iui\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:buildkit-buildkit-service-767649887c-rn6n9 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:no-sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
1753275996119	2025-07-23T13:06:36.119Z	time="2025-07-23T13:06:36Z" level=warning msg="CDI setup error /etc/buildkit/cdi: failed to monitor for changes: no such file or directory"
1753275996119	2025-07-23T13:06:36.119Z	time="2025-07-23T13:06:36Z" level=warning msg="CDI setup error /var/run/cdi: failed to monitor for changes: no such file or directory"
1753275996119	2025-07-23T13:06:36.119Z	time="2025-07-23T13:06:36Z" level=warning msg="CDI setup error /etc/cdi: failed to monitor for changes: no such file or directory"
1753275996119	2025-07-23T13:06:36.119Z	time="2025-07-23T13:06:36Z" level=warning msg="NoProcessSandbox is enabled. Note that NoProcessSandbox allows build containers to kill (and potentially ptrace) an arbitrary process in the BuildKit host namespace. NoProcessSandbox should be enabled only when the BuildKit is running in a container as an unprivileged user."
1753275996118	2025-07-23T13:06:36.118Z	time="2025-07-23T13:06:36Z" level=info msg="auto snapshotter: using overlayfs"
1753275996107	2025-07-23T13:06:36.107Z	time="2025-07-23T13:06:36Z" level=warning msg="TLS is not enabled for tcp://0.0.0.0:1234. enabling mutual TLS authentication is highly recommended"

[remidebette]

We are also thinking whether this could be a bad interaction with github action scaleset (that we use on our kubernetes) due to this issue
actions/actions-runner-controller#4148

[remidebette]

More on this,
I am now randomly investigating on my buildkit pods, after making sure no new jobs connect to them.

I can see after a while a big list of builds still ongoing
So the deadlocked builds are still ongoing on the buildkit remote side

/ $ buildctl debug info
BuildKit: github.com/moby/buildkit v0.23.2 40b2ede0ac0a37030f9959b4a28e9c6c8ea036e7
/ $ buildctl debug monitor
event: STARTED ref:wp6eg7oxqm76vlnvmzf9uhhlk
event: STARTED ref:jaw7etp10op8ujupkp3b45cli
event: STARTED ref:1lc6g7cf6iet4vg7gskk5emuj
event: STARTED ref:xuhreb4ps59uaqu1chxrnkz93
event: STARTED ref:i6wz2oh4svr9scd0fpftzu2du
event: STARTED ref:8hkkvs1m7c04k434o5so7upnz
event: STARTED ref:otsgs6qh107v2le2end8pv2lg
event: STARTED ref:wiqgncm75wohk1z41eliadwyw
event: STARTED ref:0nyi2luhbt0vzls60v3xjzjr2
event: STARTED ref:nlatcq6e2fqickjwwl0bx0u5q
event: STARTED ref:n0jeear0ts54z3b10nzlcow2k
event: STARTED ref:hi1qoq61m2ipffzq8vw5pfv4n
event: STARTED ref:mwvnjz3el0qk42jnv86pp0cd1
event: STARTED ref:wk2rrebn57s8bsx09ugribkb4
event: STARTED ref:0qi0yr2qewavv8f3c5t93hrlk
event: STARTED ref:sla2tq0lt77jeakufdl64u7ha
event: STARTED ref:p1kakvy9g59ojk9hyzw8qq3v7
event: STARTED ref:zej86r8fsny9xda9z7k6rmc1s
event: STARTED ref:5r2slvfxfqrx7t2nh0siqvybf
event: STARTED ref:k2ce2p6917maxec7t8n67o23j
event: STARTED ref:qeyvfi6eatcna84i5uxb01fg9
event: STARTED ref:vgtjxcssnxbers2jwxi385uyf
event: STARTED ref:eduo06woktcu1nqfoxdnl9vfy
event: STARTED ref:jcksy7y9qaljt1pvneds817du
event: STARTED ref:tm5u6704kr0mugzc8yjhly9or
event: STARTED ref:izic9rdn2nhajaji2d9dyq63h
event: STARTED ref:bpdw37254eii5nfmhgg2fuaxd
event: STARTED ref:gjc0agbsaqhy4wouhl7huy7ws
event: STARTED ref:l6ywwu6lguef6lmkcikqtnst3
event: STARTED ref:uizco9ypr1eoftris4xzoi7ib
event: STARTED ref:kcur4zivjxwe3r70vz01i406x
event: STARTED ref:yccx0ka8nznyi8obabxe1cjxn
event: STARTED ref:gq7viw08sejyvd2beq8qced11

And when I check the logs, those pending builds are all at the stage of writing cache manifests / preparing build cache for export

/ $ buildctl debug logs gq7viw08sejyvd2beq8qced11
[+] Building 0.0s (33/34)
(...)
 => => writing config sha256:58bec20904e4e8bc3e4cc32801bf32ca6a90ba484388bbdd08bd0f28301d4dd8                                                    0.3s
 => => writing cache image manifest sha256:8c2d3299931f864ce40755eb6b6529370238998332e83af91a09064f924d6db3                                      0.1s
 => => preparing build cache for export

So something is happening on the buildkit side as I understand, any way for me to be able to know more about those?

Edit: also those buildkit remotes are using 7 CPU even though nothing of note is currently running (all the above builds are currently deadlocked)

[remidebette]

I am starting to think my bug is linked to
#6131

[remidebette]

Also seeing #6008