Limit concurrent build requests on a single BuildKit pod

[KoshinHagimoto]

Description

Summary

It would be helpful to have a mechanism to limit the total number of build requests a single BuildKit pod processes concurrently. Currently, we rely on BuildKit’s max_parallelism setting to limit internal solver parallelism, but that does not restrict the number of builds the pod accepts at the same time.

Motivation

We are running our CI with GitHub Actions and deploying runner pods using the GitHub Actions runner controller on our Kubernetes cluster. Our BuildKit pod is deployed as a Deployment with an exposed Service, and runner pods use the remote driver by running the buildx create command (over TCP) to connect to the BuildKit pod. In our current setup, many runner pods can send concurrent build requests to the BuildKit pod, which could lead to resource exhaustion (e.g. an OOM scenario). We would like to see if there is a way to limit the number of builds being processed concurrently at the BuildKit instance level.

Use Case

• A large-scale CI environment where numerous build jobs are launched in parallel, each triggering a buildx command to connect to a shared BuildKit pod.
• Preventing one misbehaving or resource-intensive build from overwhelming the BuildKit pod and impacting other concurrent build processes.
• Ensuring a stable build system in scenarios where the pod has limited resources by strictly controlling the number of concurrent build requests.

Proposal

One approach mentioned in the docs is to set the max_parallelism in buildkitd.toml – this setting limits the parallelism within the BuildKit solver (i.e., the number of tasks that can run in parallel within a single build) and is especially useful for low-powered machines. However, it does not impose a global cap on how many builds can be executed in parallel when multiple build requests are received.

I propose that we either:

• Introduce a configuration option to set an overall concurrency limit on the BuildKit server so that when the limit is reached, new build requests are either queued or rejected until an ongoing build completes.
• Alternatively, provide guidance or a recommended pattern (perhaps via external queueing or integration with Kubernetes job controllers) on how best to control this behavior in a multi-tenant CI environment where many remote build requests are directed to a single BuildKit pod.

I would appreciate feedback on whether this is something that could be built into BuildKit itself, or if there is an existing solution or workaround to strictly limit concurrent builds at the BuildKit instance level.

[tonistiigi]

What is using resources for the parallel builds if you have already reached max_parallelism limit?

Limiting build steps is correct as a single build request can use infinite parallelism internally.