Build concurrency with bazel builds

[robertgates55]

Am interested in guidance on how best to use skaffold to run our CI builds. We're moving to using bazel for our artifact builds, and I was interested in how concurrency would be handled by the bazel server. It appears skaffold invokes multiple bazel builds, and bazel server only allows these to be run one at a time. Eg if I set concurrency: 4 and run skaffold build with 4 bazel artifacts, I get

Building 4 artifacts in parallel
Building [blah]...
Another command holds the client lock: 
pid=35039
owner=client
cwd=/Users/robgates/Repositories/blah

Waiting for it to complete...
Another command holds the client lock: 
pid=35026
owner=client
cwd=/Users/robgates/Repositories/blah

Waiting for it to complete...
Another command (pid=35039) is running. Waiting for it to complete on the server (server_pid=10136)..

... bazel waits until each build completes before allowing the next to start.

Is there a better way of setting this up? Is there any way for skaffold to invoke bazel build //src/artifact1:artifact1.tar //src/artifact2:artifact2.tar //src/artifact3:artifact3.tar //src/artifact4:artifact4.tar such that bazel handles the concurrency? (obviously I know with multiple providers they'll all behave differently so I understand the complexity; but am just wondering what the right way to handle this scenario is?

[tejal29]

@robertgates55 unfortunately like you mentioned concurrent bazel builds wont work with current implementation.

It will take some effort to re-think how we can support concurrent builds for bazel. Right now the team does not have the bandwidth to look into this.

I like the idea of bazel build //src/artifact1:artifact1.tar //src/artifact2:artifact2.tar //src/artifact3:artifact3.tar //src/artifact4:artifact4.tar Would you be open to write a design proposal and consider more aspects like how skaffold will use the concurrency config ? Does bazel support number of concurrent builds on the command line which we can pass as is?

We can definitely help you make the code changes.