Actions fetched from remote cache / executed remotely should not count towards parallelism limit

[ob]

When a fairly large application is built using the remote cache, and all the actions are fully cached. Bazel still keeps the parallelism set to the number of cores in the machine. However, most actions are just waiting for network I/O.

With the default settings on an 8-core machine, I get:

$ time bazel build //...
## output elided
INFO: Elapsed time: 97.567s, Critical Path: 26.44s
INFO: 2000 processes: 2000 remote cache hit.
INFO: Build completed successfully, 2240 total actions

real	1m38.518s
user	0m0.051s
sys	0m0.061s

But if I bump up the number of jobs to a crazy number:

$ time bazel build --jobs 2000 //...
## output elided
INFO: Elapsed time: 39.535s, Critical Path: 31.33s
INFO: 2000 processes: 2000 remote cache hit.
INFO: Build completed successfully, 2240 total actions

real	0m40.483s
user	0m0.048s
sys	0m0.058s

I think the --jobs option should only apply to local actions.

[buchgr]

Thanks @ob! A colleague of mine has a prototype of this, however I still expect this to be at least 3-6 months away to land in a released Bazel version!

[philwo]

FWIW, we "fix" this in Google by actually running with --jobs=200 (or higher). Local actions are still limited by local resources, so this is fine in general.

[buchgr]

@philwo @ulfjack has a prototype patch to no longer be limited by --jobs for remote execution.

[ittaiz]

@philwo

Local actions are still limited by local resources, so this is fine in general.

Means you run with jobs together with local_resources?

[buchgr]

@ulfjack what's the status of your work in this area? is there a tracking bug on github?

[ulfjack]

Parts of the prototype have been submitted, but I haven't even sent out some critical parts. Making it work also requires rewriting the RemoteSpawnRunner to be async and use ListenableFuture, for which I do not currently have plans. I am not aware of a tracking bug on GitHub apart from this one.

[buchgr]

Making it work also requires rewriting the RemoteSpawnRunner to be async and use ListenableFuture

That I'd be happy to take over :-)

[ulfjack]

Happy for you to start working on the RSR in parallel to me landing the Skyframe changes that are also required.

[buchgr]

Broke this out into #7182.

[ulfjack]

Commit 9beabe0 is related.

[ulfjack]

Also related:
47d29eb
57f2f58
14f8b10 rolled back as 68fc46b due to increased memory consumption

[ianoc-stripe]

It does for us, i'm not sure if others don't see it. but for tests + binaries across a repo. If you pull and get a lot of cache hits having jobs set to say 1000, means the machine locks up more or less with the unpacking of runfiles. So having the runfiles step included in local cpu usage would be great as a means to avoid that. (Now we tell folks to ctrl+c and try run with --jobs 10 when they see it which isn't ideal).

So I would love if you could change it, thank you

[benjaminp]

45a9bc2 was a change to the resource requirements of runfiles trees, which allowed more parallelism. Probably this should be discussed in another issue, though.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[meisterT]

We are working on this again, in a different way. Next step is to upgrade to a modem JDK which will allow us to use loom/virtual threads.

[Ryang20718]

Reading through this thread.

If I'm using local execution with a remote cache and almost all tests are cached. with --jobs set to 4, I see. (currently on bazel 6.0.0)

21s remote-cache, linux-sandbox ... (8 actions running)
29s remote-cache, linux-sandbox ... (8 actions running)

With jobs set to 8, I see a max of 16 actions. This would mean jobs fetching from remote cache is not counting towards parallelism. Is there a way we can configure it to do so?