Allow parallel blaze invocations

[schroederc]

The time spent waiting for another Bazel command seems wasted. Can Bazel accomplish some work in tandem with the other command since Bazel uses a server daemon?

In particular, can bazel build //some:target and bazel build //some/other:target have their action graphs joined together and each separate command finish when its requested targets are completed?

However, at the very least, can Bazel show the progress of the other command?

[ulfjack]

Doing multiple builds in parallel over possibly slightly different files is - in general - a tricky problem. That said, we're currently not even able to run commands in parallel even if we know that they can't interfere (say bazel help and bazel info).

Even passing information from one command to another isn't currently possible. The client takes an exclusive lock every time you run bazel, and the server doesn't have the ability to handle two connections at the same time.

I've made some progress on updating our core code for this, recently. Check out my changes to BlazeRuntime.java and CommandEnvironment.java. It's not a high priority for me right now, though, so progress will likely be patchy.

[ahumesky]

If you're building the targets with the same configuration though (e.g.,
same flags, same target architecture, etc), you can build both targets with
the same invocation:
bazel build //some:target //some/other:target
or even:
bazel build //some/...:all

and if they have any shared dependencies, bazel will build those only once.
If these are under different configurations, you could use --output_base to
start a separate bazel server for the same workspace, but the two servers
won't share work.

On Mon, Oct 26, 2015 at 4:56 PM Ulf Adams notifications@github.com wrote:

Doing multiple builds in parallel over possibly slightly different files
is - in general - a tricky problem. That said, we're currently not even
able to run commands in parallel even if we know that they can't interfere
(say bazel help and bazel info).

Even passing information from one command to another isn't currently
possible. The client takes an exclusive lock every time you run bazel, and
the server doesn't have the ability to handle two connections at the same
time.

I've made some progress on updating our core code for this, recently.
Check out my changes to BlazeRuntime.java and CommandEnvironment.java. It's
not a high priority for me right now, though, so progress will likely be
patchy.

—
Reply to this email directly or view it on GitHub
#532 (comment).

[jart]

@ulfjack I've been thinking about adding ibazel functionality to rules_closure. The way I'd do it is with fork() on POSIX. That way I can escape the bazel run restriction. I don't need bazel query since Skylark will give me the build graph information. Then in my forked process, I can use java.nio.file.WatchEvent (see code) to farm out bazel build //foo commands, assuming what I did originally was bazel run //foo.

[ulfjack]

bazel run now has a --direct_run option that releases the lock before running the tool (and also allows the tool direct access to the terminal), so you don't need to work around that specifically anymore. We're working on enabling that by default for all bazel run invocations, but are seeing issues due to minor changes in semantics. @lberki

[lberki]

Actually, I think we can just flip the default in Bazel. In the worst case, we can force it to be turned off internally. Let me send a change...

[jart]

@ulfjack That's the best news ever. Just added run --direct_run to my ~/.bazelrc.

[jart]

@ulfjack While we're on the topic, maybe consider making --distinct_host_configuration=false the default for Bazel. It makes protobuf take half as long to build. The TensorBoard team has found this very helpful for our Travis CI builds.

[jgarvin]

With bazel 3.2.0 I get a message saying --direct_run is deprecated and independent commands still block each other.

[lberki]

--direct_run is deprecated in the sense that now --nodirect_run is not an option anymore, i.e. it's always on and you thus don't need the command line option anymore.

It only makes it possible to run the binary run by bazel run and another Bazel command at the same time; two Bazel commands at the same time still won't work. IOW, if you run bazel run and another command at the same time, the second command will still have to wait until the binary is built and only then can it execute.

[ShreeM01]

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.

[cpsauer]

@kshyanashree, to me this seems interesting enough--certainly long term--to be worth keeping open!

[guw]

Does it make sense to create a new issue for allowing multiple bazel query operations in parallel? Would that be easier to implement?

[lberki]

Unfortunately, no. bazel query is somewhat simpler since it doesn't need the build machinery, but the first few hurdles (BlazeModule life cycle and multiple Skyframe requests) are common between the two. So I think it's best to keep it as a single issue.