Run builds or commands in parallel?

[jasongwartz]

Which mbt command are you having troubles with?
mbt build, and/or custom commands

What's the current behaviour?
the builds or commands of each module run sequentially

What's the expected behaviour?
I'm wondering if the possibility of running the various builds or commands in parallel has been discussed?

It seems like it would be possible to spin off into a goroutine somewhere like here:

mbt/lib/build.go

Line 146 in acd0ae5

err := s.execBuild(cmd, m, a, options)

My team would be happy to attempt an implementation, if the maintainers are open to it. We ended up using a workaround (see below), so the implementation of this is still up for grabs.

Cheers!

[buddhike]

@jasongwartz Sorry about the lag in my response.
This has been on our radar for a while but didn't make it to the top of the list. We would be really happy for you to contribute to this feature.

The location you are pointing would be the right place to fork different build jobs. What you need to be careful about is the dependency order. Happy to keep an eye on a provisional PR 😉

Thank you so much again!

[fllaca]

hi! I would be very interested on this feature as well, it would boost build times a lot! @jasongwartz did you have the chance to make any progress on it?

[DoHe]

@fllaca we ended up using mbt just to generate the list of dependencies and are running the actual commands separately with gnu parallel

[jasongwartz]

@fllaca Yep, @DoHe is on my team so he's correct - we ended up using a different technique, where we just have mbt print the changed modules, and then have a script run custom build commands on each. This works for us because we don't have any (or very few) "nested dependencies", I guess you could say? What I mean is, we don't have any builds that depend on the output of a previous build - so dependency order wasn't actually a problem for us. At the moment, our team doesn't have a plan to build this feature (as we found a workaround), so I'll edit the original comment accordingly.

[fllaca]

Thanks @jasongwartz @DoHe! I think your solution can work for me too, but anyway maybe I give a try to the implementation of this feature. Apart from the dependencies order, I'm not sure also how to handle the output of the build commands running in parallel, they would be mixed in standard output and difficult to read. Any idea about that @BuddySpike ?

[DoHe]

@fllaca we faced the same issue with our solution. What we opted for was printing all output as it happens (i.e. mixed) with a prefix including timestamp and the command that produced that line. We also added the option to additionally reprint all output in order, split by command, after all commands are finished.

[buddhike]

@jasongwartz & @DoHe Thanks for sharing your experience here. That's a great story 👍.
@fllaca I think it makes sense to prefix each line with the module name.