Distributed Builds using BuildXL

[mikejack07]

I am an intern responsible for evaluating the distributed build systems available for Windows and Linux.

I have few limitation which I need to consider before evaluating them

Should be Open Source/ Licensed (less preferably)
Supports both Win and Linux
Support CMake

I came across BuildXL while exploring and found that it is also a build distribution tool.
But I couldn't find any documentation which describes how to set up and use build distribution across machines connected to the network.

Is this feature only available to the Microsoft developers? Or can we set up and use build distribution as well? If yes, what are the steps that I should follow?

Thanks and regards

P.S.: I found this documentation https://github.com/microsoft/BuildXL/blob/master/Documentation/Wiki/Distributed-Builds.md but it only describes the components and its function not the setup and how to run a build using distribution.

[relascope]

hi mikejack07,

I am actually performing the same search. (again).

What I know:

• Incredibuild: worked fine and stable in 2008 (worked for a very big c++ solution with about 100 projects back then); it is commercial and they haven't answered my pricing question for about two weeks.
• distcc: setup is a bit complicated, but when it runs. it runs. Supoorts everything, that's built with GCC compiler family - you don't have to worry, if you are using make, CMake, Ninja, waf....
• FastBuild: just recently found, not used in production. seems not that mature.

Anyway, to answer your question, I had to search a bit....

distributedBuildRole commandline option will help you.
specify 2 on every worker (machines which should help build)
and to 1 or 3 (???) for initiating the actual build.

https://github.com/microsoft/BuildXL/blob/main/Public/Src/Utilities/Configuration/DistributedBuildRoles.cs

public enum DistributedBuildRoles : byte
{
    /// <summary>
    /// Not running distributed build.
    /// </summary>
    None = 0,

    /// <summary>
    /// Specifies that the current node acts as the build coordinator
    /// </summary>
    /// <remarks>
    /// To be replaced with 'Orchestrator'
    /// </remarks>
    Master = 1,

    /// <summary>
    /// Specifies that the current node acts as a build worker
    /// </summary>
    Worker = 2,

    /// <summary>
    /// Specifies that the current node acts as a build coordinator
    /// </summary>
    Orchestrator = 3,
}

used in

https://github.com/microsoft/BuildXL/blob/main/Public/Src/App/Bxl/Args.cs

OptionHandlerFactory.CreateOption2(
"distributedBuildRole",
"dbr",
opt =>
{
// TODO: Ideally we'd automatically turn on distributionConfiguration.ValidateDistribution whenever
// distribution is enabled. This probably won't work with WDG right now though, so it is not on by default
distributionConfiguration.BuildRole = CommandLineUtilities.ParseEnumOption(opt);
}),

sry, didn't check out and have no build (yet) so don't really know how to apply the option value to the command....

Plz keep us informed if you got the distributed build running.

[mpysson]

Distributed builds with BuildXL utilize the cache layer for file transport between machines. The cache service we use internally for distributed builds is not publicly available. The main obstacle for you to set up distributed builds would be to implement a cache for your specific environment. It needs to ensure consistent access to content and metadata across all machines participating in a build session.

You can check out ICache.cs for the interface you'd need to implement. You can also browse the implementations of that interface that are in the codebase. But be aware that the implementations in this codebase are not the production system. We may also evolve that ICache interface over time in order to optimize the reliability and efficiency of that component.

Check out Args.cs and Strings.resx for details on the distributed related args you'll need:

• distributedBuildRole
• distributedBuildServicePort
• distributedBuildWorker