Intermediate build results (phase A and B)

[matglas]

I am trying setup a structure that allows use to do the following.

Phase A build tool-A, tool-B and tool-C. This creates an output on tool-C.

---
title: Phase A
---
flowchart LR

subgraph tool_a ["//tool-A"]
    tool_a__build[":build"]
    tool_a__source[":source"]
end

subgraph tool_b ["//tool-B"]
    tool_b__build[":build"]
    tool_b__source[":source"]
end

subgraph tool_c ["//tool-C"]
    tool_c__build[":build"]
end

tool_a__build ---> tool_a__source
tool_b__build ---> tool_b__source
tool_b__build ---> tool_a__build
tool_c__build ---> tool_a__build
tool_c__build ---> tool_b__build

Phase B should build tool-D and use the output of build-C but it should not check the whole dependency tree for changes.

---
title: Phase B
---
flowchart LR

subgraph tool_c ["//tool-C"]
    tool_c__build[":build"]
end

subgraph tool_d ["//tool-D"]
    tool_d__build[":build"]
    tool_d__source[":source"]
end

tool_d__build ---> tool_c__build
tool_d__build ---> tool_d__source

Context

The reason for this is that I have one part being build on a Linux box (Phase A) and then I want to use that result on a MacOS box (Phase B) to build additional output.

It would be great if this could be done 'natively'. But from my understanding till now I can only have it work if I copy the artifact over to the new machine and use it as an 'src' from the repo. This feels incorrect.

Am I missing something or is this not possible?

[peterebden]

If I understand correctly, I think you could handle this by selecting tool-C differently based on the architecture:

if CONFIG.OS == 'darwin':
    filegroup(
        name = "tool-C",
        srcs = ["tool-C-bin"],
    )
else:
    go_binary(  # or whatever
        name = "tool-C",
        ....
    )

[matglas]

This does mean that tool-C-bin has to be put inside the tool-C folder as a file output. And it can not be consumed by transferring the plz-out/gen/tool-C folder to Phase B. Correct?

[matglas]

If it would be possible to use the cached results on the MacOS machine in Phase B that would be even better in our setup. But when I try to run tool-D on MacOS with the build using tool-C dependencies it would have different hash key because of different build config like the OS variable. So it become unusable.

I would rather keep it as one chain. Because actually we have Phase C too where results are combined again with dependencies from Phase A and B.

[Tatskaari]

I'm a little confused so forgive me if I've misunderstood. If you need to build a tool on linux and then use it on a macos box, then if you're using something like github actions you could do the following:

Set up a config option for the tool. This can be done in your .plzconfig like so:

    [buildconfig]
    my-tool = //tools:linux_tool

and then use this in your build rules like so:

genrule(
    ...,
    tools = [CONFIG.MY_TOOL],
)

Then you can:

1. build the tool on linux
2. save this as an artefact to your ci build e.g. to /tmp/workspace/my-tool
3. set a .plzconfig.macos_ci (or use -o buildenv.mytool:/tmp/workspace/my-tool) that configures the tool path
4. do your macos build

.plzconfig.macos_ci might look like:

    [buildconfig]
    my-tool = /tmp/workspace/my-tool

[matglas]

That is a great way indeed. I had not thought of that. Currently I use rules that uses the link label to output it to the folder of the package. Something like this will be found in tool-A and tool-B on the build rule.

labels = ["link:" + package_name() ]

Then I use an additional rule in tool-A and tool-B that uses that output path as a source.
And tool-C can use those rules as its source on the other machine.

But one thing that I was not able to do is to use the data that is already in cache. I need to copy it over in the workspace either way.

[matglas]

I am closing this issue now as I got enough input to make things work.