-
Notifications
You must be signed in to change notification settings - Fork 210
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to use multiple execution platforms #612
Comments
Correct.
There is no general way to do that, but I'll explain how to solve your problem before I explain why. You have a couple options and the best one depends on the particular situation. For this one, you have a An equally common case is that someone has a build tool like a compiler that only supports running on a particular platform. This comes up particularly for things that don't support cross-compilation, where you end up with something like "I have to choose between three different compilers based on target platform, and all of them can only run on a specific execution platform." In that case, the preferred thing to do is to say Now getting back to why there's no way to express these restrictions on the platform definition: Note that in all of the above cases, the restrictions that we told buck about only apply to a specific target or a family of targets that all share some exec dep. This acknowledges the general principle that "which platform can this be built on" is a property that is local to a specific build step, and does not apply universally across a build, a target platform, or the repo. As a result, buck is in general happy to cross-compile only some parts of your build and not others. Coming from other build systems that might be surprising, but it is actually a good thing. It can help you make efficient use of resources by compiling things locally or using a cheaper remote execution platform (ie not mac). It also allows you to do things like have a single build which packages one component that must be built on Mac and one component that must be built on Windows. |
Summary: As mentioned in #612, I have a C++ project that can be built for either Linux or Windows, without any cross-compilation so Linux and Windows are also the two execution platforms. Since there is remote execution, I wanted to be able to build for Linux from my Windows machine using RE, and vice versa. From the response I got in the above issue, I need to add an `exec_dep` to the toolchain, pointing a compiler target, that then has a `target_compatible_with` with the execution platform the compiler can run on. I tried to add the `target_compatible_with` directly to the toolchain but it didn't work, probably intentional. So I moved some parts of the current system_cxx_toolchain to a separate `NativeCompiler` provider that is then returned by some rules defining a compiler. This compiler target then has the `target_compatible_with` attribute. Can this be merged or should I just have this toolchain in my own repository? It seems useful for everyone, the only disadvantage I can see being that it is not fully backwards compatible. Hopefully I didn't completely misunderstand how execution platforms are supposed to be used. I improved the Visual Studio example to use this new updated toolchain to easily support remote execution. ## Manual testing I used the Visual Studio example and setup my RE workers with Buildbarn. Unfortunately Buildbarn's Windows workers don't currently work with the linking step, I haven't investigated why. The build targeting Windows made from a Windows host works because the linking is done locally but the one done from a Linux host fails. ### Local builds #### Windows ``` buck2 build --target-platforms //buck2_utils/platforms:windows_debug :main ``` Passes #### Linux ``` buck2 build --target-platforms //buck2_utils/platforms:linux_debug :main ``` Passes ### Remote builds #### From Windows ##### Target Windows ``` buck2 build --no-remote-cache -c buck2_re_client.enabled=true --target-platforms //buck2_utils/platforms:windows_debug :main ``` Passes ##### Target Linux ``` buck2 build --no-remote-cache -c buck2_re_client.enabled=true --target-platforms //buck2_utils/platforms:linux_debug :main ``` Passes #### From Linux ##### Target Windows ``` buck2 build --no-remote-cache -c buck2_re_client.enabled=true --target-platforms //buck2_utils/platforms:windows_debug :main ``` Linking fails (The system cannot open the device or file specified.) ##### Target Linux ``` buck2 build --no-remote-cache -c buck2_re_client.enabled=true --target-platforms //buck2_utils/platforms:linux_debug :main ``` Passes Pull Request resolved: #629 Reviewed By: JakobDegen Differential Revision: D58667378 Pulled By: KapJI fbshipit-source-id: 3934dd105900d094de71690a7097d7c3909e8cf9
Summary: As mentioned in facebook/buck2#612, I have a C++ project that can be built for either Linux or Windows, without any cross-compilation so Linux and Windows are also the two execution platforms. Since there is remote execution, I wanted to be able to build for Linux from my Windows machine using RE, and vice versa. From the response I got in the above issue, I need to add an `exec_dep` to the toolchain, pointing a compiler target, that then has a `target_compatible_with` with the execution platform the compiler can run on. I tried to add the `target_compatible_with` directly to the toolchain but it didn't work, probably intentional. So I moved some parts of the current system_cxx_toolchain to a separate `NativeCompiler` provider that is then returned by some rules defining a compiler. This compiler target then has the `target_compatible_with` attribute. Can this be merged or should I just have this toolchain in my own repository? It seems useful for everyone, the only disadvantage I can see being that it is not fully backwards compatible. Hopefully I didn't completely misunderstand how execution platforms are supposed to be used. I improved the Visual Studio example to use this new updated toolchain to easily support remote execution. ## Manual testing I used the Visual Studio example and setup my RE workers with Buildbarn. Unfortunately Buildbarn's Windows workers don't currently work with the linking step, I haven't investigated why. The build targeting Windows made from a Windows host works because the linking is done locally but the one done from a Linux host fails. ### Local builds #### Windows ``` buck2 build --target-platforms //buck2_utils/platforms:windows_debug :main ``` Passes #### Linux ``` buck2 build --target-platforms //buck2_utils/platforms:linux_debug :main ``` Passes ### Remote builds #### From Windows ##### Target Windows ``` buck2 build --no-remote-cache -c buck2_re_client.enabled=true --target-platforms //buck2_utils/platforms:windows_debug :main ``` Passes ##### Target Linux ``` buck2 build --no-remote-cache -c buck2_re_client.enabled=true --target-platforms //buck2_utils/platforms:linux_debug :main ``` Passes #### From Linux ##### Target Windows ``` buck2 build --no-remote-cache -c buck2_re_client.enabled=true --target-platforms //buck2_utils/platforms:windows_debug :main ``` Linking fails (The system cannot open the device or file specified.) ##### Target Linux ``` buck2 build --no-remote-cache -c buck2_re_client.enabled=true --target-platforms //buck2_utils/platforms:linux_debug :main ``` Passes X-link: facebook/buck2#629 Reviewed By: JakobDegen Differential Revision: D58667378 Pulled By: KapJI fbshipit-source-id: 3934dd105900d094de71690a7097d7c3909e8cf9
I have a project that can be built for either Linux or Windows, without any cross-compilation so Linux and Windows are also the two execution platforms. Since there is remote execution, I wanted to be able to build for Linux from my Windows machine using RE.
To keep this short I created a sample project: main...Overhatted:buck2-strip-prefix:multiple_target_platforms
It takes inspiration from the following issue: #487
The project just runs some Linux commands which obviously doesn't make sense to do on Windows but it's just a sample for me to understand the execution platform resolution.
When running "buck2 build --target-platforms root//platforms:linux tests:tests" on Windows, I wanted it to select the Linux execution platform automatically (since it's the only one with the "config//os/constraints:linux" constraint) but it tries to run it on the Windows platform, presumably because it is the first one declared, so I don't think I'm using the constraints correctly.
To reproduce the project above:
buck2 init --git
,cd prelude
and thengit reset --hard 3b5706b425a88a8641aad8d8ffcc6cdf441b9db0
.Running the command, "buck2 build --target-platforms root//platforms:linux tests:tests", I wanted the genrule commands to be ran on Linux but instead they are ran on Windows, demonstrated by the error message I receive: 'Failed to start process: exec: "sh": executable file not found in %PATH%'
How can I add constraints to the execution platform so that Buck2 automatically selects the correct execution platform for the selected target platform?
The text was updated successfully, but these errors were encountered: