Host Tool Scripting

[beeselmane]

Hi, I was looking over how the host toolchain is handled before the main CMake build, and it seems like it would be better to write a shell script that can handle building the host toolchain and invoking CMake with the proper environment variables instead of setting things like the compiler or linker through CMake after building them. I can write such a script, but I wanted to gauge what other people were thinking in this regard. I was thinking something like setup.sh in the root directory, and sourcing it would basically setup the build environment so that running a given command (darwin or something) would go in and build everything in the host tools directory before invoking cmake in the root directory with the proper environment variables set (specifically, changing $PATH to search the built host tools dir, selecting linker, compiler from the host tools, etc). Effectively, we can make the tools/ sub directory its own CMake root, and have everything controlled via a shell script (this is similar to how things are done in other OS projects, see Google's Fuschia or Android build systems as an example, both require the user to first source a script and then run certain commands to actually kick off a build).

If everyone else looking at this agrees, I can create a full system building toolchain including clang, ld64, mig, mtoc (for building UEFI bootloader), etc. by pulling together everything in the tools/ directory, and configure the CMake system to use the target toolchain. Let me know what everyone thinks of the idea, and if people agree, I'll set to it.

The workflow I was thinking would basically be something like:

source setup.sh
darwin build (darwin would be a shell script installed into the toolchain by setup.sh)

darwin build would then invoke something like

• cmake -S ${DARWIN_ROOT}/tools -B ${DARWIN_BUILDROOT}/host -DCMAKE_INSTALL_PREFIX=${DARWIN_TOOLCHAIN}
• make -C ${DARWIN_BUILDROOT}/host install
• ``
• env -i PATH=${DARWIN_TOOLCHAIN}/usr/bin:${PATH} ${DARWIN_ROOT}=${DARWIN_ROOT} ... cmake -S ${DARWIN_ROOT} -B ${DARWIN_BUILDROOT} -DCMAKE_INSTALL_PREFIX=${DARWIN_PRODUCT_ROOT}
• make -C ${DARWIN_BUILDROOT} install
  ...
• Any packaging steps to create a bootable image would then follow here

The other (somewhat related) suggestion I would make is to move to the ninja generator for CMake, this seems to be what all the cool kids are doing (and this can be pulled in as a git submodule in the host toolchain, so users don't have to have ninja preinstalled)

[wjk]

Brilliant! I hadn’t yet figured out how to handle building submodules in GitHub Actions independently of the entire OS; your proposal addresses that situation in a very elegant manner. Feel free to start work on it. All I ask is that you send your PRs to PureDarwin/dbuild, so the build support code can maintained be separately from the top-level build here. (On a related note, I always use Ninja with CMake because it‘s so much faster. As such, ninja has always Worked On My Machine™. Is there a situation for you where ninja didn’t work, and you had to fall back to using makefiles?) Thanks!

[beeselmane]

Brilliant! I hadn’t yet figured out how to handle building submodules in GitHub Actions independently of the entire OS; your proposal addresses that situation in a very elegant manner. Feel free to start work on it. All I ask is that you send your PRs to PureDarwin/dbuild, so the build support code can maintained be separately from the top-level build here. (On a related note, I always use Ninja with CMake because it‘s so much faster. As such, ninja has always Worked On My Machine™. Is there a situation for you where ninja didn’t work, and you had to fall back to using makefiles?) Thanks!

Alright, sounds good, I've been busy recently, but I should start up working on this in a few days, I already have a skeleton I wrote for a personal project of mine I can pull in here within minor modifications.

Ninja works fine for me, it's just that not everyone will have it installed by default (vs make which ships with everything by default). We can always just add it into the host toolchain if we progress in the way I suggest here, so we'd be building a known version that can be used to build the system without issue. Otherwise, I'm fine with just adding it as a prerequisite. (since we would no longer be invoking CMake builds directly, the generator selection becomes more important, since most people will just want sane defaults to come from the build script)