Add repository rules for fetching prebuilt build tools (cmake, ninja, etc...)

[UebelAndre]

Currently users can either use there's a set of tools toolchains that either rely on a pre-installed build tool or compile one from source. In general, it's not always convenient to expect one of these tools to be installed on the host and it can be very time consuming to have to compile these tools. In addition to the following

rules_foreign_cc/tools/build_defs/native_tools/BUILD

Lines 7 to 62 in 14520d2

	native_tool_toolchain(
	name = "preinstalled_make",
	path = "make",
	visibility = ["//visibility:public"],
	)
	make_tool(
	name = "make_tool",
	make_srcs = "@make//:all",
	tags = ["manual"],
	)
	native_tool_toolchain(
	name = "built_make",
	path = "make/bin/make",
	target = ":make_tool",
	visibility = ["//visibility:public"],
	)
	native_tool_toolchain(
	name = "preinstalled_cmake",
	path = "cmake",
	visibility = ["//visibility:public"],
	)
	cmake_tool(
	name = "cmake_tool",
	cmake_srcs = "@cmake//:all",
	tags = ["manual"],
	)
	native_tool_toolchain(
	name = "built_cmake",
	path = "cmake/bin/cmake",
	target = ":cmake_tool",
	visibility = ["//visibility:public"],
	)
	native_tool_toolchain(
	name = "preinstalled_ninja",
	path = "ninja",
	visibility = ["//visibility:public"],
	)
	ninja_tool(
	name = "ninja_tool",
	ninja_srcs = "@ninja_build//:all",
	tags = ["manual"],
	)
	native_tool_toolchain(
	name = "built_ninja",
	path = "ninja/ninja",
	target = ":ninja_tool",
	visibility = ["//visibility:public"],
	)

A new rule should be added that will download cmake and ninja for the current platform and make it available for users to register. A similar example of what I'm hoping for can be seen in @rules_rust//rust:repositories.bzl

[attilaolah]

It is true that compiling these from source, while seems like a good idea, can take forever.

Not to mention the rabbit hole it produces: compiling make requires autotools, m4 and whatnot. Then building those too from source can result in a bootstrapping problem…

I could give this a try, but we first need to find out if these tools provide binaries for the major platforms. (I don't want to be in the business of having to maintain a binary release…)

[UebelAndre]

My thought for this change was to write a basic shells script that generates a bzl file with repository definitions for a predefined set of cmake and ninja versions. Then update the interface for rules_foreign_cc_dependencies to be something like

def rules_foreign_cc_dependencies(
        native_tools_toolchains = [],
        register_default_tools = True,
        cmake_version = "3.18.0",
        ninja_version = "1.10.2",
        additional_shell_toolchain_mappings = [],
        additional_shell_toolchain_package = None):

Where users can pick from our built in set of supported releases that will be downloaded from cmake/github/google with all the appropriate sha256 values already defined.

Specifically, something like @rules_rust//util:fetch_shas.sh

Though I'm not sure what you mean with

I don't want to be in the business of having to maintain a binary release…

That said though, I haven't had the time to work on this but think it'd help immensely in making builds more hermetic. If you wanted to work on this I'd greatly appreciate it! 😄

[attilaolah]

Though I'm not sure what you mean with

What I meant there is that this would work as long as cmake/ninja/etc. provide binaries somewhere, on GitHub or some place else. I.e. we don't need to build and release binaries.

[UebelAndre]

What I meant there is that this would work as long as cmake/ninja/etc. provide binaries somewhere, on GitHub or some place else. I.e. we don't need to build and release binaries.

Ah, 100% agree. If this were not the case, I'd say the "preinstalled" and compiled toolchains are as good as it's going to get.

[UebelAndre]

As mentioned in #506, CMake is no longer installed on MacOS test runners for CI. This change would solve for this issue. I feel this is higher priority now since other users may run into this issue. That PR has a work-around but hopefully one of us can find time to implement this feature.

[jheaff1]

It is true that compiling these from source, while seems like a good idea, can take forever.

Not to mention the rabbit hole it produces: compiling make requires autotools, m4 and whatnot. Then building those too from source can result in a bootstrapping problem…

I could give this a try, but we first need to find out if these tools provide binaries for the major platforms. (I don't want to be in the business of having to maintain a binary release…)

It also seems that m4 requires make to build, so the cyclic dependency of make -> m4 -> make would be a challenge to overcome. It’s a shame, as for full hermeticity all dependencies are built from source and cached for later use or prebuilt binaries are fetched and cached.

[attilaolah]

I think what we have for now, with the fetched pre-built binaries, works nicely.

For those cyclic dependencies, they tend to include some bootstrap script or similar, i.e. building a bare-bones make without m4 might be possible, but again, at this point it seems like too much effort for little benefit. Although I like the idea of including as much as possible in the build system, and then using a remote cache to speed up future builds.

[jheaff1]

I see, thank you for the reply.

FYI - I submitted PR #520, which has been merged, which allows the make_tool rule to be successful on machines that do not have autotools installed