diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 451608026..fdd2f3f6e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -56,6 +56,19 @@ various Cloud CLIs for you: @rules_rust//:rustfmt ``` +## Updating Rust dependencies + +After modifying the corresponding `Cargo.toml` file in either the top level or +one of the crate subdirectories run the following command to rebuild the crate +index: + +``` +CARGO_BAZEL_REPIN=1 bazel sync --only=crate_index +``` + +This updates `Cargo.lock` and `Cargo.Bazel.lock` with the new dependency +information. + ## Conduct Native Link Code of Conduct is available in the diff --git a/README.md b/README.md index b0874b0ad..4eeb1ae44 100644 --- a/README.md +++ b/README.md @@ -2,132 +2,132 @@ [![CI](https://github.com/tracemachina/native-link/workflows/CI/badge.svg)](https://github.com/tracemachina/native-link/actions/workflows/main.yml) -An extremely fast and efficient bazel cache service (CAS) written in rust. +Native link is an extremely (blazingly?) fast and efficient build cache and +remote executor for systems that communicate using the [Remote execution +protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) such as [Bazel](https://bazel.build), [Buck2](https://buck2.build), [Goma](https://chromium.googlesource.com/infra/goma/client/) and +[Reclient](https://github.com/bazelbuild/reclient). -The goals of this project are: -1. Stability - Things should work out of the box as expected -2. Efficiency - Don't waste time on inefficiencies & low resource usage -3. Tested - Components should have plenty of tests & each bug should be regression tested -4. Customers First - Design choices should be optimized for what customers want +Supports Unix based operating systems and Windows. -## Overview +## โ„๏ธ Installing with Nix -Native Link is a project that implements the Bazel's [Remote Execution protocol](https://github.com/bazelbuild/remote-apis) (both CAS/Cache and remote execution portion). +**Installation requirements:** -When properly configured this project will provide extremely fast and efficient build cache for any systems that communicate using the [RBE protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) and/or extremely fast, efficient and low foot-print remote execution capability. +* Nix with [flakes](https://nixos.wiki/wiki/Flakes) enabled -Unix based operating systems and Windows are fully supported. +This build does not require cloning the repository, but you need to provide a +config file, for instance the one at [native-link-config/examples/basic_cas.json](./native-link-config/examples/basic_cas.json). -## TL;DR +The following command builds and runs Native Link in release (optimized) mode: -If you have not updated Rust or Cargo recently, run: - -`rustup update` - -To compile and run the server: ```sh -# Install dependencies needed to compile Native Link with bazel on -# worker machine (which is this machine). -apt install -y gcc g++ lld python3 +nix run github:TraceMachina/native-link ./basic_cas.json +``` -# Install cargo (if needed). -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +For use in production pin the executable to a specific revision: -# --release causes link-time-optmization to be enabled, which can take a while -# to compile, but will result in a much faster binary. -cargo run --release --bin cas -- ./native-link-config/examples/basic_cas.json -``` -In a separate terminal session, run the following command to connect the running server launched above to Bazel or another RBE client: ```sh -bazel test //... \ - --remote_instance_name=main \ - --remote_cache=grpc://127.0.0.1:50051 \ - --remote_executor=grpc://127.0.0.1:50051 \ - --remote_default_exec_properties=cpu_count=1 +nix run github:TraceMachina/native-link/ ./basic_cas.json ``` -This will cause bazel to run the commands through an all-in-one `CAS`, `scheduler` and `worker`. See [here](https://github.com/tracemachina/native-link/tree/master/native-link-config) for configuration documentation and [here](https://github.com/tracemachina/native-link/tree/main/deployment-examples/terraform) for an example of multi-node cloud deployment example. -## Example Deployments -We currently have a few example deployments in [deployment-examples directory](https://github.com/tracemachina/native-link/tree/master/deployment-examples). +## ๐ŸŒฑ Building with Bazel -### Terraform -The [terraform deployment](https://github.com/tracemachina/native-link/tree/master/deployment-examples/terraform) is the currently preferred method as it leverages a lot of cloud resources to make everything much more robust. +**Build requirements:** -The terraform deployment is very easy to setup and configure. This deployment will show off remote execution capabilities and cache capabilities. +* Bazel 6.4.0+ +* A recent C++ toolchain with LLD as linker -## Status +> [!TIP] +> This build supports Nix/direnv which provides Bazel but no C++ toolchain +> (yet). -This project can be considered ~stable~ and is currently used in production systems. Future API changes will be kept to a minimum. +The following commands place an executable in `./bazel-bin/cas/cas` and start +the service: -## Build Requirements -We support building with Bazel or Cargo. Cargo **might** produce faster binaries because LTO (Link Time Optimization) is enabled for release versions, where Bazel currently does not support LTO for rust. +```sh +# Unoptimized development build on Unix +bazel run cas -- ./native-link-config/examples/basic_cas.json -### Bazel requirements -* Bazel 6.4.0+ -* gcc -* g++ -* lld -* python3 +# Optimized release build on Unix +bazel run -c opt cas -- ./native-link-config/examples/basic_cas.json -#### Bazel building for deployment -```sh -# On Unix -bazel build cas +# Unoptimized development build on Windows +bazel run --config=windows cas -- ./native-link-config/examples/basic_cas.json -# On Windows -bazel build --config=windows cas +# Optimized release build on Windows +bazel run --config=windows -c opt cas -- ./native-link-config/examples/basic_cas.json ``` -#### Bazel building for release -```sh -# On Unix -bazel build -c opt cas -# On Windows -bazel build --config=windows -c opt cas -``` -> **Note** -> Failing to use the `-c opt` flag will result in a very slow binary (~10x slower). +## ๐Ÿฆ€ Building with Cargo -These will place an executable in `./bazel-bin/cas/cas` that will start the service. +**Build requirements:** -### Cargo requirements * Cargo 1.73.0+ -#### Cargo building for deployment -```sh -cargo build +* A recent C++ toolchain with LLD as linker + +> [!TIP] +> This build supports Nix/direnv which provides Cargo but no C++ +> toolchain/stdenv (yet). + +```bash +# Unoptimized development build +cargo run --bin cas -- ./native-link-config/examples/basic_cas.json + +# Optimized release build +cargo run --release --bin cas -- ./native-link-config/examples/basic_cas.json ``` -#### Cargo building for release + +## ๐Ÿงช Evaluating Native Link + +Once you've built Native Link and have an instance running with the +`basic_cas.json` configuration, launch a separate terminal session and run the +following command to connect the running server launched above to Bazel or +another RBE client: + ```sh -cargo build --release +bazel test //... \ + --remote_instance_name=main \ + --remote_cache=grpc://127.0.0.1:50051 \ + --remote_executor=grpc://127.0.0.1:50051 \ + --remote_default_exec_properties=cpu_count=1 ``` -> **Note** -> Failing to use the `-c opt` flag will result in a very slow binary (~10x slower). -> This is also significantly slower than building without `--release` because link-time-optimization -> is enabled by default with the flag. -### Configure +This causes bazel to run the commands through an all-in-one `CAS`, `scheduler` +and `worker`. -Configuration is done via a JSON file that is passed in as the first parameter to the `cas` program. See [here](https://github.com/tracemachina/native-link/tree/master/native-link-config) for more details and examples. +## โš™๏ธ Configuration -## How to update internal or external rust deps +The `cas` executable reads a JSON file as it's only parameter. See [native-link-config](./native-link-config) +for more details and examples. -In order to update external dependencies `Cargo.toml` is not the source of truth, instead7 these are tracked in `tools/cargo_shared.bzl`. It is done this way so both Bazel and Cargo can use the same dependencies that can be derived from the same source location. +## ๐Ÿš€ Example Deployments -All external dependencies are tracked in a generated `@crate_index` workspace and locked in `Cargo.Bazel.lock`. Some updates to `BUILD` files will require regenerating the `Cargo.toml` files. This is done with the `build_cargo_manifest.py`. +You can find a few example deployments in the [deployment-examples directory](./deployment-examples). -To regenerate the `@crate_index`: -```bash -# This will pin the new dependencies and generate new lock files. -CARGO_BAZEL_REPIN=1 bazel sync --only=crate_index -# This will update the Cargo.toml files with the new dependencies -# weather they are local or external. -python3 ./tools/build_cargo_manifest.py -``` +See the [terraform deployments](./deployment-examples/terraform) for an example +deployments that show off remote execution and cache capabilities. + +## ๐Ÿบ History -## History +This project was first created due to frustration with similar projects not +working or being extremely inefficient. Rust was chosen as the language to write +it in because at the time Rust was going through a revolution in the new-ish +feature `async-await`. This made making multi-threading extremely simple when +paired with a runtime like [tokio](https://github.com/tokio-rs/tokio) while +still giving all the lifetime and other protections that Rust gives. This pretty +much guarantees that we will never have crashes due to race conditions. This +kind of project seemed perfect, since there is so much asynchronous activity +happening and running them on different threads is most preferable. Other +languages like `Go` are good candidates, but other similar projects rely heavily +on channels and mutex locks which are cumbersome and have to be carefully +designed by the developer. Rust doesn't have these issues, since the compiler +will always tell you when the code you are writing might introduce undefined +behavior. The last major reason is because Rust is extremely fast, +/- a few +percent of C++ and has no garbage collection (like C++, but unlike `Java`, `Go`, +or `Typescript`). -This project was first created due to frustration with similar projects not working or being extremely inefficient. Rust was chosen as the language to write it in because at the time rust was going through a revolution in the new-ish feature `async-await`. This made making multi-threading extremely simple when paired with a runtime (like [tokio](https://github.com/tokio-rs/tokio)) while still giving all the lifetime and other protections that Rust gives. This pretty much guarantees that we will never have crashes due to race conditions. This kind of project seemed perfect, since there is so much asynchronous activity happening and running them on different threads is most preferable. Other languages like `Go` are good candidates, but other similar projects rely heavily on channels and mutex locks which are cumbersome and have to be carefully designed by the developer. Rust doesn't have these issues, since the compiler will always tell you when the code you are writing might introduce undefined behavior. The last major reason is because Rust is extremely fast, +/- a few percent of C++ and has no garbage collection (like C++, but unlike `Java`, `Go`, or `Typescript`). +## ๐Ÿ“œ License -# License +Copyright 2020-2023 Trace Machina, Inc. -Software is licensed under the Apache 2.0 License. Copyright 2020-2023 Trace Machina, Inc. +Licensed under the Apache 2.0 License, SPDX identifier `Apache-2.0`. diff --git a/flake.lock b/flake.lock index 1e2537eb4..44aeb2df5 100644 --- a/flake.lock +++ b/flake.lock @@ -1,5 +1,25 @@ { "nodes": { + "crane": { + "inputs": { + "nixpkgs": [ + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1701386725, + "narHash": "sha256-w4aBlMYh9Y8co1V80m5LzEKMijUJ7CBTq209WbqVwUU=", + "owner": "ipetkov", + "repo": "crane", + "rev": "8b9bad9b30bd7a9ed08782e64846b7485f9d0a38", + "type": "github" + }, + "original": { + "owner": "ipetkov", + "repo": "crane", + "type": "github" + } + }, "flake-compat": { "flake": false, "locked": { @@ -74,11 +94,11 @@ }, "nixpkgs": { "locked": { - "lastModified": 1699099776, - "narHash": "sha256-X09iKJ27mGsGambGfkKzqvw5esP1L/Rf8H3u3fCqIiU=", + "lastModified": 1701253981, + "narHash": "sha256-ztaDIyZ7HrTAfEEUt9AtTDNoCYxUdSd6NrRHaYOIxtk=", "owner": "NixOS", "repo": "nixpkgs", - "rev": "85f1ba3e51676fa8cc604a3d863d729026a6b8eb", + "rev": "e92039b55bcd58469325ded85d4f58dd5a4eaf58", "type": "github" }, "original": { @@ -91,11 +111,11 @@ "nixpkgs-lib": { "locked": { "dir": "lib", - "lastModified": 1698611440, - "narHash": "sha256-jPjHjrerhYDy3q9+s5EAsuhyhuknNfowY6yt6pjn9pc=", + "lastModified": 1701253981, + "narHash": "sha256-ztaDIyZ7HrTAfEEUt9AtTDNoCYxUdSd6NrRHaYOIxtk=", "owner": "NixOS", "repo": "nixpkgs", - "rev": "0cbe9f69c234a7700596e943bfae7ef27a31b735", + "rev": "e92039b55bcd58469325ded85d4f58dd5a4eaf58", "type": "github" }, "original": { @@ -122,36 +142,22 @@ "type": "github" } }, - "nixpkgs_2": { - "locked": { - "lastModified": 1689261696, - "narHash": "sha256-LzfUtFs9MQRvIoQ3MfgSuipBVMXslMPH/vZ+nM40LkA=", - "owner": "NixOS", - "repo": "nixpkgs", - "rev": "df1eee2aa65052a18121ed4971081576b25d6b5c", - "type": "github" - }, - "original": { - "owner": "NixOS", - "ref": "nixpkgs-unstable", - "repo": "nixpkgs", - "type": "github" - } - }, "pre-commit-hooks": { "inputs": { "flake-compat": "flake-compat", "flake-utils": "flake-utils", "gitignore": "gitignore", - "nixpkgs": "nixpkgs_2", + "nixpkgs": [ + "nixpkgs" + ], "nixpkgs-stable": "nixpkgs-stable" }, "locked": { - "lastModified": 1700064067, - "narHash": "sha256-1ZWNDzhu8UlVCK7+DUN9dVQfiHX1bv6OQP9VxstY/gs=", + "lastModified": 1700922917, + "narHash": "sha256-ej2fch/T584b5K9sk1UhmZF7W6wEfDHuoUYpFN8dtvM=", "owner": "cachix", "repo": "pre-commit-hooks.nix", - "rev": "e558068cba67b23b4fbc5537173dbb43748a17e8", + "rev": "e5ee5c5f3844550c01d2131096c7271cec5e9b78", "type": "github" }, "original": { @@ -162,6 +168,7 @@ }, "root": { "inputs": { + "crane": "crane", "flake-parts": "flake-parts", "nixpkgs": "nixpkgs", "pre-commit-hooks": "pre-commit-hooks" diff --git a/flake.nix b/flake.nix index 58cd2e254..98e224902 100644 --- a/flake.nix +++ b/flake.nix @@ -3,18 +3,68 @@ inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; - pre-commit-hooks.url = "github:cachix/pre-commit-hooks.nix"; + pre-commit-hooks = { + url = "github:cachix/pre-commit-hooks.nix"; + inputs.nixpkgs.follows = "nixpkgs"; + }; + crane = { + url = "github:ipetkov/crane"; + inputs.nixpkgs.follows = "nixpkgs"; + }; }; - outputs = inputs@{ nixpkgs, flake-parts, ... }: + outputs = inputs @ { flake-parts, crane, ... }: flake-parts.lib.mkFlake { inherit inputs; } { systems = [ "x86_64-linux" ]; imports = [ inputs.pre-commit-hooks.flakeModule ]; perSystem = { config, self', inputs', pkgs, system, ... }: let + customStdenv = import ./tools/llvmStdenv.nix { inherit pkgs; }; + + craneLib = crane.lib.${system}; + + src = pkgs.lib.cleanSourceWith { + src = craneLib.path ./.; + filter = path: type: + (builtins.match "^.+/data/SekienAkashita\\.jpg" path != null) || + (craneLib.filterCargoSources path type); + }; + + commonArgs = { + inherit src; + strictDeps = true; + buildInputs = [ ]; + nativeBuildInputs = [ pkgs.autoPatchelfHook pkgs.cacert ]; + stdenv = customStdenv; + }; + + # Additional target for external dependencies to simplify caching. + cargoArtifacts = craneLib.buildDepsOnly commonArgs; + + native-link = craneLib.buildPackage (commonArgs + // { + inherit cargoArtifacts; + }); + hooks = import ./tools/pre-commit-hooks.nix { inherit pkgs; }; in { + apps = { + default = { + type = "app"; + program = "${native-link}/bin/cas"; + }; + }; + checks = { + # TODO(aaronmondal): Fix the tests. + # tests = craneLib.cargoNextest (commonArgs + # // { + # inherit cargoArtifacts; + # cargoNextestExtraArgs = "--all"; + # partitions = 1; + # partitionType = "count"; + # }); + }; pre-commit.settings = { inherit hooks; }; devShells.default = pkgs.mkShell { nativeBuildInputs = [ diff --git a/tools/llvmStdenv.nix b/tools/llvmStdenv.nix new file mode 100644 index 000000000..8431dccd1 --- /dev/null +++ b/tools/llvmStdenv.nix @@ -0,0 +1,41 @@ +{ pkgs, ... }: + +let +llvmPackages = pkgs.llvmPackages_16; +in + +# This toolchain uses Clang as compiler, Mold as linker, libc++ as C++ standard +# library and compiler-rt as compiler runtime. Resulting rust binaries depend +# dynamically linked on the nixpkgs distribution of glibc. C++ binaries +# additionally depend dynamically on libc++, libunwind and libcompiler-rt. Due +# to a bug we also depend on libgcc_s. +# +# TODO(aaronmondal): At the moment this toolchain is only used for the Cargo +# build. The Bazel build uses a different mostly hermetic LLVM toolchain. We +# should merge the two by generating the Bazel cc_toolchain from this stdenv. +# This likely requires a rewrite of +# https://github.com/bazelbuild/bazel-toolchains as the current implementation +# has poor compatibility with custom container images and doesn't support +# generating toolchain configs from image archives. +# +# TODO(aaronmondal): Due to various issues in the nixpkgs LLVM toolchains we're +# not getting a pure Clang/LLVM toolchain here. My guess is that the runtimes +# were not built with the degenerate LLVM toolchain but with the regular GCC +# stdenv from nixpkgs. +# +# For instance, outputs depend on libgcc_s since libcxx seems to have been was +# built with a GCC toolchain. We're also not using builtin atomics, or at least +# we're redundantly linking libatomic. +# +# Fix this as it fixes a large number of issues, including better cross-platform +# compatibility, reduced closure size, and static-linking-friendly licensing. +# This requires building the llvm project with the correct multistage +# bootstrapping process. +pkgs.useMoldLinker ( + pkgs.overrideCC ( + llvmPackages.libcxxStdenv.override { + targetPlatform.useLLVM = true; + } + ) + llvmPackages.clangUseLLVM +)