diff --git a/nativelink-docs/about.mdx b/nativelink-docs/about.mdx new file mode 100644 index 000000000..4c553319c --- /dev/null +++ b/nativelink-docs/about.mdx @@ -0,0 +1,56 @@ +--- +title: About +description: 'What is NativeLink?' +--- + +Hero Light +Hero Dark + +Native link is an extremely (blazingly?) fast and efficient build cache and +remote executor for systems that communicate using the [Remote execution +protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) +such as [Bazel](https://bazel.build), [Buck2](https://buck2.build), +[Goma](https://chromium.googlesource.com/infra/goma/client/) and +[Reclient](https://github.com/bazelbuild/reclient). + +Supports Unix-based operating systems and Windows. + +**๐ŸŽฏ Goals** + + + Things should work out of the box as expected. + + + Don't waste time on inefficiencies & low resource usage. + + + Design choices should be optimized for what users want. + + + +**๐Ÿบ History** + +This project was first created due to frustration with similar projects not +working or being extremely inefficient. Rust was chosen as the language to +write it in because at the time Rust was going through a revolution in the +new-ish feature async-await. This made making multi-threading extremely +simple when paired with a runtime like tokio while still giving all the +lifetime and other protections that Rust gives. This pretty much guarantees +that we will never have crashes due to race conditions. This kind of project +seemed perfect, since there is so much asynchronous activity happening and +running them on different threads is most preferable. Other languages like +Go are good candidates, but other similar projects rely heavily on channels +and mutex locks which are cumbersome and have to be carefully designed by +the developer. Rust doesn't have these issues, since the compiler will +always tell you when the code you are writing might introduce undefined +behavior. The last major reason is because Rust is extremely fast, +/- a +few percent of C++ and has no garbage collection (like C++, but unlike Java, +Go, or Typescript). diff --git a/nativelink-docs/introduction.mdx b/nativelink-docs/introduction.mdx deleted file mode 100644 index bd57df135..000000000 --- a/nativelink-docs/introduction.mdx +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: Introduction -description: 'What is NativeLink?' ---- - -Hero Light -Hero Dark - -Native link is an extremely (blazingly?) fast and efficient build cache and -remote executor for systems that communicate using the [Remote execution -protocol](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto) -such as [Bazel](https://bazel.build), [Buck2](https://buck2.build), -[Goma](https://chromium.googlesource.com/infra/goma/client/) and -[Reclient](https://github.com/bazelbuild/reclient). - -Supports Unix-based operating systems and Windows. - -**The goals of this project are:** - - - Things should work out of the box as expected. - - - Don't waste time on inefficiencies & low resource usage. - - - Design choices should be optimized for what users want. - - diff --git a/nativelink-docs/mint.json b/nativelink-docs/mint.json index 986cf8d35..1e4ecb280 100644 --- a/nativelink-docs/mint.json +++ b/nativelink-docs/mint.json @@ -43,7 +43,7 @@ "navigation": [ { "group": "Get Started", - "pages": ["introduction", "quickstart", "contributing"] + "pages": ["quickstart", "contributing", "about"] } ], "footerSocials": { diff --git a/nativelink-docs/quickstart.mdx b/nativelink-docs/quickstart.mdx index 8edb94849..bf827c536 100644 --- a/nativelink-docs/quickstart.mdx +++ b/nativelink-docs/quickstart.mdx @@ -1,13 +1,12 @@ --- title: 'Quickstart' -description: 'Build with NativeLink remote execution in 5 minutes' +description: 'Getting started with NativeLink remote execution in <5 minutes' --- -## Setup your development -There are a few pre-requisites to get started if you do not use Docker to use NativeLink. +## ๐Ÿฆ€ Installing with Cargo -### Install or Update Rust +Install or Update Rust @@ -20,13 +19,32 @@ rustup update ``` -### Bazel Dependencies +```bash +cargo install --git https://github.com/TraceMachina/nativelink +``` -* Bazel 6.4.0+ -* Clang -* lld (exc. MacOS) -* python3 -* curl +### โš™๏ธ Configuration + +The `cas` executable reads a JSON file as it's only parameter, +`--config`. See [nativelink-config](./nativelink-config/examples/basic_cas.json) +for more details and examples. + +To grab the example in your current working directory, run: + +```bash +curl -O https://raw.githubusercontent.com/TraceMachina/nativelink/main/nativelink-config/examples/basic_cas.json +``` + +### ๐Ÿš€ Start Native Link + +```bash +cas basic_cas.json +``` + +## ๐Ÿงช Evaluating Native Link + +If Bazel is not installed follow the instructions below based on your +machine. @@ -40,6 +58,9 @@ bazel --version ``` ```shell Docker-ARM +# The commands for creating docker containers can fill up space very quickly when run frequently. +# If `apt update` throws an out of space error you can run `docker system prune` +# to free up space. Be careful to make sure it's not removing any containers/images you want to keep. docker run -it --name=NL -v $(pwd):/nativelink ubuntu:latest # If above has already been run or trying to start in new terminal: @@ -64,6 +85,10 @@ bazel --version ``` ```shell Docker-x86 +# The commands for creating docker containers can fill up space very quickly +# when run frequently. If `apt update` throws an out of space error you can +# run `docker system prune` to free up space. Be careful to make sure it's +# not removing any containers/images you want to keep. docker run -it --name=NL -v $(pwd):/nativelink ubuntu:latest # If above has already been run or trying to start in new terminal: @@ -99,72 +124,83 @@ bazel --version -The commands for creating docker containers can fill up space very quickly when run frequently. If `apt update` throws an out of space error you can run `docker system prune` to free up space. Be careful to make sure it's not removing any containers/images you want to keep. - -### Compile and Run NativeLink Server - -The following command will allow you to compile and run the NativeLink server for the first time. +Once you've built Native Link and have an instance running with the +`basic_cas.json` configuration, launch a separate terminal session and run the +following command to connect the running server launched above to Bazel or +another RBE client: - ```shell - apt install -y gcc g++ lld python3 - # Install cargo (if needed). - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh - source "$HOME/.cargo/env" - - # Run with Cargo: - # Unoptimized development build - cargo run --bin cas -- ./nativelink-config/examples/basic_cas.json +```sh +bazel test //... \ + --remote_instance_name=main \ + --remote_cache=grpc://127.0.0.1:50051 \ + --remote_executor=grpc://127.0.0.1:50051 \ + --remote_default_exec_properties=cpu_count=1 +``` - # Optimized release build - cargo run --release --bin cas -- ./nativelink-config/examples/basic_cas.json +For Windows Powershell; - # Run with Bazel: - # Unoptimized development build on Unix. - bazel run cas -- ./nativelink-config/examples/basic_cas.json +```powershell +bazel test //... ` + --remote_instance_name=main ` + --remote_cache=grpc://127.0.0.1:50051 ` + --remote_executor=grpc://127.0.0.1:50051 ` + --remote_default_exec_properties=cpu_count=1 +``` +This causes Bazel to run the commands through an all-in-one `CAS`, `scheduler` +and `worker`. - # Optimized release build on Unix. - bazel run -c opt cas -- ./nativelink-config/examples/basic_cas.json - ``` +๐ŸŽ‰ Tada ๐ŸŽ‰ Native Link is working. -The `--release` flag causes link-time-optmization to be enabled, which can take a while to compile, but will result in a much faster binary. +## ๐ŸŒฑ Building with Bazel -In a separate terminal, run the following command to start the NativeLink client. +**Build requirements:** +* Bazel 6.4.0+ +* A recent C++ toolchain with LLD as linker -```shell -bazel test //... \ - --remote_instance_name=main \ - --remote_cache=grpc://127.0.0.1:50051 \ - --remote_executor=grpc://127.0.0.1:50051 \ - --remote_default_exec_properties=cpu_count=1 +> [!TIP] +> This build supports Nix/direnv which provides Bazel but no C++ toolchain +> (yet). - # This causes bazel to run the commands through an all-in-one `CAS`, `scheduler` - # and `worker`. -``` -If the NativeLink server was run in a docker container, the above command for starting the NativeLink client will need to be run in the same container as the server in order to connect. +The following commands place an executable in `./bazel-bin/cas/cas` and start +the service: -### Configuration +```sh +# Unoptimized development build on Unix +bazel run cas -- ./nativelink-config/examples/basic_cas.json -The `cas` executable reads a JSON file as it's only parameter. See [nativelink-config](./nativelink-config) -for more details and examples. +# Optimized release build on Unix +bazel run -c opt cas -- ./nativelink-config/examples/basic_cas.json -### Example Deployments +# Unoptimized development build on Windows +bazel run --config=windows cas -- ./nativelink-config/examples/basic_cas.json -You can find a few example deployments in the [deployment-examples directory](./deployment-examples). +# Optimized release build on Windows +bazel run --config=windows -c opt cas -- ./nativelink-config/examples/basic_cas.json +``` -See the [terraform deployments](./deployment-examples/terraform) for an example -deployments that show off remote execution and cache capabilities. +The `--release` flag causes link-time-optmization to be enabled, which can take a while to compile, but will result in a much faster binary. +## ๐Ÿฆ€ Building with Cargo -### That's It! +**Build requirements:** -NativeLink is now running and you can start using it in your projects. +* Cargo 1.74.0+ +* A recent C++ toolchain with LLD as linker +> [!TIP] +> This build supports Nix/direnv which provides Cargo but no C++ +> toolchain/stdenv (yet). -## Advanced Setup +```bash +# Unoptimized development build +cargo run --bin cas -- ./nativelink-config/examples/basic_cas.json +# Optimized release build +cargo run --release --bin cas -- ./nativelink-config/examples/basic_cas.json +``` -### Installing with Nix +## โ„๏ธ Installing with Nix **Installation requirements:** @@ -185,16 +221,6 @@ For use in production pin the executable to a specific revision: nix run github:TraceMachina/nativelink/ ./basic_cas.json ``` -### Using the OCI image - -See the published [OCI images](https://github.com/TraceMachina/nativelink/pkgs/container/nativelink) -for pull commands. - -Images are tagged by nix derivation hash. The most recently pushed image -corresponds to the `main` branch. Images are signed by the GitHub action that -produced the image. Note that the [OCI workflow](https://github.com/TraceMachina/nativelink/actions/workflows/image.yaml) -might take a few minutes to publish the latest image. - ```sh # Get the tag for the latest commit export LATEST=$(nix eval github:TraceMachina/nativelink#image.imageTag --raw) @@ -203,7 +229,7 @@ export LATEST=$(nix eval github:TraceMachina/nativelink#image.imageTag --raw) cosign verify ghcr.io/tracemachina/nativelink:${LATEST} \ --certificate-identity=https://github.com/TraceMachina/nativelink/.github/workflows/image.yaml@refs/heads/main \ --certificate-oidc-issuer=https://token.actions.githubusercontent.com -``` + For use in production pin the image to a specific revision: @@ -217,10 +243,15 @@ cosign verify ghcr.io/tracemachina/nativelink:${PINNED_TAG} \ --certificate-oidc-issuer=https://token.actions.githubusercontent.com ``` - - The images are reproducible on `X86_64-unknown-linux-gnu`. If you're on - such a system you can produce a binary-identical image by building the - `.#image` flake output locally. Make sure that your `git status` is - completely clean and aligned with the commit you want to reproduce. - Otherwise the image will be tainted with a `"dirty"` revision label. - +For use in production pin the executable to a specific revision: + +```sh +nix run github:TraceMachina/nativelink/ ./basic_cas.json +``` + +## ๐Ÿš€ Example Deployments + +You can find a few example deployments in the [deployment-examples directory](./deployment-examples). + +See the [terraform deployments](./deployment-examples/terraform) for an example +deployments that show off remote execution and cache capabilities.