Rewrite the build infrastructure (#394)

Our previous build layout was fairly custom and hard to maintain.
Migrate to a more standards-conform layout that is compatible with good
Cargo and Bazel practices and feels natural to develop in.

This commit swaps the previous "one-crate-per-file" layout to a layout
where we have a few medium sized crates that are just fine-grained
enough to allow almost the same amount of parallelization as the
previous layout.

The new layout enables dependency auto-updaters, standard cargo tooling
and doc autogeneration. This should improve DX quite a bit and gives us
some much needed compatibility with nix/cargo interop and should make
most standard cargo extensions compatible with the repository.

Some notable changes to the file layout:

- The previous `error` crate is no longer part of `utils`. It is now a
  standalone module. This was necessary to keep macro exports compatible
  with the Cargo build.
- The old `scheduler.rs` file is now split into `action_scheduler.rs`
  and `worker_scheduler.rs` since we don't want imports like
  `native_link_scheduler::scheduler`.
- Some tests now use a slightly different import mechanism for test-only
  dependencies.
- The store traits are now in `native_link_utils`, as it was used
  virtually everywhere. Moving this up in the dependency graph allows
  for higher parallelization of the other crates.

Added documentation targets for every crate (except the `cas` binary
itself, as this will require a dedicated commit). These doc targets can
be viewed by pointing a browser to
`bazel-bin/thetarget.rustdoc/thetarget/index.html`.

Added rustfmt configuration to standardize imports. We now use `crate::`
syntax which plays well with rusfmts's import formatting. The new layout
makes it easier to spot crate/directory interdependencies so that we can
remove them in future commits.

.github/workflows/main.yml

@@ -49,11 +49,6 @@ jobs:
           curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=1.70.0 \
         ' && \
         docker commit native-link-cargo trace_machina/native-link:cargo
-    - name: Check Cargo.toml is up to date
-      run: |
-        docker run --rm -w /root/native-link -v $PWD:/root/native-link trace_machina/native-link:cargo python3 ./tools/build_cargo_manifest.py && \
-        git diff --exit-code || \
-        (echo "Cargo.toml is out of date. Please run: python ./tools/build_cargo_manifest.py" && exit 1)
     - name: Compile & test with cargo
       run: |
         docker run --rm -w /root/native-link -v $PWD:/root/native-link trace_machina/native-link:cargo bash -c ' \

.rustfmt.toml

@@ -1 +1,8 @@
 max_width = 120
+reorder_imports = true
+imports_granularity = "Module"
+group_imports = "StdExternalCrate"
+
+# Don't format out of line modules.
+unstable_features = true
+skip_children = true

BUILD.bazel

@@ -0,0 +1,51 @@
+load("@rules_rust//rust:defs.bzl", "rust_binary")
+
+exports_files(
+    [
+        ".rustfmt.toml",
+    ],
+    visibility = ["//visibility:public"],
+)
+
+rust_binary(
+    name = "cas",
+    srcs = [
+        "src/bin/cas.rs",
+    ],
+    deps = [
+        "//error",
+        "//native-link-config",
+        "//native-link-scheduler",
+        "//native-link-service",
+        "//native-link-store",
+        "//native-link-util",
+        "//native-link-worker",
+        "//proto",
+        "@crate_index//:async-lock",
+        "@crate_index//:axum",
+        "@crate_index//:clap",
+        "@crate_index//:env_logger",
+        "@crate_index//:futures",
+        "@crate_index//:hyper",
+        "@crate_index//:json5",
+        "@crate_index//:parking_lot",
+        "@crate_index//:prometheus-client",
+        "@crate_index//:rustls-pemfile",
+        "@crate_index//:scopeguard",
+        "@crate_index//:tokio",
+        "@crate_index//:tokio-rustls",
+        "@crate_index//:tonic",
+        "@crate_index//:tower",
+    ],
+)
+
+genrule(
+    name = "dummy_test_sh",
+    outs = ["dummy_test.sh"],
+    cmd = "echo \"sleep .1;   echo $$(printf '=%.0s' {1..100})\" > \"$@\"",
+)
+
+sh_test(
+    name = "dummy_test",
+    srcs = [":dummy_test_sh"],
+)