Merge branch 'fix/improve-rust-builder' into fix/improve-rust-builder-sj

.config/dictionaries/project.dic

@@ -72,7 +72,10 @@ rustdoc
 rustdocflags
 rustflags
 rustfmt
+Rustup
+rustup
 sqlfluff
+stdcfgs
 subproject
 subprojects
 superfences
@@ -82,6 +85,7 @@ testdocs
 testpackage
 testunit
 Timoni
+toolsets
 transpiling
 UDCs
 uniseg

docs/src/guides/languages/rust.md

@@ -9,8 +9,6 @@ tags:
 # :simple-rust: Rust
 <!-- markdownlint-enable single-h1 -->
 
-<!-- cspell: words toolsets stdcfgs depgraph Rustup -->
-
 ## Introduction
 
 <!-- markdownlint-disable max-one-sentence-per-line -->
@@ -59,26 +57,31 @@ builder:
 
     COPY --dir .cargo .config crates .
     COPY Cargo.toml .
+    COPY clippy.toml deny.toml rustfmt.toml .
 ```
 
 The first target `builder` is responsible for preparing configured Rust environments and,
 install all needed tools and dependencies.
 
 #### Builder steps
 
-1. First step of `builder` target is to prepare a Rust environment via `+installer` target,
-which is called in `SETUP` FUNCTION.
-The `installer` target installs necessary tools for `+rust-base` target and copies
+1. First step of `+builder` target is to prepare a Rust environment via `+installer` target,
+which is called in `+SETUP` FUNCTION.
+The `+installer` target installs necessary tools for `+rust-base` target and copies
 common scripts and standardized Rust configs.
-The `rust-base` provides a base Rustup build environment.
-  It installs necessary
-packages, including development libraries and tools.
+The `+rust-base` provides a base Rustup build environment.
+It installs necessary packages, including development libraries and tools.
 Clippy linter, LLVM tools for generating code coverage, and nightly toolchain are installed.
 2. Next step is to copy source code of the project.
 Note that you need to copy only needed files for Rust build process,
 any other irrelevant stuff should omitted.
 3. And finally finalize the build with `+SETUP` FUNCTION which takes no arguments.
 
+<!-- markdownlint-disable max-one-sentence-per-line -->
+!!! Warning
+    Please ensure that Rust version set in `rust-toolchain.toml` matches the Docker image tag uses in `+rust-base` target.
+<!-- markdownlint-enable max-one-sentence-per-line -->
+
 ### Running checks
 
 ```Earthfile
@@ -96,7 +99,7 @@ all-hosts-check:
 ```
 
 With prepared environment and all data, we're now ready to start operating with the source code and configuration files.
-The `check` target performs all checks and validation procedures
+The `+check` target performs all checks and validation procedures
 using the help of `std_checks.py` script.
 This script performs static checks of the Rust project as
 `cargo fmt`, `cargo machete`, `cargo deny` which will validate formatting,
@@ -115,7 +118,7 @@ to be the same as defined in `earthly/rust/stdcfgs` directory of the `catalyst-c
 So when you are going to setup a new Rust project, copy these configuration files
 described above to the appropriate location of your Rust project.
 
-Another target as `all-hosts-check` just invokes `check` with the specified `--platform`.
+Another target as `+all-hosts-check` just invokes `+check` with the specified `--platform`.
 It is needed for the local development to double check that everything works for different platforms.
 It is important to define a `linux` target platform with a proper CPU architecture
 for the Rust project when you are building it inside Docker
@@ -146,16 +149,16 @@ all-hosts-build:
     BUILD --platform=linux/amd64 --platform=linux/arm64 +build
 ```
 
-After successful performing checks of the Rust project we can finally `build` artifacts.
-Obviously it inherits `builder` target environment and then performs build of the binary.
+After successful performing checks of the Rust project we can finally build artifacts.
+Obviously it inherits `+builder` target environment and then performs build of the binary.
 Important to note that in this particular example we are dealing with the executable Rust project,
 so it produces binary as a final artifact.
 We will discuss another scenario of building a Rust library later.
 Actual build process is done with the `std_build.py` script.
 Here is the full list of configuration of this script:
 
 ```bash
- usage: std_build.py [-h] [--build_flags BUILD_FLAGS]
+ usage: std_build.py [-h] [-v] [--build_flags BUILD_FLAGS]
                      [--doctest_flags DOCTEST_FLAGS] [--test_flags TEST_FLAGS]
                      [--bench_flags BENCH_FLAGS] [--with_test]
                      [--cov_report COV_REPORT] [--with_bench] [--libs LIBS]
@@ -164,7 +167,8 @@ Here is the full list of configuration of this script:
  Rust build processing.
 
  options:
-   -h, --help            show this help message and exit
+   -h, --help            Show this help message and exit.
+   -v --verbose          Show the output of executed commands verbosely.
    --build_flags BUILD_FLAGS
                          Additional command-line flags that can be passed to
                          the `cargo build` command.
@@ -225,6 +229,7 @@ for the specified `--libs="crate1"` argument.
 10. Running smoke tests on provided binary names (`--bins` argument).
 
 Final step is to provide desired artifacts: docs and binary.
+Note that all commands within the `std_build.py` are written to be run in parallel, resulting in a faster speeds.
 
 ### Test
 
@@ -247,6 +252,10 @@ Unfortunately, Rust tooling does not have the capability to preserve and maintai
 `stable` and `nightly` toolchains simultaneously.
 In our builds, we only preserve the `stable` toolchain version (`rust-toolchain.toml` file).
 
+## Rust tools
+
+All the necessary Rust tools can be found in [tool](../../../../earthly/rust/tools/Earthfile).
+
 ## Rust FUNCTIONs
 
 While leveraging the [Earthly lib/rust](https://github.com/earthly/lib/tree/main/rust),
@@ -267,6 +276,8 @@ that our project needed.
 
 * `CARGO` : This FUNCTION serves as a shim of the original lib/rust `CARGO` FUNCTION
   to guarantee consistent usage of the appropriate upstream Rust library.
+  Therefore, users of `catalyst-ci` who wish to use `rust+CARGO` from `lib/rust`
+  should utilize the `+CARGO` implementation provided in this repository.
 
 ```Earthfile
     # Example of using `CARGO` to install a Rust tool

earthly/rust/Earthfile

@@ -1,16 +1,16 @@
 VERSION 0.8
 
-# In order to ensure all consumers of CAT-CI use the correct upstream
-# Rust library from Earthly, this is the only place that Earthfile can be imported.
-# Functions or targets in that Earthfile MUST be shimmed here.
+# To ensure all consumers of `catalyst-ci` use the correct upstream,
+# the Rust library from Earthly can only be imported in this Earthfile.
+# FUNCTIONs or targets in that Earthfile MUST be shimmed here.
 # No other earthfile should reference the rust library in the earthly/lib repo.
 IMPORT github.com/earthly/lib/rust:3.0.2 AS rust
 
 # Local Earthfile reference imports
 IMPORT ./tools AS rust-tools
 IMPORT ../../utilities/scripts AS scripts
 
-# cspell: words rustup miri ripgrep stdcfgs toolset depgraph lcov psycopg
+# cspell: words miri ripgrep toolset lcov psycopg
 # cspell: words TARGETPLATFORM TARGETOS TARGETARCH TARGETVARIANT USERPLATFORM USEROS USERARCH USERVARIANT
 # cspell: words findutils fileset
 
@@ -56,7 +56,7 @@ rust-base:
             openssl-dev \
             openssl-libs-static
 
-    # Fix up font cache
+    # Fix up font cache.
     RUN fc-cache -f
 
     # Make sure we have cargo.
@@ -111,7 +111,33 @@ rust-base-plus-tools:
 installer:
     FROM +rust-base-plus-tools
 
-    # Call +INIT before copying the source file to avoid installing dependencies every time source code changes. 
+    # Preset CARGO_HOME and the application search path.
+    ENV CARGO_HOME="$HOME/.cargo"
+    ENV PATH="$PATH:$CARGO_HOME/bin"
+
+# Add all tools we use for Rust builds to the base builder image.
+rust-base-plus-tools:
+    FROM +rust-base
+
+    COPY rust-tools+tool-cargo-nextest/cargo-nextest $CARGO_HOME/bin/cargo-nextest
+    COPY rust-tools+tool-cargo-machete/cargo-machete $CARGO_HOME/bin/cargo-machete
+    COPY rust-tools+tool-refinery/refinery $CARGO_HOME/bin/refinery
+    COPY rust-tools+tool-cargo-deny/cargo-deny $CARGO_HOME/bin/cargo-deny
+    COPY rust-tools+tool-cargo-modules/cargo-modules $CARGO_HOME/bin/cargo-modules
+    COPY rust-tools+tool-cargo-depgraph/cargo-depgraph $CARGO_HOME/bin/cargo-depgraph
+    COPY rust-tools+tool-cargo-llvm-cov/cargo-llvm-cov $CARGO_HOME/bin/cargo-llvm-cov
+    COPY rust-tools+tool-wasm-tools/wasm-tools $CARGO_HOME/bin/wasm-tools
+    COPY rust-tools+tool-cargo-expand/cargo-expand $CARGO_HOME/bin/cargo-expand
+    COPY rust-tools+tool-wit-bindgen-cli/wit-bindgen $CARGO_HOME/bin/wit-bindgen
+    COPY rust-tools+tool-wasmtime/verify-component-adapter $CARGO_HOME/bin/verify-component-adapter
+    COPY rust-tools+tool-cargo-sweep/cargo-sweep $CARGO_HOME/bin/cargo-sweep
+
+
+# installer - fully setup our Rust caching.
+installer:
+    FROM +rust-base-plus-tools
+
+    # Call `+INIT` before copying the source file to avoid installing dependencies every time source code changes. 
     # This parametrization will be used in future calls to functions of the library
     # Init using the common cat-ci cache prefix.
     DO rust+INIT --keep_fingerprints=true
@@ -178,7 +204,7 @@ EXECUTE:
        DO rust+REMOVE_SOURCE_FINGERPRINTS
     END
 
-    # target/debug should not be present, so purge it just in case
+    # target/debug should not be present, so purge it just in case.
     IF [ "$ALLOW_DEBUG_TARGET" != "true" ]
         RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE \
             set -e; \
@@ -190,7 +216,7 @@ EXECUTE:
     # We then check the existence of the "fail" file after any artifacts are saved that
     # should be saved regardless.
     RUN --mount=$EARTHLY_RUST_CARGO_HOME_CACHE --mount=$EARTHLY_RUST_TARGET_CACHE \
-        set +e; \
+        set -e; \
         rm -f fail; \
         $cmd $args1 $args2 $args3 $args4 $args5 $args6 || touch fail; \
         cargo sweep -r -t $EARTHLY_SWEEP_DAYS; \
@@ -257,8 +283,8 @@ CARGO:
 # This is a shim which MUST be used by projects consuming catalyst-ci rather than using the 
 # upstream earthly rust library directly.
 #
-# Use this function when you want to SAVE an ARTIFACT from the target folder (mounted cache), always trying to minimize the total size of the copied fileset.
-# Notice that in order to run this function, +SET_CACHE_MOUNTS_ENV or +CARGO must be called first.
+# Use this FUNCTION when you want to SAVE an ARTIFACT from the target folder (mounted cache), always trying to minimize the total size of the copied fileset.
+# Notice that in order to run this FUNCTION, +SET_CACHE_MOUNTS_ENV or +CARGO must be called first.
 # Arguments:
 #   - output: Regex matching output artifacts files to be copied to ./target folder in the caller filesystem (image layers).
 # Example: