feat: implement RFC 3553 to add SBOM support #13709

justahero · 2024-04-05T11:23:51Z

What does this PR try to resolve?

This PR is an implementation of RFC 3553 to add support to generate pre-cursor SBOM files for compiled artifacts in Cargo.

How should we test and review this PR?

The RFC 3553 adds a new option to Cargo to emit SBOM pre-cursor files. A project can be configured either by the new Cargo config field sbom.

# .cargo/config.toml
[build]
sbom = true

or using the environment variable CARGO_BUILD_SBOM=true. The sbom option is an unstable feature and requires the -Zsbom flag to enable it.

Check out this branch & compile Cargo. Pick a Cargo project to test it on, then run:

CARGO_BUILD_SBOM=true <path/to/compiled/cargo>/target/debug/cargo build -Zsbom

All generated *.cargo-sbom.json files are located in the target folder alongside their artifacts. To list all generated files use:

find ./target -name "*.cargo-sbom.json"

then check their content. To see the current output format, see these examples.

Additional information

There are a few things that I would like to get feedback on, in particular the generated JSON format is not final. Currently it holds the information listed in the RFC 3553, but it could be further enriched with information only available during builds.

During the implementation a number of questions arose.

Is using the UnitGraph the right structure to determine all dependencies?
Is the location in the compile method to generate the SBOM files appropriate?
Is there important information missing from the generated output, some fields that should be omitted?
How best to check JSON output in the related tests in testsuite, are useful tests missing?
Are custom build script dependencies correctly resolved?

Thanks @arlosi, @RobJellinghaus and @lfrancke for initial guidance & feedback.

RFC: #RFC: cargo-sbom rfcs#3553
SBOM output examples (serde_json & axum-core crates)

rustbot · 2024-04-05T11:23:56Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

justahero · 2024-04-05T11:24:23Z

src/cargo/core/compiler/build_runner/mod.rs

+    /// Returns the list of SBOM output file paths for a given Unit.
+    ///
+    /// Only call this function when `sbom` is active.
+    pub fn sbom_output_files(&self, unit: &Unit) -> CargoResult<Vec<PathBuf>> {


I'm uncertain if this function should return a Vec or if a single PathBuf is actually sufficient.

It might output several different crate-types, e.g. rlib, cdylib.

heisen-li · 2024-04-07T12:12:11Z

Much respect for your contribution.

From my kind reminders, it seems appropriate to modify the documentation of the corresponding sections, e.g. Configuration, Environment Variables.

weihanglo · 2024-04-07T14:35:13Z

Thanks for the reminder, @heisen-li. Would love to see a doc update, though we should probably focus on the design discussion first, as the location of the configuration is not yet decided. (See rust-lang/rfcs#3553 (comment)).

epage · 2024-04-09T16:02:21Z

One approach for the docs (if this is looking to be merged) is to put the env and config documentation fragments in the Unstable docs.

Similar to the generation of `depinfo` files, a function is called to generated SBOM precursor file named `output_sbom`. It takes the `BuildRunner` & the current `Unit`. The `sbom` flag can be specified as a cargo build option, but it's currently not configured correctly. To test the generation the flag is set to `true`. This passes in the cargo build config `sbom`.

This ignores dependencies for custom build scripts. The output should be similar to what `cargo tree` reports.

This is similar to what the `cargo metadata` command outputs.

This extracts the logic to get the list of SBOM output file paths into its own function in `BuildRunner` for a given Unit.

* extract sbom config into helper function

Still needs to check output.

weihanglo · 2024-05-02T20:13:27Z

src/cargo/core/compiler/mod.rs

        base.arg("-Z").arg("binary-dep-depinfo");
    }

    if is_primary {
        base.env("CARGO_PRIMARY_PACKAGE", "1");
+
+        if gctx.cli_unstable().sbom && build_runner.bcx.build_config.sbom {


IMO, paths are better to be joined using semi-colons, maybe via std::env::join_paths, to maximize the compatibility.

weihanglo · 2024-05-02T20:25:24Z

src/cargo/core/compiler/build_runner/mod.rs

+    /// Returns the list of SBOM output file paths for a given Unit.
+    ///
+    /// Only call this function when `sbom` is active.
+    pub fn sbom_output_files(&self, unit: &Unit) -> CargoResult<Vec<PathBuf>> {


It might output several different crate-types, e.g. rlib, cdylib.

weihanglo · 2024-05-02T20:30:23Z

src/cargo/core/compiler/build_runner/mod.rs

+            .iter()
+            .filter(|o| matches!(o.flavor, FileFlavor::Normal | FileFlavor::Linkable))
+            .filter_map(|output_file| output_file.hardlink.as_ref())
+            .map(|link_dst| link_dst.with_extension("cargo-sbom.json"))


Do we want to handle name collisions here? For example dylibs might collide with each their as their name have no -<hash> suffix.

See #6313 for some details.

weihanglo · 2024-05-02T20:32:01Z

src/cargo/core/compiler/output_sbom.rs

+//! cargo-sbom precursor files for external tools to create SBOM files from.
+//! See [`output_sbom`] for more.
+
+use std::{


nit: in Cargo we don't do nested multi-line import, though this is not enforced :)

weihanglo · 2024-05-02T20:35:25Z

tests/testsuite/sbom.rs

+    assert!(p.bin("foo").with_extension("cargo-sbom.json").is_file());
+    assert_eq!(
+        1,
+        p.glob(p.target_debug_dir().join("libfoo.cargo-sbom.json"))


I guess we might need to deal with different naming convention on different platform. (Windows specifically?)

weihanglo · 2024-05-02T20:36:23Z

tests/testsuite/sbom.rs

+                build = "build.rs"
+
+                [build-dependencies]
+                cc = "1.0.46"


Should avoid introducing external crates.io dependency. Use something like

Package::new("baz", "0.0.1").publish();

weihanglo · 2024-05-02T20:41:51Z

tests/testsuite/sbom.rs

I would like to see some UI test for the JSON output. It's easier to check what's inside the SBOM file.

https://doc.crates.io/contrib/tests/writing.html#ui-tests

weihanglo

Just note that I reviewed this as-is, didn't really think too much for the design itself. Thank you for working on this!

weihanglo · 2024-05-02T20:51:06Z

src/cargo/core/compiler/build_config.rs

@@ -102,6 +104,16 @@ impl BuildConfig {
            anyhow::bail!("-Zbuild-std requires --target");
        }

+        // If sbom flag is set, it requires the unstable feature
+        let sbom = match gctx.get_env_os("CARGO_BUILD_SBOM") {


This was mentioned in office hours. Just wrote it down for reference. GlobalContext::get_bool could cover both env and config.

weihanglo · 2024-05-02T20:57:15Z

src/cargo/core/compiler/build_config.rs

+        if sbom && !gctx.cli_unstable().sbom {
+            anyhow::bail!("Cargo build config 'sbom' is unstable; pass `-Zsbom` to enable it");
+        }


maybe a soft error (warning) is more friendly and won't block people using the same config from different toolchain. Something like

cargo/src/cargo/core/workspace.rs

Lines 325 to 327 in 8733d5a

self.gctx()

.shell()

.warn("ignoring `resolver` config table without `-Zmsrv-policy`")?;

(And provide a link to that unstable documentation when we have one)

weihanglo · 2024-05-02T21:02:44Z

src/cargo/core/compiler/output_sbom.rs

+        let sbom = Sbom::new(unit, packages.clone(), rustc.clone());
+
+        let mut outfile = BufWriter::new(paths::create(sbom_output_file)?);
+        let output = serde_json::to_string_pretty(&sbom)?;


Perhaps pretty not important at this moment. Just wonder how big the size of the output sbom would be, and at some point we might want to use serde_json::to_writer_pretty instead.

rustbot assigned ehuss Apr 5, 2024

rustbot added A-build-execution Area: anything dealing with executing the compiler A-configuration Area: cargo config files and env vars A-unstable Area: nightly unstable support S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 5, 2024

justahero commented Apr 5, 2024

View reviewed changes

justahero force-pushed the rfc3553/cargo-sbom-support branch from 74dafa0 to 190682e Compare April 6, 2024 14:31

justahero added 14 commits May 2, 2024 21:46

Add Sbom types to serialize data

9262fe7

Output source & profile

4ddede9

Output dependencies for unit deps.

1ced968

Trying to fetch all dependencies

8bbca37

This ignores dependencies for custom build scripts. The output should be similar to what `cargo tree` reports.

Output package dependencies

890a096

This is similar to what the `cargo metadata` command outputs.

Extract logic to fetch sbom output files

3c2e7f3

This extracts the logic to get the list of SBOM output file paths into its own function in `BuildRunner` for a given Unit.

Add test file to check sbom output

f11a92f

Add test to check project with bin & lib

656c599

* extract sbom config into helper function

Add build type to dependency

9e248c5

Add test to read JSON

011a461

Still needs to check output.

Guard sbom logic behind unstable feature

3530268

Add test with custom build script

dcd2ea1

Remove calls to '.stream'

ae0881c

justahero force-pushed the rfc3553/cargo-sbom-support branch from 190682e to ae0881c Compare May 2, 2024 19:54

weihanglo reviewed May 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement RFC 3553 to add SBOM support #13709

feat: implement RFC 3553 to add SBOM support #13709

justahero commented Apr 5, 2024 •

edited

rustbot commented Apr 5, 2024

justahero Apr 5, 2024

weihanglo May 2, 2024

heisen-li commented Apr 7, 2024

weihanglo commented Apr 7, 2024

epage commented Apr 9, 2024

weihanglo May 2, 2024

weihanglo May 2, 2024

weihanglo May 2, 2024

weihanglo May 2, 2024

weihanglo May 2, 2024

weihanglo May 2, 2024

weihanglo May 2, 2024

weihanglo left a comment

weihanglo May 2, 2024

weihanglo May 2, 2024

weihanglo May 2, 2024

	self.gctx()
	.shell()
	.warn("ignoring `resolver` config table without `-Zmsrv-policy`")?;

feat: implement RFC 3553 to add SBOM support #13709

Are you sure you want to change the base?

feat: implement RFC 3553 to add SBOM support #13709

Conversation

justahero commented Apr 5, 2024 • edited

What does this PR try to resolve?

How should we test and review this PR?

Additional information

rustbot commented Apr 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heisen-li commented Apr 7, 2024

weihanglo commented Apr 7, 2024

epage commented Apr 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weihanglo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justahero commented Apr 5, 2024 •

edited