Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add package-to-file location evidence relationships #1698

Merged
merged 9 commits into from
Apr 14, 2023

Conversation

wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Mar 27, 2023

Each package tracks which files contributed to syft being able to detect the package's existence via the pkg.Package.Locations field. This PR exposes these locations as package-to-file relationships to show that package X is "evident by" file Y. This is made possible with the new evident-by relationship type, which maps to the OTHER SPDX relationship type and depends on a comment value to support decoding effectively.

The current implementation takes all locations from pkg.Package.Location with the "primary" evidence annotation and makes relationships for them. This is nice since there is a single, downstream point in the code that would be responsible for creating these relationships.

In the future application configuration can be introduced which allows for supporting evidence to also be represented with an evident-by relationship type.

This PR also:

  • Fixes a small problem where decoding SPDX relationships of the type OTHER meant for "ownership-by-file-overlap" was interpreted as "dependency-of" instead
  • Fixes Decoding of the syft-json format does not handle files #1534 since this PR adds file nodes to the SBOM which would not be able to be interpreted on multiple encode-decode-encode cycles) which would fail the encode-decode-encode integration test (specifically for the syft json format).

CC: @jedevc @cdupuis

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman added the enhancement New feature or request label Mar 27, 2023
@wagoodman wagoodman requested a review from a team March 27, 2023 17:40
@github-actions
Copy link

github-actions bot commented Mar 27, 2023

Benchmark Test Results

Benchmark results from the latest changes vs base branch
goos: linux%0Agoarch: amd64%0Apkg: github.com/anchore/syft/test/integration%0Acpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz%0A                                                          │ ./.tmp/benchmark-bf92326.txt │%0A                                                          │            sec/op            │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   11.49m ±  1%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             897.6µ ±  1%25%0AImagePackageCatalogers/python-package-cataloger-2                           3.110m ±  2%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   764.1µ ±  4%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       385.7µ ±  1%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   571.0µ ±  1%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   523.7µ ± 18%25%0AImagePackageCatalogers/java-cataloger-2                                     11.19m ±  3%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     91.75µ ±  1%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    613.2µ ±  4%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         92.89µ ±  1%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              1.199m ±  1%25%0AImagePackageCatalogers/portage-cataloger-2                                  377.6µ ±  1%25%0AImagePackageCatalogers/nix-store-cataloger-2                                273.5µ ±  1%25%0AImagePackageCatalogers/sbom-cataloger-2                                     117.6µ ±  1%25%0AImagePackageCatalogers/binary-cataloger-2                                   199.8µ ±  2%25%0AImagePackageCatalogers/linux-kernel-cataloger-2                             49.61m ±  4%25%0Ageomean                                                                     802.7µ%0A%0A                                                          │ ./.tmp/benchmark-bf92326.txt │%0A                                                          │             B/op             │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   5.070Mi ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             140.1Ki ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                           982.8Ki ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   179.8Ki ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       98.55Ki ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   169.6Ki ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   178.1Ki ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                     2.777Mi ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     8.594Ki ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    144.9Ki ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         9.906Ki ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              409.6Ki ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                  85.80Ki ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                48.99Ki ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                     14.20Ki ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                   30.90Ki ± 0%25%0AImagePackageCatalogers/linux-kernel-cataloger-2                             78.44Mi ± 0%25%0Ageomean                                                                     194.5Ki%0A%0A                                                          │ ./.tmp/benchmark-bf92326.txt │%0A                                                          │          allocs/op           │%0AImagePackageCatalogers/alpmdb-cataloger-2                                    86.83k ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                              2.280k ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                            15.95k ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                    3.797k ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                        1.321k ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                    2.989k ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                    3.876k ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                      39.45k ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                       228.0 ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                     3.666k ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                           281.0 ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                               6.326k ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                   1.660k ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                  886.0 ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                       394.0 ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                     874.0 ± 0%25%0AImagePackageCatalogers/linux-kernel-cataloger-2                              3.001k ± 0%25%0Ageomean                                                                      2.657k

@wagoodman wagoodman self-assigned this Apr 10, 2023
@wagoodman
Copy link
Contributor Author

For interested folks, this PR will be refactored to use location annotations being introduced in #1723 and allow for expressing only primary evidence.

…ships

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman marked this pull request as ready for review April 14, 2023 18:55
@wagoodman
Copy link
Contributor Author

JSON schema diff for reviewers:

# ❯ diff schema/json/schema-7.1.3.json schema/json/schema-7.1.4.json
571a572,574
>         },
>         "size": {
>           "type": "integer"
580c583,584
<         "mimeType"
---
>         "mimeType",
>         "size"

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman merged commit 4442285 into main Apr 14, 2023
9 checks passed
@wagoodman wagoodman deleted the add-evidence-relationships branch April 14, 2023 19:08
spiffcs added a commit that referenced this pull request Apr 17, 2023
* main: (35 commits)
  Fix kernel cataloger test fixtures (#1742)
  feat: Support scanning license files in golang packages over the network (#1630)
  Add package-to-file location evidence relationships (#1698)
  Add Linux Kernel cataloger (#1694)
  Add annotations for evidence on package locations (#1723)
  add format make target (#1733)
  Update tests to not fail on Mac M1's. (#1730)
  chore(deps): update bootstrap tools to latest versions (#1728)
  Add support for nar files. (#1727)
  add highlevel details about catalogers (#1726)
  chore(deps): bump golang.org/x/net from 0.8.0 to 0.9.0 (#1722)
  chore(deps): update stereoscope to e95d60a265e384df29b7a139f5c5402d6ad72e06 (#1721)
  feat: gradle lockfile support (#1719)
  chore(deps): bump github.com/docker/docker (#1715)
  chore(deps): bump golang.org/x/mod from 0.9.0 to 0.10.0 (#1713)
  chore(deps): bump golang.org/x/term from 0.6.0 to 0.7.0 (#1714)
  chore(deps): bump github.com/spf13/cobra from 1.6.1 to 1.7.0 (#1716)
  chore(deps): bump peter-evans/create-pull-request from 4 to 5 (#1712)
  chore: update tools-golang to v0.5.0 (#1717)
  Add Nix cataloger (#1696)
  ...

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
* add evident-by relationship

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* wire up evident-by relationship geneation

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* handle evident-by relationship in spdx formats

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix decoding file info for syft json format

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bump json schema to incorporate file size attribute

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* refactor to create relationships for primary evidence only

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove unused 7.0.2 json schema

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

---------

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Decoding of the syft-json format does not handle files
2 participants