feat: add RelationshipsBySourceOwnership to syft json output #1248

spiffcs · 2022-10-05T19:20:07Z

feat: add new optional source id to json schema

Update json schema to include the source id field.
This new field allows users to view relationships between a source and its packages

Currently src --> packages contain all cataloged packages.
This will become more nuanced as multiple source support is added.

Eventually, syft will be able to generate an SBOM with a single root node that contains multiple
sources, which then contain multiple packages, which then contain some
amount of files.

These entities are also not strictly confined to the described tree structure.
They can behave more like a graph where other directional edges may be applied across entities.

TODO:

Integration test updates encode/decode cycle

Signed-off-by: Christopher Phillips christopher.phillips@anchore.com

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

Update json schema to include the optional source id field. This new field allows users to view relationships between packages and the source. Currently src --> contains all packages cataloged. This will become more nuanced as multiple source support is added. Eventually we can generate an SBOM with a single root node that contains multiple sources, which then contain multiple packages, which then contain some amount of files. These entities are also not strictly confined into the described tree structure and can behave more like a graph where other directional edges may be applied across entities. Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

github-actions · 2022-10-05T19:28:47Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/alpmdb-cataloger-2                    11.1ms ± 1%    14.3ms ± 2%  +28.86%  (p=0.016 n=4+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.25ms ± 1%    1.64ms ±11%  +31.62%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            3.18ms ± 0%    4.03ms ± 2%  +26.58%  (p=0.008 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2    1.01ms ± 1%    1.30ms ± 2%  +28.54%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         704µs ± 0%     905µs ± 1%  +28.52%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     825µs ± 0%    1087µs ± 2%  +31.82%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpm-db-cataloger-2                    1.18ms ± 0%    1.54ms ± 2%  +30.05%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      14.0ms ± 0%    18.0ms ± 1%  +28.59%  (p=0.016 n=4+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.14ms ± 1%    1.50ms ± 1%  +32.43%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          2.09µs ± 2%    2.71µs ± 2%  +29.66%  (p=0.008 n=5+5)
ImagePackageCatalogers/dotnet-deps-cataloger-2               1.30ms ± 1%    1.68ms ± 3%  +29.81%  (p=0.008 n=5+5)
ImagePackageCatalogers/portage-cataloger-2                    647µs ± 0%     856µs ± 2%  +32.16%  (p=0.008 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/alpmdb-cataloger-2                    5.26MB ± 0%    5.26MB ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2               202kB ± 0%     202kB ± 0%     ~     (p=0.151 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2             945kB ± 0%     945kB ± 0%     ~     (p=0.151 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     214kB ± 0%     214kB ± 0%     ~     (p=0.690 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         158kB ± 0%     158kB ± 0%     ~     (p=0.794 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     203kB ± 0%     203kB ± 0%     ~     (p=0.310 n=5+5)
ImagePackageCatalogers/rpm-db-cataloger-2                     302kB ± 0%     302kB ± 0%     ~     (p=0.690 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.44MB ± 0%    3.44MB ± 0%     ~     (p=0.421 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.25MB ± 0%    1.25MB ± 0%     ~     (p=0.056 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            672B ± 0%      672B ± 0%     ~     (all equal)
ImagePackageCatalogers/dotnet-deps-cataloger-2                369kB ± 0%     369kB ± 0%     ~     (p=0.222 n=5+5)
ImagePackageCatalogers/portage-cataloger-2                    136kB ± 0%     136kB ± 0%     ~     (p=0.421 n=5+5)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/alpmdb-cataloger-2                     85.7k ± 0%     85.7k ± 0%     ~     (p=0.444 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2               4.25k ± 0%     4.25k ± 0%     ~     (all equal)
ImagePackageCatalogers/python-package-cataloger-2             16.6k ± 0%     16.6k ± 0%     ~     (p=0.111 n=4+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     5.53k ± 0%     5.53k ± 0%     ~     (p=1.000 n=4+5)
ImagePackageCatalogers/javascript-package-cataloger-2         3.32k ± 0%     3.32k ± 0%     ~     (all equal)
ImagePackageCatalogers/dpkgdb-cataloger-2                     4.60k ± 0%     4.60k ± 0%     ~     (all equal)
ImagePackageCatalogers/rpm-db-cataloger-2                     8.13k ± 0%     8.13k ± 0%     ~     (all equal)
ImagePackageCatalogers/java-cataloger-2                       57.5k ± 0%     57.5k ± 0%     ~     (p=0.913 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      5.43k ± 0%     5.43k ± 0%     ~     (all equal)
ImagePackageCatalogers/go-module-binary-cataloger-2            15.0 ± 0%      15.0 ± 0%     ~     (all equal)
ImagePackageCatalogers/dotnet-deps-cataloger-2                7.27k ± 0%     7.27k ± 0%     ~     (all equal)
ImagePackageCatalogers/portage-cataloger-2                    3.59k ± 0%     3.59k ± 0%     ~     (all equal)

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

update integration tests to pass by refactoring ID onto metadata and promote schema changes to be non optional Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs · 2022-10-11T02:45:05Z

syft/formats/syftjson/to_syft_model.go

+	newSrc := &source.Source{
+		Metadata: *toSyftSourceData(s),
+	}
+	newSrc.SetID()


This method will generate a consistent ID for the source across encode/decode

Will this source ID change if other fields are set? It seems this is unnecessary to do here since we're using the incoming doc.Source.ID to do the mappings?

“We need to call SetID since id is a private field. Even if doc was in the function signature newSrc.id = doc.Source.ID is invalid

Our SetID should be consistent across serialization/deserialization as long as the source metadata remains unchanged between the ser/deser calls. SetID() will consistently produce the same ID. This is also shown to be correct in this case since our encode/decode integration tests are passing with the current changes.

I could also update SetID for this case to take (id string) as an argument which would override the field rather than calculate the hash.”

^--^ ^------------^ | | | +-> Summary in present tense. | +-------> Type: chore, docs, feat, fix, refactor, style, or test. [optional body] [optional footer(s)] Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

* main: refactor: Remove experimental Anchore Enterprise upload functionality (#1257) Update syft bootstrap tools to latest versions. (#1254) Update Stereoscope to d24c9d626b33fa720210b007a20767801827b532 (#1253) Update syft bootstrap tools to latest versions. (#1244) fix apkdb checksum representation (#1247)

syft/source/metadata.go

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

kzantow

No blocking comments, just a few questions 👍

kzantow · 2022-10-11T14:06:59Z

syft/formats/syftjson/to_syft_model.go

+	newSrc := &source.Source{
+		Metadata: *toSyftSourceData(s),
+	}
+	newSrc.SetID()


Will this source ID change if other fields are set? It seems this is unnecessary to do here since we're using the incoming doc.Source.ID to do the mappings?

syft/source/source.go

syft/lib.go

syft/source/source.go

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

syft/formats/syftjson/test-fixtures/snapshot/TestDirectoryEncoder.golden

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

syft/formats/syftjson/test-fixtures/snapshot/TestEncodeFullJSONDocument.golden

syft/formats/syftjson/test-fixtures/snapshot/TestImageEncoder.golden

spiffcs · 2022-10-11T16:13:37Z

syft/formats/syftjson/to_format_model.go

@@ -17,8 +17,6 @@ import (
 )

 // ToFormatModel transforms the sbom import a format-specific model.
-// note: this is needed for anchore import functionality
-// TODO: unexport this when/if anchore import functionality is removed


This is still needed to be exported since thetemplate package is using it in its Encode function

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

kzantow

👍

* main: (45 commits) feat: add RelationshipsBySourceOwnership to syft json output (#1248) fix: reset merged package into map; (#1258) refactor: Remove experimental Anchore Enterprise upload functionality (#1257) Update syft bootstrap tools to latest versions. (#1254) Update Stereoscope to d24c9d626b33fa720210b007a20767801827b532 (#1253) Update syft bootstrap tools to latest versions. (#1244) fix apkdb checksum representation (#1247) feat: add identifiable field to source object (#1243) feat: attest support for Singularity images (#1201) Update syft bootstrap tools to latest versions. (#1239) Update Stereoscope to 1b1b744a919964f38d14e1416fb3f25221b761ce (#1240) fix: Follow symlinks when searching for globs in all-layers scope (#1221) update requires to use list; remove field (#1234) Add Conan (C/C++) conan.lock file support (#1230) add sequence diagrams and flesh out TODO notes (#1233) Do not fail if unable to parse `.rpm` file (#1232) fix: support exclude patterns on Windows (#1228) Update syft bootstrap tools to latest versions. (#1225) Update Stereoscope to 56552770e555d764ea72b99d3c810326b27ead4a (#1224) Update syft bootstrap tools to latest versions. (#1223) ... Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

…#1248)

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

…#1248)

spiffcs added 3 commits October 5, 2022 15:16

feat: add RelationshipsBySourceOwnership

f1acfb9

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

feat: add ID to syft json Source struct

acee220

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

fix: update unit test snapshots

d12efaf

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs self-assigned this Oct 6, 2022

fix: update integration tests encode/decode

5e20ee8

update integration tests to pass by refactoring ID onto metadata and promote schema changes to be non optional Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs marked this pull request as ready for review October 11, 2022 02:38

spiffcs commented Oct 11, 2022

View reviewed changes

spiffcs added 3 commits October 10, 2022 22:50

fix: fix static analysis

0a0511e

^--^ ^------------^ | | | +-> Summary in present tense. | +-------> Type: chore, docs, feat, fix, refactor, style, or test. [optional body] [optional footer(s)] Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

fix: update unit tests and snapshot images

8f87cce

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

wagoodman reviewed Oct 11, 2022

View reviewed changes

syft/source/metadata.go Outdated Show resolved Hide resolved

fix: ignore id

17ab95f

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

kzantow approved these changes Oct 11, 2022

View reviewed changes

wagoodman reviewed Oct 11, 2022

View reviewed changes

syft/source/source.go Outdated Show resolved Hide resolved

fix: update source unmarshal test to include ID

f60446d

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs commented Oct 11, 2022

View reviewed changes

syft/formats/syftjson/test-fixtures/snapshot/TestDirectoryEncoder.golden Outdated Show resolved Hide resolved

spiffcs added 2 commits October 11, 2022 11:37

fix: PR Feedback Summary

56e9a21

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

fix: make SetID more explicit when gen new src

fcb08d1

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs commented Oct 11, 2022

View reviewed changes

syft/formats/syftjson/test-fixtures/snapshot/TestEncodeFullJSONDocument.golden Outdated Show resolved Hide resolved

spiffcs commented Oct 11, 2022

View reviewed changes

syft/formats/syftjson/test-fixtures/snapshot/TestImageEncoder.golden Show resolved Hide resolved

spiffcs commented Oct 11, 2022

View reviewed changes

fix: update unit tests in format to include src ID

b577534

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

kzantow approved these changes Oct 11, 2022

View reviewed changes

wagoodman approved these changes Oct 11, 2022

View reviewed changes

spiffcs merged commit 8957519 into main Oct 11, 2022

spiffcs deleted the 1241-source-relationships branch October 11, 2022 19:11

aiwantaozi pushed a commit to aiwantaozi/syft that referenced this pull request Oct 20, 2022

feat: add RelationshipsBySourceOwnership to syft json output (anchore…

e61bca3

…#1248)

spiffcs added a commit that referenced this pull request Oct 21, 2022

feat: add RelationshipsBySourceOwnership to syft json output (#1248)

1f32d24

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

spiffcs added a commit that referenced this pull request Oct 21, 2022

feat: add RelationshipsBySourceOwnership to syft json output (#1248)

61e5768

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024

feat: add RelationshipsBySourceOwnership to syft json output (anchore…

f984843

…#1248)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add RelationshipsBySourceOwnership to syft json output #1248

feat: add RelationshipsBySourceOwnership to syft json output #1248

spiffcs commented Oct 5, 2022 •

edited

Loading

github-actions bot commented Oct 5, 2022 •

edited

Loading

spiffcs Oct 11, 2022 •

edited

Loading

kzantow Oct 11, 2022

spiffcs Oct 11, 2022 •

edited

Loading

kzantow left a comment

kzantow Oct 11, 2022

spiffcs Oct 11, 2022

kzantow left a comment

feat: add RelationshipsBySourceOwnership to syft json output #1248

feat: add RelationshipsBySourceOwnership to syft json output #1248

Conversation

spiffcs commented Oct 5, 2022 • edited Loading

feat: add new optional source id to json schema

TODO:

github-actions bot commented Oct 5, 2022 • edited Loading

Benchmark Test Results

spiffcs Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

kzantow Oct 11, 2022

Choose a reason for hiding this comment

spiffcs Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

kzantow left a comment

Choose a reason for hiding this comment

kzantow Oct 11, 2022

Choose a reason for hiding this comment

spiffcs Oct 11, 2022

Choose a reason for hiding this comment

kzantow left a comment

Choose a reason for hiding this comment

spiffcs commented Oct 5, 2022 •

edited

Loading

github-actions bot commented Oct 5, 2022 •

edited

Loading

spiffcs Oct 11, 2022 •

edited

Loading

spiffcs Oct 11, 2022 •

edited

Loading