Start emitting multi-arch SBOMs for `SPDX` with `ko` #743

mattmoor · 2022-07-01T17:42:57Z

Tweaking some things to prepare for the meat of: #655

codecov-commenter · 2022-07-01T17:45:02Z

Codecov Report

Merging #743 (5fd12eb) into main (d17aca8) will decrease coverage by 0.25%.
The diff coverage is 27.53%.

@@            Coverage Diff             @@
##             main     #743      +/-   ##
==========================================
- Coverage   51.38%   51.13%   -0.26%     
==========================================
  Files          44       44              
  Lines        3295     3317      +22     
==========================================
+ Hits         1693     1696       +3     
- Misses       1390     1408      +18     
- Partials      212      213       +1

Impacted Files	Coverage Δ
pkg/commands/deps.go	`13.08% <0.00%> (ø)`
pkg/build/gobuild.go	`60.10% <30.15%> (-1.35%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d17aca8...5fd12eb. Read the comment docs.

imjasonh

Looks good so far

mattmoor · 2022-07-01T21:47:08Z

Ok, I pushed what I have so far. I haven't tried things yet to make sure they work, but I'll do that now.

mattmoor · 2022-07-01T21:51:06Z

Oh fun, I think there are some Go 1.18 problems :-/

mattmoor · 2022-07-01T21:52:06Z

Here's an example of an image SBOM with the latest refactoring:

{
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "sbom-sha256:8955c18a6362f783536ec377fa0508646102895d89c08ee9d2f186c26e949d85",
  "spdxVersion": "SPDX-2.2",
  "creationInfo": {
    "created": "2022-07-01T02:27:39Z",
    "creators": [
      "Tool: ko (devel)"
    ]
  },
  "dataLicense": "CC0-1.0",
  "documentNamespace": "http://spdx.org/spdxdocs/kosha256:8955c18a6362f783536ec377fa0508646102895d89c08ee9d2f186c26e949d85",
  "documentDescribes": [
    "SPDXRef-Package-sha256-8955c18a6362f783536ec377fa0508646102895d89c08ee9d2f186c26e949d85"
  ],
  "packages": [
    {
      "SPDXID": "SPDXRef-Package-sha256-8955c18a6362f783536ec377fa0508646102895d89c08ee9d2f186c26e949d85",
      "name": "sha256:8955c18a6362f783536ec377fa0508646102895d89c08ee9d2f186c26e949d85",
      "versionInfo": "",
      "filesAnalyzed": false,
      "licenseDeclared": "NOASSERTION",
      "licenseConcluded": "NOASSERTION",
      "downloadLocation": "",
      "copyrightText": "NOASSERTION",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE_MANAGER",
          "referenceLocator": "pkg:oci/image@sha256:8955c18a6362f783536ec377fa0508646102895d89c08ee9d2f186c26e949d85",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-github.com.google.ko-(devel)",
      "name": "github.com/google/ko",
      "versionInfo": "",
      "filesAnalyzed": false,
      "licenseDeclared": "NOASSERTION",
      "licenseConcluded": "NOASSERTION",
      "downloadLocation": "https://github.com/google/ko",
      "copyrightText": "NOASSERTION",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE_MANAGER",
          "referenceLocator": "pkg:golang/github.com/google/ko@(devel)?type=module",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-github.com.google.go-containerregistry-v0.10.0",
      "name": "github.com/google/go-containerregistry",
      "versionInfo": "v0.10.0",
      "filesAnalyzed": false,
      "licenseDeclared": "NOASSERTION",
      "licenseConcluded": "NOASSERTION",
      "downloadLocation": "https://proxy.golang.org/github.com/google/go-containerregistry/@v/v0.10.0.zip",
      "copyrightText": "NOASSERTION",
      "checksums": [
        {
          "algorithm": "SHA256",
          "checksumValue": "a9dfdfbf69d06a319925e9da35c75a82197048f8d0d0da6137d87302baf65968"
        }
      ],
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE_MANAGER",
          "referenceLocator": "pkg:golang/github.com/google/go-containerregistry@v0.10.0?type=module",
          "referenceType": "purl"
        }
      ]
    }
  ],
  "relationships": [
    {
      "spdxElementId": "SPDXRef-DOCUMENT",
      "relationshipType": "DESCRIBES",
      "relatedSpdxElement": "SPDXRef-Package-sha256-8955c18a6362f783536ec377fa0508646102895d89c08ee9d2f186c26e949d85"
    },
    {
      "spdxElementId": "SPDXRef-Package-sha256-8955c18a6362f783536ec377fa0508646102895d89c08ee9d2f186c26e949d85",
      "relationshipType": "CONTAINS",
      "relatedSpdxElement": "SPDXRef-Package-github.com.google.ko-(devel)"
    },
    {
      "spdxElementId": "SPDXRef-Package-github.com.google.ko-(devel)",
      "relationshipType": "DEPENDS_ON",
      "relatedSpdxElement": "SPDXRef-Package-github.com.google.go-containerregistry-v0.10.0"
    }
  ]
}

mattmoor · 2022-07-01T21:52:41Z

Here's an example of a multi-arch SBOM with this change:

{
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "sbom-sha256:3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6",
  "spdxVersion": "SPDX-2.2",
  "creationInfo": {
    "created": "2022-07-01T02:27:39Z",
    "creators": [
      "Tool: ko (devel)"
    ]
  },
  "dataLicense": "CC0-1.0",
  "documentNamespace": "http://spdx.org/spdxdocs/kosha256:3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6",
  "documentDescribes": [
    "SPDXRef-Package-sha256-3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6"
  ],
  "packages": [
    {
      "SPDXID": "SPDXRef-Package-sha256-3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6",
      "name": "sha256:3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6",
      "versionInfo": "",
      "filesAnalyzed": false,
      "licenseDeclared": "NOASSERTION",
      "licenseConcluded": "NOASSERTION",
      "downloadLocation": "",
      "copyrightText": "NOASSERTION",
      "checksums": [
        {
          "algorithm": "SHA256",
          "checksumValue": "3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6"
        }
      ],
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE_MANAGER",
          "referenceLocator": "pkg:oci/index@sha256:3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6",
          "referenceType": "purl"
        }
      ]
    }
  ],
  "relationships": [
    {
      "spdxElementId": "DocumentRef-amd64-image-sbom:SPDXRef-Package-sha256-f03f678ab9cefa198a2334c4ea1e4d740c9b9686643db54b0053a80e6d5041e9",
      "relationshipType": "VARIANT_OF",
      "relatedSpdxElement": "SPDXRef-Package-sha256-3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6"
    },
    {
      "spdxElementId": "DocumentRef-arm64-image-sbom:SPDXRef-Package-sha256-569a8f91aed6b872d3e0c00f339b834ba671b2d04da80a14823f0db381890928",
      "relationshipType": "VARIANT_OF",
      "relatedSpdxElement": "SPDXRef-Package-sha256-3b72bb7f7ce12357a17d40f5198708823333790dd20d460613a9cdaabc97bdd6"
    }
  ],
  "externalDocumentRefs": [
    {
      "checksum": {
        "algorithm": "SHA256",
        "checksumValue": "c066df5c956def2fd36a340d4dcf67a51bbaf9e31201ae43088406b5ff99681a"
      },
      "externalDocumentId": "DocumentRef-amd64-image-sbom",
      "spdxDocument": "pkg:oci/sbom@sha256:f03f678ab9cefa198a2334c4ea1e4d740c9b9686643db54b0053a80e6d5041e9"
    },
    {
      "checksum": {
        "algorithm": "SHA256",
        "checksumValue": "03789b9d525f55260ea83ee2c173a54c40fda3f99110fd328d270e8a01e8b3e1"
      },
      "externalDocumentId": "DocumentRef-arm64-image-sbom",
      "spdxDocument": "pkg:oci/sbom@sha256:569a8f91aed6b872d3e0c00f339b834ba671b2d04da80a14823f0db381890928"
    }
  ]
}

puerco · 2022-07-01T22:12:04Z

OK, its missing the external document refs, you need to merge this patch first:

#741

mattmoor · 2022-07-01T23:16:50Z

It's in there, I didn't see your PR until just now!

  "externalDocumentRefs": [
    {
      "checksum": {
        "algorithm": "SHA256",
        "checksumValue": "c066df5c956def2fd36a340d4dcf67a51bbaf9e31201ae43088406b5ff99681a"
      },
      "externalDocumentId": "DocumentRef-amd64-image-sbom",
      "spdxDocument": "pkg:oci/sbom@sha256:f03f678ab9cefa198a2334c4ea1e4d740c9b9686643db54b0053a80e6d5041e9"
    },
    {
      "checksum": {
        "algorithm": "SHA256",
        "checksumValue": "03789b9d525f55260ea83ee2c173a54c40fda3f99110fd328d270e8a01e8b3e1"
      },
      "externalDocumentId": "DocumentRef-arm64-image-sbom",
      "spdxDocument": "pkg:oci/sbom@sha256:569a8f91aed6b872d3e0c00f339b834ba671b2d04da80a14823f0db381890928"
    }
  ]

puerco · 2022-07-02T01:04:41Z

Ah sorry I ran it through the validator and it failed because it could not find the ref. I assumed it was not there. But I just saw that it's failing because the spdxDocument field has the purl in it. This prevents the validator from pulling the doc to complete the sbom data :(

puerco · 2022-07-02T01:12:00Z

If we cannot reliably get a link to the SBOMs of the arch images, we can simply not reference the sboms, only reference the image as a package.

We can think about it semantically as having two separate SBOM as opposed to this approach which is more akin to having one single SBOM broken into different files. From our discussion today and what I was discussing with @jonjohnsonjr I think I'm leaning more towards defining them this way.

mattmoor · 2022-07-02T14:52:09Z

Let me try getting the validator running against index SBOMs here. I was being a bit lazy since it doesn't have a registry and just uses ko deps.

mattmoor · 2022-07-02T14:56:28Z

oh it does 😈

mattmoor · 2022-07-02T15:09:48Z

From our discussion today and what I was discussing with @jonjohnsonjr I think I'm leaning more towards defining them this way

Just as packages? Ok, yeah I think that makes sense, and then we have the same problem for index -> image as we do for image -> other stuff's sboms.

mattmoor · 2022-07-02T15:37:23Z

Ok I switched it to using Packages, lmk if there’s a better ref type

mattmoor · 2022-07-02T16:17:56Z

Ok, so I realized my first crack at packages lost the platform information, so I brought it back in the version field of the package: https://github.com/google/ko/runs/7163036130?check_suite_focus=true#step:7:130

mattmoor · 2022-07-02T19:46:28Z

Here's a screenshot highlighting the digest of the image

mattmoor · 2022-07-02T19:49:52Z

Another reason I just thought of for why we need a non-native SPDX way of linking to other SBOMs is that we might have a mix of SPDX (derived) and CycloneDX (base). That's not the case here, but clearly we need to handle that for base images.

puerco · 2022-07-02T20:05:57Z

The platform info can be encoded in the purl. But I'm looking and for some reason, it only has an arch field!?!?!
https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#oci

We could propose another os field or simply cram the whole linux/arm/v8 field in there. I'd like to have it there as well because the purl is that key to link elements spread across sboms in spdx and cyclonedx.

mattmoor · 2022-07-03T14:18:16Z

Sure, just lmk what you'd like to see here and we can adjust!

mattmoor · 2022-07-04T17:08:31Z

Added the architecture to the pURLs as well.

puerco · 2022-07-04T20:43:22Z

I've opened package-url/purl-spec#179 to propose adding a os qualifier to the purl spec. Its the missing bit to complete the platform information.

imjasonh · 2022-07-05T17:11:09Z

internal/sbom/mod_1.18.go

@@ -32,3 +33,29 @@ func ParseBuildInfo(data string) (*BuildInfo, error) {
 	bi := BuildInfo(*dbi)
 	return &bi, nil
 }
+
+func modulePackageName(mod *debug.Module) string {


Should these be in some shared file that both 1.18 and pre-1.18 code can reuse?

I had to move them here because I wanted to use the *debug.Module type :-/

Boooo I guess what you had was fine. I hate this pre-1.18 and count down the days until we don't have to support it.

Yeah, I had no idea until CI blew up on me, and I had to contort things to be able to leverage module types.

Will be good to kill this with 🔥

imjasonh · 2022-07-05T17:12:26Z

internal/sbom/spdx.go

@@ -125,6 +170,151 @@ func GenerateSPDX(koVersion string, date time.Time, mod []byte, imgDigest v1.Has
 	return buf.Bytes(), nil
 }

+func extractDate(sii oci.SignedImageIndex) (*time.Time, error) {


Instead of extracting the date from the index we just built, can we pass the created time from gobuild.go into where we generate the SBOMs?

Or is this just to support ko deps?

That's what the image SBOM logic was doing, but extracting it from the image's config, which is frankly more accessible. I'll take a quick look at how hard it would be to pass it in without diverging the source of truth logic for setting these timestamps.

Looks doable, lemme plumb that through instead. 👍

Ugh, so it looks like we only use g.creationTime under certain circumstances, and we let the base image's creation time through otherwise, so for the index SBOM I think we'll end up needing some sort of extraction method to make that really work.

e.g.

empty := v1.Time{} if g.creationTime != empty { cfg.Created = g.creationTime }

I'm reverting to what I had, as I think it's cleanest modulo a bigger refactor of this code.

pkg/build/gobuild_test.go

imjasonh · 2022-07-05T17:14:47Z

pkg/commands/deps.go

@@ -30,6 +30,7 @@ import (
 	"github.com/google/go-containerregistry/pkg/v1/mutate"
 	"github.com/google/go-containerregistry/pkg/v1/remote"
 	"github.com/google/ko/internal/sbom"
+	"github.com/sigstore/cosign/pkg/oci/signed"


Is this signed package new? Why is there oci.SignedImage and oci/signed.Image?

No, it is a convenient way of turning a v1.Image into an oic.SignedImage with empty signatures et al

This plumbs through support for building multi-arch SPDX SBOMs largely based on Puerco's outline, but with a few adaptations. I added a few minor refactorings to try to enable consistency across the Image/Index SBOMs. Related: ko-build#655

…refactor bomRef to match goRef change.

imjasonh previously approved these changes Jul 1, 2022

View reviewed changes

mattmoor dismissed imjasonh’s stale review via 90b3ac6 July 1, 2022 21:46

mattmoor force-pushed the sbom-multi-arch-plumbing branch from cbb70cc to 90b3ac6 Compare July 1, 2022 21:46

mattmoor force-pushed the sbom-multi-arch-plumbing branch from 90b3ac6 to 6392dcb Compare July 1, 2022 21:49

mattmoor force-pushed the sbom-multi-arch-plumbing branch 3 times, most recently from fc472b5 to 64a1748 Compare July 1, 2022 23:52

mattmoor changed the title ~~[WIP] Start to do some plumbing for ko multi-arch SBOMs~~ Start emitting multi-arch SBOMs for SPDX with ko Jul 1, 2022

mattmoor mentioned this pull request Jul 2, 2022

Our SBOMs should reference the base image's SBOMs #654

Open

mattmoor force-pushed the sbom-multi-arch-plumbing branch 2 times, most recently from 1df8e75 to d19552e Compare July 2, 2022 16:11

mattmoor mentioned this pull request Jul 3, 2022

Add base image information to SPDX SBOMs #744

Merged

puerco mentioned this pull request Jul 4, 2022

Restructure SPDX SBOMs to microsboms, matching ko chainguard-dev/apko#264

Merged

imjasonh reviewed Jul 5, 2022

View reviewed changes

mattmoor force-pushed the sbom-multi-arch-plumbing branch 2 times, most recently from 5fd12eb to db6125e Compare July 5, 2022 18:50

mattmoor added 8 commits July 5, 2022 11:56

Plumb through multi-arch SBOMs for SPDX.

8ce4555

This plumbs through support for building multi-arch SPDX SBOMs largely based on Puerco's outline, but with a few adaptations. I added a few minor refactorings to try to enable consistency across the Image/Index SBOMs. Related: ko-build#655

Move ociRef by SPDX (only consumer), move goRef/bomRef into mod*.go, …

c6e55fa

…refactor bomRef to match goRef change.

Add DownloadLocation to all our packages

7f8fcdb

Fix merge conflict

a9a6ce5

Add an e2e test leg to try and validate the SBOM

bcd0348

Try just referencing child images as packages.

dd00b77

Add arch to the purls

7ab92b0

Drop underscore params

13ee953

mattmoor force-pushed the sbom-multi-arch-plumbing branch from db6125e to 13ee953 Compare July 5, 2022 18:57

imjasonh approved these changes Jul 5, 2022

View reviewed changes

mattmoor merged commit 2299765 into ko-build:main Jul 5, 2022

mattmoor deleted the sbom-multi-arch-plumbing branch July 5, 2022 19:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start emitting multi-arch SBOMs for `SPDX` with `ko` #743

Start emitting multi-arch SBOMs for `SPDX` with `ko` #743

mattmoor commented Jul 1, 2022

codecov-commenter commented Jul 1, 2022 •

edited

imjasonh left a comment

mattmoor commented Jul 1, 2022

mattmoor commented Jul 1, 2022

mattmoor commented Jul 1, 2022

mattmoor commented Jul 1, 2022

puerco commented Jul 1, 2022

mattmoor commented Jul 1, 2022

puerco commented Jul 2, 2022

puerco commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

puerco commented Jul 2, 2022 •

edited

mattmoor commented Jul 3, 2022

mattmoor commented Jul 4, 2022

puerco commented Jul 4, 2022

imjasonh Jul 5, 2022

mattmoor Jul 5, 2022

imjasonh Jul 5, 2022

mattmoor Jul 5, 2022

imjasonh Jul 5, 2022

mattmoor Jul 5, 2022

mattmoor Jul 5, 2022

mattmoor Jul 5, 2022

mattmoor Jul 5, 2022

mattmoor Jul 5, 2022

mattmoor Jul 5, 2022

imjasonh Jul 5, 2022

mattmoor Jul 5, 2022

Start emitting multi-arch SBOMs for SPDX with ko #743

Start emitting multi-arch SBOMs for SPDX with ko #743

Conversation

mattmoor commented Jul 1, 2022

codecov-commenter commented Jul 1, 2022 • edited

Codecov Report

imjasonh left a comment

Choose a reason for hiding this comment

mattmoor commented Jul 1, 2022

mattmoor commented Jul 1, 2022

mattmoor commented Jul 1, 2022

mattmoor commented Jul 1, 2022

puerco commented Jul 1, 2022

mattmoor commented Jul 1, 2022

puerco commented Jul 2, 2022

puerco commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

mattmoor commented Jul 2, 2022

puerco commented Jul 2, 2022 • edited

mattmoor commented Jul 3, 2022

mattmoor commented Jul 4, 2022

puerco commented Jul 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Start emitting multi-arch SBOMs for `SPDX` with `ko` #743

Start emitting multi-arch SBOMs for `SPDX` with `ko` #743

codecov-commenter commented Jul 1, 2022 •

edited

puerco commented Jul 2, 2022 •

edited