Capture CycloneDX JAR SHAs in SBoM #3538

sxa · 2023-11-16T18:20:28Z

Note to reviewers: please allow me to merge this when approved.

This is a test of including the CycloneDX jars (downloaded during the build process) into the SBoM.
There are several JAR files associated with the CycloneDX tool and this only covers one of them so is a "discussion starter" (hence raising this as a draft) as to how we should move forward if we want to capture this information in the SBoM. Is this the right place for it to be stored? Do we definitely need it? Would we exclude it if we downloaded this onto the machines via the playbooks? If we capture all of the JAR SHAs then what should the format be?

      {
        "name" : "FreeMarker",
        "version" : "N.A"
      },
      {
        "name" : "CycloneDX core java JAR SHA",
        "version" : "88193228f85a955127dc73e1c72efc9e08e18a01d227df47d0865dc20eceffd1"
      },
      {
        "name" : "Docker image SHA1",
        "version" : "adoptopenjdk/centos7_build_image@sha256:0dcc7d6c846a7ca4fa7814e0721c05e14a5a88f4bf67c43202586610
      }

Cc @andrew-m-leonard @netomi - this is part of satisfying #3535

References to the new Formula stuff in CycloneDX 1.5 used in this PR:

Signed-off-by: Stewart X Addison <sxa@redhat.com>

andrew-m-leonard · 2023-11-21T16:21:26Z

@sxa One thought would be do we need the other dependencies as well that cyclonedx pulls in jackson...
Then one method maybe to add an extrat ant target to the build.xml to dump the checksum's that are already in that file?

temurin-build/cyclonedx-lib/build.xml

Line 46 in c7f1ad1

checksum="88193228f85a955127dc73e1c72efc9e08e18a01d227df47d0865dc20eceffd1"

sxa · 2023-11-21T16:31:15Z

@sxa One thought would be do we need the other dependencies as well that cyclonedx pulls in jackson... Then one method maybe to add an extrat ant target to the build.xml to dump the checksum's that are already in that file?

Yeah I reckon I'd be ok with that as another potential option for retrieving the details.

Any thoughts on the format of the entries in the SBoM file?

andrew-m-leonard · 2023-11-22T17:12:58Z

Any thoughts on the format of the entries in the SBoM file?

I asked on the CycloneDX slack, I think this is where we should have these:
https://cyclonedx.org/docs/1.5/json/#formulation_items_components

Signed-off-by: Stewart X Addison <sxa@redhat.com>

sxa · 2023-11-30T18:32:36Z

I asked on the CycloneDX slack, I think this is where we should have these: https://cyclonedx.org/docs/1.5/json/#formulation_items_components

Looks like the formulation classes aren't available in our cyclonedx-core-java.jar, so I'm not sure how to do this.

Signed-off-by: Stewart X Addison <sxa@redhat.com>

stevespringett · 2023-11-30T20:25:54Z

@sxa Formulation support was added to v1.5 of the CycloneDX specification, which is supported in cyclonedx-core-java v8.0 and higher. If your version of the jar does not support formulation, its likely an older version. The recommendation would be to update the jar to the current release.

See: https://github.com/CycloneDX/cyclonedx-core-java/tree/master/src/main/java/org/cyclonedx/model/formulation

sxa · 2023-12-01T10:30:23Z

@sxa Formulation support was added to v1.5 of the CycloneDX specification, which is supported in cyclonedx-core-java v8.0 and higher.

Yeah looks like we're on 1.4 just now so will need to upgrade if we're going to use that field for this. I guess https://cyclonedx.github.io/cyclonedx-core-java/org/cyclonedx/model/formulation/Formula.html is the class I'll need for the Formulation section?

sxa · 2023-12-01T11:48:20Z

Current plan of action after discussion with @andrew-m-leonard:

Step 1: Update the version of CycloneDX to 8.0.3:

Note: Plan would be to merge phase 1 next week - ideally Monday morning - as doing something like this on a Friday afternoon is asking for trouble :-)

Update to to 8.0.3 (latest from https://github.com/CycloneDX/cyclonedx-core-java/tags) in here and update the SHA - this will pull the new one down into the cache in the build.getDependency job (Do we really need to be refreshing that every day?)
Update the SHA in the build.xml scripts where we pull from the getDependency job

The above are covered under #3558

Step 2: Add the support to generate a Formulation section in our SBoM):

(Note that this step can be done indepedent of step 1, but step 1 will need to be completed before we can merge step 2 as it will have a dependency on the new jars)

Make sure we can run an ant build against build.xml against the updated CycloneDX. Can be tested locally by updating the URL in build.xml to pull directly from maven instead of our jenkins job. Verify whether other updates to the other jars will be required too based on whether it builds correctly
Add support into TemurinGenSBOM.java for creating formulations (This PR)
Add support into build.sh for calling TemurinGenSBOM with the formulation options (Also this PR)
Add tests for the new formulations options into build.xml (Probably this PR again ...)
Get approval that the place this is going to be stored is valid and that we cover everything needed. I'm currently envisioning Using formulations -> Component (CycloneDX jars) -> Properties with a name/value pair for each jar+sha256sum. If anyone has a better idea, let me know :-)

Noting also that CycloneDX (unsurprisingly!) provides GPG signatures and we should look to validate those instead of relying on fixed SHA checksums in order to make the process of upgrading easier (and because it's a better way of verifying)

Signed-off-by: Stewart X Addison <sxa@redhat.com>

sxa · 2023-12-06T17:30:33Z

Note to reviewers: please allow me to merge this when approved.

Unless reviews explicitly object I'd like to merge this and then sort out the naming of the formulation entry (BomRef) in a subsequent PR to avoid accessing entry (0) in the lists.

netomi · 2023-12-07T07:17:13Z

cant access the pipeline test build.

sxa · 2023-12-07T10:57:41Z

cant access the pipeline test build.

The github check for JDK8u has equivalent output and prints out the SBoM at the end: https://github.com/adoptium/temurin-build/actions/runs/7117734755/job/19379150291?pr=3538

For reference, the subjobs kicked off by that pipeline should be visible here - look for ===METADATA OUTPUT=== in the log:

@andrew-m-leonard Do we need the non-release pipelines to be restricted access now that we've split out the release ones?

Signed-off-by: Stewart X Addison <sxa@redhat.com>

sxa · 2023-12-07T12:43:00Z

Unless reviews explicitly object I'd like to merge this and then sort out the naming of the formulation entry (BomRef) in a subsequent PR to avoid accessing entry (0) in the lists.

Latest push has resolved this and is using a BomRef of CycloneDX to identify the correct entry in the list

 "formulation" : [
    {
      "components" : [
        {
          "name" : "CycloneDX jar SHAs",
          "properties" : [
            {
              "name" : "commons-codec.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "commons-io.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "cyclonedx-core-java.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "github-package-url.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "jackson-annotations.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "jackson-core.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "jackson-databind.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "jackson-dataformat-xml.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "json-schema.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            },
            {
              "name" : "temurin-gen-sbom.jar",
              "value" : "ecc371d12808dfe76047f87f8235665d74dd6cf8ec12c41d052715a3fd79e0b5"
            }
          ],
          "type" : "framework"
        }
      ],
      "bom-ref" : "CycloneDX"
    }
  ]

A couple of quesions (Maybe for @stevespringett ?)

Is a type of framework suitable for the list here, or would application be better?
Is using bom-ref like this as an identifier for the Component list entry a valid thing to do, and is CyloneDX a good identifier for this purpose (Storing the CycloneDX jar details) Maybe CycloneDX formula to help ensure uniqueness?

Re-running full pipeline test at https://ci.adoptium.net/job/build-scripts/job/openjdk21-pipeline/230/console which has spawned:

The next three pushes were linter/comment fixes and do not impact functionality or invalidate those jobs.

Signed-off-by: Stewart X Addison <sxa@redhat.com>

andrew-m-leonard

lgtm

sxa · 2023-12-08T10:16:36Z

build runs are failing with an error compiling the TKG code:

     [javac] Compiling 27 source files to /__w/temurin-build/temurin-build/aqa-tests/TKG/bin
    [javac] /__w/temurin-build/temurin-build/aqa-tests/TKG/src/org/testKitGen/TestDivider.java:28: error: package org.json.simple does not exist
    [javac] import org.json.simple.JSONArray;

Should be unrelated to this PR

sxa added 2 commits November 16, 2023 13:00

sbom: Updates to include CycloneDX jar SHA

f72ff6c

Signed-off-by: Stewart X Addison <sxa@redhat.com>

Use correct path to cyclonedx jar downloads

0587ad8

Signed-off-by: Stewart X Addison <sxa@redhat.com>

github-actions bot added macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites labels Nov 16, 2023

sxa changed the title ~~SBOM cyclonedx shas~~ Capture CycloneDX JAR SHAs in SBoM Nov 16, 2023

Move CycloneDX SHAs to formulation section of SBoM

b3dcc76

Signed-off-by: Stewart X Addison <sxa@redhat.com>

github-actions bot added macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites and removed macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites labels Nov 30, 2023

Try to fix CycloneDX Formula class

824a411

Signed-off-by: Stewart X Addison <sxa@redhat.com>

github-actions bot added macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites and removed macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites labels Nov 30, 2023

sxa mentioned this pull request Dec 1, 2023

Update CycloneDX core jar to 8.0.3 for 1.5 spec #3558

Merged

github-actions bot added macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites and removed macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites labels Dec 1, 2023

SBoM: UPdate TemurinGenSBOM.java to build formulation support

28712a8

Signed-off-by: Stewart X Addison <sxa@redhat.com>

sxa force-pushed the sbom_cyclonedx_shas branch from d77d8f7 to 28712a8 Compare December 1, 2023 18:13

github-actions bot removed the macos Issues that affect or relate to the MAC OS label Dec 1, 2023

sbom: Used named bom-ref in formula object

3709a25

Signed-off-by: Stewart X Addison <sxa@redhat.com>

github-actions bot added macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites and removed macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites labels Dec 7, 2023

sxa force-pushed the sbom_cyclonedx_shas branch from a0bb853 to 09b8235 Compare December 7, 2023 12:57

github-actions bot added macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites and removed macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites labels Dec 7, 2023

Placate linter again

cf7f8ba

Signed-off-by: Stewart X Addison <sxa@redhat.com>

sxa force-pushed the sbom_cyclonedx_shas branch from 09b8235 to cf7f8ba Compare December 7, 2023 13:00

github-actions bot added macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites and removed macos Issues that affect or relate to the MAC OS testing Issues that enhance or fix our test suites labels Dec 7, 2023

andrew-m-leonard approved these changes Dec 7, 2023

View reviewed changes

karianna approved these changes Dec 11, 2023

View reviewed changes

sxa merged commit 380140f into adoptium:master Dec 11, 2023
8 of 25 checks passed

sxa mentioned this pull request Dec 11, 2023

TKG build failure on temurin-build Github actions checks adoptium/aqa-tests#4914

Closed

karianna mentioned this pull request Jan 8, 2024

fix labeler #3601

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capture CycloneDX JAR SHAs in SBoM #3538

Capture CycloneDX JAR SHAs in SBoM #3538

sxa commented Nov 16, 2023 •

edited

andrew-m-leonard commented Nov 21, 2023

sxa commented Nov 21, 2023

andrew-m-leonard commented Nov 22, 2023 •

edited

sxa commented Nov 30, 2023

stevespringett commented Nov 30, 2023

sxa commented Dec 1, 2023 •

edited

sxa commented Dec 1, 2023 •

edited

sxa commented Dec 6, 2023 •

edited

netomi commented Dec 7, 2023

sxa commented Dec 7, 2023 •

edited

sxa commented Dec 7, 2023 •

edited

andrew-m-leonard left a comment

sxa commented Dec 8, 2023 •

edited

Capture CycloneDX JAR SHAs in SBoM #3538

Capture CycloneDX JAR SHAs in SBoM #3538

Conversation

sxa commented Nov 16, 2023 • edited

andrew-m-leonard commented Nov 21, 2023

sxa commented Nov 21, 2023

andrew-m-leonard commented Nov 22, 2023 • edited

sxa commented Nov 30, 2023

stevespringett commented Nov 30, 2023

sxa commented Dec 1, 2023 • edited

sxa commented Dec 1, 2023 • edited

Step 1: Update the version of CycloneDX to 8.0.3:

Step 2: Add the support to generate a Formulation section in our SBoM):

sxa commented Dec 6, 2023 • edited

netomi commented Dec 7, 2023

sxa commented Dec 7, 2023 • edited

sxa commented Dec 7, 2023 • edited

andrew-m-leonard left a comment

Choose a reason for hiding this comment

sxa commented Dec 8, 2023 • edited

sxa commented Nov 16, 2023 •

edited

andrew-m-leonard commented Nov 22, 2023 •

edited

sxa commented Dec 1, 2023 •

edited

sxa commented Dec 1, 2023 •

edited

sxa commented Dec 6, 2023 •

edited

sxa commented Dec 7, 2023 •

edited

sxa commented Dec 7, 2023 •

edited

sxa commented Dec 8, 2023 •

edited