SBOM provenance artefact should be signed

[tellison]

The SBOM is used to determine the provenance authenticity and integrity of the Temurin binaries, and as such should be able to be verified by the consumer to be authentic.

This issue is to enhance the build process to include a digital signature of the SBOM artefact using the trusted code signing key.

This is an obligation of SLSA Level 2.

[tellison]

An example of the signing of the SBOM is given as a CycloneDX use-case

[tellison]

Unlike the detached signature of the runtime binary, the SBOM signature should be part of the SBOM itself, presumably in JSON signature format. Is that how you see it too @sxa ?

[sxa]

That isn't what I've done for the first pass, but that may be a good option for the subsequent releases.

[tellison]

What have you done? This will be part of the "contract" with consumers so best we get it right first time rather than plan on changing it (and expecting users to update their code later).

[sxa]

I've done it the same way as for the tarballs - put up a file with a .sig extension that can be verified in the same way as the tarballs.

[tellison]

Maybe check the CycloneDX CLI tool that can sign the BOM (and presumably in the required format). Available in a docker image.

[smlambert]

https://github.com/CycloneDX/cyclonedx-cli#sign-command

[tellison]

$ ./cyclonedx validate --input-file OpenJDK-sbom_aarch64_linux_hotspot_2022-11-18-03-30.json --input-format json --input-version v1_4 --fail-on-errors
Validating JSON BOM...
BOM validated successfully.

:-)

$ ./cyclonedx sign bom OpenJDK-sbom_aarch64_linux_hotspot_2022-11-18-03-30.json --key-file ~/.ssh/adoptopenjdk
Loading private key...
Only XML BOMs are currently supported for signing.

:-(

[smlambert]

We could generate both json and xml formats, but also we can check when that signing feature might be supported for json BOMs

[tellison]

The tool can convert from json to XML format, so

$ ./cyclonedx convert --input-file OpenJDK-sbom_aarch64_linux_hotspot_2022-11-18-03-30.json --input-format json --output-file sbom.xml --output-format xml
works as expected and there is a valid XML SBOM in sbom.xml
$ ./cyclonedx validate --input-file sbom.xml --input-format xml --input-version v1_4 --fail-on-errors
Validating XML BOM...
BOM validated successfully.

but signing still not working for me:

$ ./cyclonedx keygen
Generating new public/private key pair...
Saving public key to public.key
Saving private key to private.key
$ ./cyclonedx sign bom sbom.xml --key-file private.key 
Loading private key...
Loading XML BOM...
Generating signature...
Unhandled exception: System.Security.Cryptography.CryptographicException: Could not create hash algorithm object.
...

I'm going to chat to the CycloneDX folks and see if they can help.

[tellison]

See also this issue for signing JSON format files directly CycloneDX/cyclonedx-cli#260

The error I get when trying to sign XML files is already reported back in July, but no follow-up from the project.

[tellison]

Not making much progress on this. The CycloneDX tools don't do what we want, and it doesn't sound like that is going to change imminently.

Options as I see it are:

1. Look for another tool that will sign our SBOM
   There is nothing CycloneDX-specific about the signature, so we could use anything that signs our JSON file in JSON Signature Format (JSF), however, I've not found any such tool with a bit of a search.

2. Do the signature ourselves
   The spec for the signature properties are well-defined (see 7. Signature Creation in the JSF docs), so we could write code to generate them ourselves, but that seems undesirable as others must have done this already. It would require canonicalization of the JSON for example. :-( Option 1. is far more palatable

[smlambert]

From the JSF doc, they show sample Java code in the Usage in Applications section, that shows example of signing a json doc using https://github.com/cyberphone/openkeystore. We could potentially integrate that into our TemurinGenSBOM code as an interim solution while we await the fix in the CycloneDX CLI tool.

[tellison]

Thanks @smlambert that looks promising:

$ cd openkeystore/library/
$ ant build
Buildfile: /openkeystore/library/build.xml

build:

_jdktest:
   [delete] Deleting directory /openkeystore/library/dist

_preprocess:
    [mkdir] Created dir: /openkeystore/library/.tmp
     [copy] Copying 22 files to /openkeystore/library/.tmp
    [javac] Compiling 347 source files to /openkeystore/library/.tmp
    [mkdir] Created dir: /openkeystore/library/dist
      [jar] Building jar: /openkeystore/library/dist/webpki.org-libext-1.00.jar
      [jar] Building jar: /openkeystore/library/dist/webpki.org-webutil-1.00.jar
    [javac] Compiling 4 source files to /openkeystore/library/.tmp

_preprocess:

BUILD SUCCESSFUL
Total time: 1 minute 13 seconds

$ ls -l dist/webpki.org-libext-1.00.jar 
-rw-r--r-- 1 tellison tellison 1224183 Nov 24 17:03 dist/webpki.org-libext-1.00.jar

$ ant testjson
Buildfile: /openkeystore/library/build.xml

testjson:

_test:
    [mkdir] Created dir: /openkeystore/library/testout
    [junit] Testsuite: org.webpki.json.JSONTest
    [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.967 sec
    [junit] 
    [junit] Testcase: UnreadProperties took 0.015 sec
    [junit] Testcase: Whitespace took 0.008 sec
    [junit] Testcase: SingleLineCreation took 0.034 sec
    [junit] Testcase: DocumentCache took 0.015 sec
    [junit] Testcase: OuterArrays took 0.003 sec
    [junit] Testcase: PrettyPrinting took 0.004 sec
    [junit] Testcase: ParserPrimitives took 0.312 sec
    [junit] Testcase: JavaScriptMode took 0.001 sec
    [junit] Testcase: KeySerializing took 0.11 sec
    [junit] Testcase: Encryption took 2.381 sec
    [junit] Testcase: Signatures took 0.947 sec
    [junit] Testcase: ObjectInclusion took 0.001 sec
    [junit] Testcase: Operations took 0.001 sec

BUILD SUCCESSFUL
Total time: 5 seconds
$ 

Agreed that can be used in the TemurinGenSBOM code based on the example you linked to above.
Wouldn't need to be an interim solution either would it?

[smlambert]

Correct, I think it does not need to be interim.

[julian55455]

@smlambert can I take up this?

[andrew-m-leonard]

@julian55455 The above sample using openkeystore can be built as part of our CycloneDX java library, here:

temurin-build/sbin/build.sh

Line 708 in 9204887

buildCyclonedxLib() {

And the signing can take part similarly at the end of the SBOM generation, here:

temurin-build/sbin/build.sh

Line 800 in 9204887

# Print SBOM json

[andrew-m-leonard]

Although access to the necessary key might be an issue...

We will probably have to restructure the calls to the above, so that they are called from the pipeline scripts within a node stage on the signing node, maybe in a similar fashion to the gpgSign():
https://github.com/adoptium/ci-jenkins-pipelines/blob/505c5f44da16f3f71700e162cf2be78efd463ede/pipelines/build/common/openjdk_build_pipeline.groovy#L813

[andrew-m-leonard]

@julian55455 So my thoughts are we could extend our TemurinGenSBOM.java to have a new "sign" operation, so add to this class: https://github.com/adoptium/temurin-build/blob/master/cyclonedx-lib/src/temurin/sbom/TemurinGenSBOM.java
a new sign(PrivateKey privateKey, PublicKey publicKey) method that basically implements the example here.

Then in sbin/build.sh add a new function signSBoM(), that builds the above CycloneDX TemurinGenSBOM.java class (call

temurin-build/sbin/build.sh

Line 708 in 0c31102

buildCyclonedxLib() {

), then calls the sign(private, public) operation you added to TemurinGenSBOM.

The signSBom() function in sbin/build.sh will need calling on the correct node with the keys, to do that, we can do something similar to ASSEMBLE_EXPLODED_IMAGE for Mac, where the ci-jenkins-pipelines code calls sbin/build.sh with a special mode. So we could add a new BUILD_CONFIG param "SIGN_SBOM", in the sbin/build.sh main procedure it would be something like:

loadConfigFromFile
fixJavaHomeUnderDocker
cd "${BUILD_CONFIG[WORKSPACE_DIR]}"

parseArguments "$@"

if [[ "${BUILD_CONFIG[SIGN_SBOM]}" == "true" ]]; then
    javaHome="$(setupAntEnv)"
    buildCyclonedxLib "${javaHome}"
    signSBoM $privateKey $publicKey
    unset javaHome
    exit 0
fi

The above will need a bit more thought...but something like that maybe the easiest.

[sxa]

Not making much progress on this. The CycloneDX tools don't do what we want, and it doesn't sound like that is going to change imminently.

FYI On the basis of this, I've made an adjustment to the signing job so that it will replicate what I did for the October release automatically for now until we can identify an alternate mechanism. If we can make something else work then we can use that instead, but we can use this as a failsafe to avoid breaking out claimed SLSA2 compliance.

[andrew-m-leonard]

@julian55455 this is going to require some more thinking, I suspect we could mimic what is done to sign windows & mac builds, using https://github.com/adoptium/temurin-build/blob/master/build-farm/sign-releases.sh
which gets called from here.

We could move the build of the TemurinGenSBOM.java out of sbin/build.sh to a common script, and call the same from a new build-farm/sign-sbom.sh script ...?

For signing there's a few things that all have to be in place:

1. Signing key from running on the private signing node
2. The java code built to perform the sbom signing
3. Calling the script from a job that brings together 1 & 2

[julian55455]

@andrew-m-leonard Do you mean

Modify the sign-releases.sh script to also include the signing of sboms or similarly create a new script

The modified sign-releases.sh would probably look like

#!/bin/bash

set -eu

BUILD_ARGS=${BUILD_ARGS:-""}
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

export SIGN_TOOL
export OPERATING_SYSTEM

# Check if the operating system is mac or windows
if [ "${OPERATING_SYSTEM}" == "mac" ] ; then
  EXTENSION="tar.gz"
elif [ "${OPERATING_SYSTEM}" == "windows" ] ; then
  EXTENSION="zip"
else
  echo "OS does not need signing ${OPERATING_SYSTEM}"
  exit 0
fi

# Find all files with the extension specified above
find workspace/target/ -name "OpenJDK*.${EXTENSION}" | while read -r file;
do
  # Check if the file is a debug image, test image, or sbom file
  case "${file}" in
    *debugimage*) 
      # Skip signing debug images
      echo "Skipping ${file} because it's a debug image"
      ;;
    *testimage*) 
      # Skip signing test images
      echo "Skipping ${file} because it's a test image"
      ;;
    *sbom*) 
      # Sign the sbom file
      echo "Signing ${file} because it's an sbom archive"
      # You can use the same bash "${SCRIPT_DIR}/../sign.sh" command as for the other files
      # Just make sure to pass the path to the sbom file as an argument
      bash "${SCRIPT_DIR}/../sign.sh" ${CERTIFICATE} "${file}"
      ;;
    *)
      # Sign other files
      echo "Signing ${file}"
      # shellcheck disable=SC2086
      bash "${SCRIPT_DIR}/../sign.sh" ${CERTIFICATE} "${file}"
      ;;
  esac
done

2. The java code built to perform the sbom signing

@andrew-m-leonard would this particularly be a java code not a script, ie I mean something like this

3. Calling the script from a job that brings together 1 & 2

This is okay

cc @zdtsw @smlambert

[zdtsw]

to sum up discussion we had in today's call:

1. start getting familiar with "ant" build system and what we already have in the temurin-build/cyclonedx-lib
2. try to add a new flag "--sign" into cycloneedx-lib's code and be able to compile it into binary temurin-gen-sbom.jar
3. add a new script into build-farm similar to sign-releases.sh but doing the sign for sbom (e.g sign-sbom.sh)
4. in sign-sbom.sh the main logic is to copy temurin-gen-sbom.jar and all the sbom jsons to the running node, and do ${javaHome}"/bin/java -cp "${classpath}" temurin.sbom.TemurinGenSBOM --sign <sbom.jsons>
5. in openjdk_build_pipeline.groovy add new function (e.g SBOMsign()) to trigger new jenkins job (e.g release/sign_sbom) and add caller of SBOMsign() probably somewhere around gsgSign(). also need to add post sign copy artifacts to get these signed files back to pipeline
6. create new jenkins job release/sign_sbom and allocate node where we have the signing key available.

[andrew-m-leonard]

Think a bit further ahead, I think the following next high-level steps are needed:

1. Move buildCyclonedxLib() from

   temurin-build/sbin/build.sh

   Line 708 in 72a91c4

   buildCyclonedxLib() {

   to https://github.com/adoptium/temurin-build/blob/master/sbin/common/sbom.sh
2. Update cyclonedx-lib/build.xml to clone openkeystore and build as part of the "build" ant target
3. Add a new function called signSBOM() to sbin/common/sbom.sh, which calls the new "--sign" operation of TemurinGenSBOM.java added above
4. Create a new build-farm/sign-sbom.sh script which:
   • imports ../sbin/common/sbom.sh
   • Takes Jenkins job parameters privateKey, publicKey, inputSbom, outputSignedSbom
   • calls buildCyclonedxLib()
   • calls signSBOM(PrivateKey, PublicKey, inputSbom, outputSignedSbom)
5. Create a new ci.adoptopenjdk.net Jenkins job build-scripts/release/sign_sbom.sh
   • node label: <certificate key holding node>
   • pipline script: build-farm/sign-sbom.sh <sbom.json> <sbom.json.signed>
6. Plumb a build job call to build-scripts/release/sign_sbom.sh into about here pipelines/build/common/openjdk_build_pipeline.groovy

[zdtsw]

• calls buildCyclonedxLib()

assume, we keep where buildCyclonedxLib() as-is to generate sbom json but extended with openkeystore ?
if move it into new sign-sbom.sh, the current generateSBoM() wont work.
or unless we want to call buildCyclonedxLib() twice: once before generateSBoM() once in sign-sbom.sh?

[andrew-m-leonard]

• calls buildCyclonedxLib()

assume, we keep where buildCyclonedxLib() as-is to generate sbom json but extended with openkeystore ? if move it into new sign-sbom.sh, the current generateSBoM() wont work. or unless we want to call buildCyclonedxLib() twice: once before generateSBoM() once in sign-sbom.sh?

So it is building the same thing, if the --signSbom occurs on the same node, then it will already be built and the ant make will just do nothing..

[zdtsw]

if I do not misunderstand,
either openkeystore will do a release of webpki.org-libext-*.jar which we can download and use it when building our TemurinGenSBOM,
or we need to git clone source of openkeystore and do "ant build" locally, then put the jar in cyclonedx-lib/build/jar
since the former is not available, we have to do the "build from source"

[andrew-m-leonard]

if I do not misunderstand, either openkeystore will do a release of webpki.org-libext-*.jar which we can download and use it when building our TemurinGenSBOM, or we need to git clone source of openkeystore and do "ant build" locally, then put the jar in cyclonedx-lib/build/jar since the former is not available, we have to do the "build from source"

Yes, openkeystore does not have any releases, so we will clone and build from source

[andrew-m-leonard]

PR: #3243

[andrew-m-leonard]

Now that PR #3243 is nearing completion, the SBOM signing Intergration needs adding to complete this issue: adoptium/ci-jenkins-pipelines#610