Add an SBOM file to our release artifacts

[nhorman]

We currently have an SBOM file, but its not at all useful, in that its just a template that downstream consumers might use to build their own sboms, but thats not really what an SBOM is for.

SBOMs provide a level of assurance that the release production process was done securely. OpenSSL produces source releases, for which we leverage git archive when done on a github CI runner, which we don't have direct control over (at least not the contents of the runner image), so what we really should be doing, is generating an SBOM on the fly, for which we validate the computed SHA256 sum of each file in the tarball against the SHA256 sum of the corresponding file from the source git tree (to ensure that tooling local to the system didn't modify the files during archive construction.

Replace the static sbom file with some additional ci work to do that, and make the sbom part of the release process so that downstream users can properly consume it.

[quarckster]

I believe the validation must be done before we push the tag. Perhaps make it a part of release-tools scripts. Besides, did you consider using https://github.com/anchore/sbom-action

[levitte]

Why before?

[quarckster]

If the validation fails the release will fail and we will need to push a new tag

[levitte]

??? And? This is no different from the release tarball itself.

[levitte]

Point is, make-release.yml was made to create release assets, yeah? Like the tarball. The SBOM is also a release asset, so why it should be prepare elsewhere, I do not understand.

[levitte]

... but, perhaps we want to have an SBOM producing script, like util/mktar.sh, and invoke that script from here? That would make a lot of sense.

[quarckster]

I'm talking about the validation and not about creating SBOM artifact

[levitte]

🤦🏼

Yeah, ok, that does make sense

[nhorman]

Please see my thoughts in openssl/project#1716

The goal in my mind is to provide a level of assurance that the archive we create as a release artifact matches what we expect to get from our git sources (i.e. we want to prevent an attacker who has managed to compromise our release CI runner from introducing a change during the period of time between when the CI runner clones the git tree and the time it creates the archive itself). This is the form of attack that was used in the sunburst attack on solarwinds and similar to that which was used in the lzma attack.

If we do the validation prior to pushing the tag, then that validation isn't useful, as there still exists a window of time in which we are trusting that the runner which creates that archive isn't compromised, during which such a code injection can take place.

Though I do agree that doing the validation at the same time as the the SBOM creation doesn't make sense (this PR was really just an effort to tinker with this a bit).

What I would propose instead is that we move the validation to a separate script (which I've now done) to be run during the release process, in which the artifacts pushed to the draft release are validated prior to publishing that release. Its a pain if something goes wrong, and yes we would have to push a new tag if a problem was found (and that problem was benign and not the result of malicious intrusion), but it does give us independent validation, in that it confirms for us at the individual source file level that each element in the SBOM has a sha256sum that matches both the contents of the archive, and the contents of the git tree.

To give some more concrete behavior to this, normally when I'm participating in a release, after the CI runner has created the draft release, I go in an update the draft release with release notes, etc. During that time I normally ask you @quarckster to validate that the signature asc file/sha1sum/sha256sum file pass a routine check, which you do and report back "no they don't", or more likely "yes, they do". What I'm suggesting here, is that, instead of verbally asking for such a check, we have people run a script, that, in addition to validating those signatures/etc, also interrogates the generated SBOM file, and confirms that, for each file in the archive, there is an entry in the SBOM and that the SBOM entry sha256sum matches the sha256sum of the corresponding file in the archive, and in the git tree at the corresponding tag.

[levitte]

Why do you do this? A previous step has already checked out the code in "$GITHUB_REF_NAME"

[nhorman]

see below, the desire is to do integrity checking of the sha256sum recorded in the SBOM file against both the contents of the tarball and the contents of the corresponding file of the git tree -i.e. if an attacker inserts code by (for instance modifying the git archive command), we can catch that here.

[levitte]

At this point, that point is pretty moot.

But if you really want to be safe against git archive corruption, you could unpack into ../sbom (to make it parallel to the original $GITHUB_REF_NAME), and then see the exit code of diff -rq ../$GITHUB_REF_NAME ../sbom/$GITHUB_REF_NAME.

Either way, you'll have to trust some tools. If git archive can be corrupted, so can sha256sum and diff.

[nhorman]

yes, you do have to trust some tools, thats why I'm proposing the workflow change in openssl/project#1716

Specifically that, as part of our release process, the two individuals that are reviewing the release need to download the draft release and run the validation in the script implemented by openssl/tools#231. We have to trust our infrastructure to some degree but we can preform a more rigorous validation of the result preformed by that infrastructure in multiple semi-random locations, the impact being that an intruder not only needs to compromise the infrastructure we trust, but also the systems of multiple (semi-unknown) people preforming the release.

[nhorman]

I really just made this draft pr to record some ideas surrounding sbom inclusion in our release artifacts after spending all day yesterday talking about it. We should probably have a meeting to discuss how best to do this.

[nhorman]

I suppose some explanation is warranted here to drive some conversation here around the solution we want for SBOM inclusion in our release artifacts, understanding that whats in this draft PR isn't ideal, and really just meant to capture some ideas for how we might implement this.

Background
I spent Wednesday at FedCertWeek where the topic of conversation was motivators for the use of SBOM data. The motivation that I took away from it was based on Tim Browns discussion of how the suburst attack was carried out. The attack itself impacted users of the solarwinds product, orion, and was done via an attack on one of their ephemeral VM's which are used to construct release artifacts.

The attackers managed to infiltrate the solarwinds network in such a way that the VM template image which was used to construct releases was altered, so that binary data was inserted into the orion release itself during the release process. As such, the compromised orion code never existed in their source repository and so was undetected. Similarly, the VM images used to build the release was considered trusted (its not public facing, and was stored behind the firewall of their corporate network). This is very similar to the lzma attack.

Takeaways
Based on the above, I think the addition of an SBOM offers us two main advantages:

1. It improves the security of our release process, in that it adds an additional check to validate the integrity of our generated release tarball at the source file level, moving us in the direction of further securing our release process.

2. It provides consumers of OpenSSL a link in for the supply chain, so that those consumers can identify product components used in their builds and deployments to identify products that are impacted by various vulnerabilities. Production of such SBOMS will (likely) soon become required in some industry spaces as per NIST SP 800-218 and other corollary requirements in other jurisdictions.

Implementation Thoughts

I think (2) is somewhat self-evident. The use of SBOMs is moving toward general adoption and requirement for consumers of OpenSSL in various spaces. I think its likely that customers who require FIPS certification will also need to adhere to various SBOM requirements in the near future, and so its prudent for us to include this as a release artifact.

How we achieve (1) is the more interesting question here. the securing of the release process is an exercise left to the producer of the artifact, and so its up to us to determine how best to do that. My personal opinion here is that better is the enemy of good, and so we should do what we can to secure the release pipeline and iterate on that to improve the process going forward. To that end, here's what I see as steps we might take to do so:

a) The releaser ci runner, on which we produce and sign tarballs should probably have some level of pre-run image integrity check. How to accomplish this, I don't know, and should certainly be left to a separate effort outside the scope of this one. doing so however would help insulate us against attack vectors such as those experienced by solarwinds and lzma.

b) We should build an sbom, using syft or a similar tool (i.e. the core of what this PR does). I agree that downloading syft dynamically when we do this is a lousy idea, As it exposes us to the proverbial next turtle down in the stack getting hacked (i.e. if syft gets hacked, it can impact our release process). Interestingly, the use of sbom-action as you note @quarckster doesn't really help here, as the action does exactly what we do in this workflow now, i.e. it downloads syft from the same url when the action runs). I think the better solution would be to manually install syft on our runner and trust that its present when the release action runs, allowing for (a) above to verify its integrity should we undertake that effort in the future.

c) We should validate the accuracy of the SBOM against both the contents of the extracted/generated tarball and the contents of the git tree from which it was generated at a file granularity. That is to say, we should, for each file included in the SBOM, we should compare its recorded sha256sum to that of both the file in the release tarball, and the corresponding file in the git tree that we generated the tarball from. Is this foolproof? No, absolutely not, but it does make it more difficult for an attacker to affect change in our source release without being detected.

d) After validation of (c) we should ensure that the initial git tree from which we have generated the release hasn't been modified - i.e. we should ensure that the state of the source tree hasn't changed between the time we cloned it, and the time we produced the tarball.

None of this is perfect of course. Theres lots of other stuff we could do, but this does feel to me like a step in the effort to address low hanging fruit in securing our release process.

Thoughts?