generate SLSA provenance by default #378

jepio · 2022-07-13T14:24:25Z

generate SLSA provenance by default

Enable SLSA provenance report generation by default for board packages (including toolchains). SLSA provenance generation happens through a profile.bashrc file in profiles/coreos/base, it is enabled by defining GENERATE_SLSA_PROVENANCE=true in make.conf (or environment).

Digging deeper, provenance generation is enabled during ./build_packages but also during ./build_toolchains, because that is where we get binpkgs for board glibc/gcc and their dependencies (some 23 packages) from. An alternative way is available: ./setup_board --nousepkg --nogetbinpkg --board=amd64-usr has been revived and can be used to skip ./build_toolchains.

How to use

Build a full image starting from toolchains (or sdk in case of container pipeline).

Testing done

http://jenkins.infra.kinvolk.io:8080/job/os/job/manifest/6068/cldsv/

Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)

jepio · 2022-07-13T14:30:01Z

The provenance report generation assumes that it's running from within the sdk docker container. Generated reports are in $ROOT/usr/share/SLSA.

pothos · 2022-07-13T15:55:25Z

I didn't fully understand it yet: Is calling setup_board after the SDK is built with the annotation data now necessary or not? Edit: E.g., do we need to add it to ci-automation/packages.sh or is this just for development purposes when the used SDK does not have the annotation data for the pre-existing board (packages)?

jepio · 2022-07-14T07:44:40Z

I didn't fully understand it yet: Is calling setup_board after the SDK is built with the annotation data now necessary or not?

If the toolchains are built with annotation, then no changes to the existing image build flow are necessary.

With un-annotated toolchains, it would be necessary to run ./setup_board --force --nousepkg --nogetbinpkg to generate the annotations. This didn't work when @t-lo tried it, because it had suffered from bitrot but it now works again.

Edit: E.g., do we need to add it to ci-automation/packages.sh or is this just for development purposes when the used SDK does not have the annotation data for the pre-existing board (packages)?

It's only for development with unannotated sdk containers. The current failure in the github workflow is because it does not rebuild sdk/toolchains and I didn't add that ./setup_board --nousepkg --nogetbinpkg --force call.

jepio · 2022-07-14T08:13:04Z

This is the current gap between slsa package list and the actual list of packages in the image. The torcx packages are understandable (and SLSA data is in /run/torcx/unpack/docker/usr/share/SLSA). intel-microcode is initramfs only so also ok. bootengine needs to be investigated, this is unexpected.

--- slsa.pkgs.list	2022-07-14 09:58:27.284946297 +0200
+++ image.pkgs.list	2022-07-14 09:58:33.784946094 +0200
@@ -75,7 +75,12 @@ app-crypt/tpmpolicy-20160404
 app-crypt/trousers-0.3.14-r2
 app-editors/vim-8.2.4328
 app-emulation/actool-0.8.11
+app-emulation/containerd-1.6.6
 app-emulation/cri-tools-1.19.0
+app-emulation/docker-20.10.17
+app-emulation/docker-cli-20.10.17
+app-emulation/docker-proxy-0.8.0_p20210525
+app-emulation/docker-runc-1.1.3
 app-emulation/xenserver-pv-version-6.2.0
 app-emulation/xenstore-4.14.2-r1
 app-misc/ca-certificates-3.80
@@ -115,6 +120,7 @@ dev-libs/libgcrypt-1.9.4
 dev-libs/libgpg-error-1.42
 dev-libs/libksba-1.5.1
 dev-libs/liblinear-243
+dev-libs/libltdl-2.4.6
 dev-libs/libnl-3.5.0
 dev-libs/libpcre2-10.40
 dev-libs/libpcre-8.44
@@ -228,6 +234,7 @@ sys-boot/efibootmgr-17
 sys-cluster/ipvsadm-1.27-r1
 sys-devel/binutils-config-5.4
 sys-devel/gcc-10.3.0-r2
+sys-firmware/intel-microcode-20220510_p20220508
 sys-fs/btrfs-progs-5.15.1
 sys-fs/cryptsetup-2.4.3-r1
 sys-fs/dosfstools-4.2
@@ -238,6 +245,7 @@ sys-fs/mdadm-4.2-r1
 sys-fs/multipath-tools-0.8.7
 sys-fs/quota-4.06
 sys-fs/xfsprogs-5.14.2
+sys-kernel/bootengine-0.0.38-r13
 sys-kernel/coreos-firmware-20220610
 sys-kernel/coreos-kernel-5.15.51
 sys-kernel/coreos-modules-5.15.51
@@ -267,6 +275,7 @@ sys-power/acpid-2.0.33
 sys-process/audit-3.0.6
 sys-process/lsof-4.94.0-r1
 sys-process/procps-3.3.17-r1
+sys-process/tini-0.19.0-r1
 virtual/awk-1
 virtual/editor-0-r3
 virtual/krb5-0-r1

krnowak · 2022-07-14T11:33:08Z

Probably because bootengine uses cros-workon, which sets EGIT_REPO_URI conditionally during unpack, instead of unconditionally at some top level. I think that this conditional setting of EGIT_REPO_URI variable is already forgotten by the time the post install hook is run. We could either fix cros-workon eclass in some way, or add a special case (detecting if CROS_WORKON_REPO, CROS_WORKON_PROJECT and maybe CROS_WORKON_COMMIT are set) to the script in coreos overlay.

jepio · 2022-07-14T14:34:41Z

Probably because bootengine uses cros-workon, which sets EGIT_REPO_URI conditionally during unpack, instead of unconditionally at some top level. I think that this conditional setting of EGIT_REPO_URI variable is already forgotten by the time the post install hook is run. We could either fix cros-workon eclass in some way, or add a special case (detecting if CROS_WORKON_REPO, CROS_WORKON_PROJECT and maybe CROS_WORKON_COMMIT are set) to the script in coreos overlay.

That part works fine, but bootengine is not actually present in the image, the entry in the packages.txt list comes from here: https://github.com/flatcar-linux/scripts/blob/main/build_library/build_image_util.sh#L278-L290. Bootengine only consists of dracut modules and the update-bootengine script, nothing used at runtime.

pothos · 2022-07-15T12:07:11Z

Should we merge it with the missing bootengine info or is there a way to add a workaround somewhere?

jepio · 2022-07-20T08:48:14Z

Should we merge it with the missing bootengine info or is there a way to add a workaround somewhere?

bootengine info is now included thanks to f143828. So this is in mergeable/reviewable shape.

t-lo · 2022-07-26T10:04:02Z

Picking this up; started reviewing the PR.

Build a full image starting from toolchains (or sdk in case of container pipeline).

If toolchains is all we need then it should be sufficient to start with ./build_sdk_container_image I think; will check.

@jepio could you please have a look at the remaining feedback on flatcar-archive/coreos-overlay#2027 ?

t-lo · 2022-07-26T11:31:33Z

If we're building provenance by default should we set GENERATE_SLSA_PROVENANCE in build_packages?
(I'm currently running my tests with this variable exported manually)

jepio · 2022-07-26T11:40:33Z

If we're building provenance by default should we set GENERATE_SLSA_PROVENANCE in build_packages? (I'm currently running my tests with this variable exported manually)

The change to ./setup_board sets this variable in $BOARD_ROOT/etc/portage/make.conf, so this will be applied by default but you would need a new SDK to test this locally.

t-lo · 2022-07-26T11:53:44Z

If we're building provenance by default should we set GENERATE_SLSA_PROVENANCE in build_packages? (I'm currently running my tests with this variable exported manually)

The change to ./setup_board sets this variable in $BOARD_ROOT/etc/portage/make.conf, so this will be applied by default but you would need a new SDK to test this locally.

Ah gotcha, thanks. I went with with ./build_sdk_container_image (which builds the container image based on the "classic" tarball, and builds toolchains in the process) and I do indeed see a GENERATE_SLSA_PROVENANCE="true" in /build/amd64-usr/etc/portage/make.conf.

jepio · 2022-07-26T14:25:56Z

Building here: http://jenkins.infra.kinvolk.io:8080/job/container/job/sdk/156/

t-lo · 2022-07-27T07:27:05Z

Building here: http://jenkins.infra.kinvolk.io:8080/job/container/job/sdk/156/

This currently fails at the sdk-container step in build_toolchains with fatal: unsafe repository ('/mnt/host/source/src/scripts' is owned by someone else) (i.e. here). This git security feature is what made me use cat in the provenance implementation.

(Oddly enough ./build_sdk_container worked locally for me tonight when reviewing / test building this very PR, so there might be other ways to fix this.)

t-lo · 2022-07-27T07:44:43Z

@jepio The PR looks good to me but currently fails CI. I did local testing and did not encounter the CI breakage (see previous comment).

Local tests done:

Rebuilt SDK container based on PR source tree, using flatcar-sdk-amd64-3277.0.0.tar.bz2 as base SDK tarball.
1. Verified toolchain provenance was generated in /build/amd64-usr/usr/share/SLSA/
Ran build_packages in custom SDK container from 1.
1. Verified package provenance was generated in /build/amd64-usr/usr/share/SLSA/

Ran build_image and image_to_vm.sh.

Started VM in qemu and saved a listing of /usr/share/SLSA

 ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" \
    core@localhost -p 2222 'ls -1 /usr/share/SLSA' > slsa-list.txt

Using flatcar_production_image_packages.txt from the image build and the above generated SLSA directory listing, checked for missing packages in the SLSA directory listing

for line in $(sed -e 's/:.*//' -e 's:/:_:' flatcar_production_image_packages.txt); do
    if grep -q $line slsa-list.txt; then
        continue
    else
        echo "Missing package $line in SLSA"
    fi
done

This came up empty.
2. Again using flatcar_production_image_packages.txt from the image build and the above generated SLSA directory listing, checked for for files present in the directory listing but not the package list

for line in $(sed -e 's:_:/:' -e 's/\.json.*//' slsa-list.txt); do
    if grep -q $line flatcar_production_image_packages.txt; then
        continue
    else
        echo "spurious slsa $line missing from package list"
    fi
done

This came up with a single hit, app-torcx/docker-20.10, which is entirely reasonable since this is a torcx package not contained in the base image packages list.

PR LGTM as far as I am concerned as soon as the CI issue is fixed.

jepio · 2022-07-27T08:06:27Z

This currently fails at the sdk-container step in build_toolchains with fatal: unsafe repository ('/mnt/host/source/src/scripts' is owned by someone else) (i.e. here). This git security feature is what made me use cat in the provenance implementation.

Ugh, I actually hit the same issue as well and that's why i open-coded this:
flatcar-archive/coreos-overlay@5d479a0#diff-45695e997e08abb9c418b7a41bb439644568ceec70dd7da7a1da188fb56c69f0R142

I thought something like this should work: git -c 'safe.directory=*' ... but for some reason it doesn't, even though git -c ... config -l confirms the value is set correctly.

I've pushed a fix and have restarted the CI.

`./setup_board --nousepkg --nogetbinpkg` currently fails with a circular dependency due to pulling in the whole systemd-cryptsetup-udev dependency chain. This is due to several issue: * `emerge --root=$ROOT --emptytree` considers ROOT=/ to also be empty, so it pulls in all host packages. This must've not always been the case. So we need to pipe the dependency package list through `egrep $ROOT` to filter only those that would get installed into the desired ROOT * if SYSROOT=/ and not SYSROOT=ROOT, then virtual/os-headers is missing from $ROOT package list * the final filter expression tries to previously looked like this: (=sys-devel/gcc|sys-devel/binutils-0.9) which also matches sys-devel/gcc-config and sys-devel/binutils-config, which are necessary dependencies. Rework the match expression to not filter those out.

install_cross_libs installs TOOLCHAIN_PKGS deps into /usr/$BOARD_CHOST, so that TOOLCHAIN_PKGS binpkgs can be built. We also need binpkgs for the TOOLCHAIN_PKGS deps so that we can install them into /build/$BOARD later together with TOOLCHAIN_PKGS. This is where the flow is currently broken. Due to a change in semantics, --emptyroot tries to rebuild host packages as well, and dropping it leaves --onlydeps which results in no binpkgs being built because they are already installed. We can solve resolve this by reusing the dependency list generated by install_cross_libs, and explicitly building those binpkgs. It is worth remarking that this flow of building the toolchain binpkgs through setup_board is not in use in Flatcar, because we normally build toolchains with catalyst. We are interested in reviving it because we want to build everything with SLSA provenance information.

Catalyst runs builds with copies of the portage/coreos overlays in a chroot, which prevents us from accessing the git metadata necessary to create provenance information. Copy some files over into the root_overlay used by the toolchains catalyst build so that provenance can be correctly captured.

This needs to be done in setup_board for ROOT=/build/$BOARD, but also in toolchain_util because basic toolchains packages are built through catalyst.

Prod images need libstdc++.so and other libraries produced by sys-devel/gcc build, but because we don't want all of gcc in the image, the binpkg is manually unpacked instead of installed with emerge. Make sure to preserve SLSA metadata when unpacking as well.

Some packages are currently missing from the /usr/share/SLSA directory compared to flatcar_production_image_packages.txt. For torcx packages, extract the reports from the torcx bundle when adding it to the rootfs. For initramfs packages, as a substitute we enumerate build dependencies of coreos-kernel (image_packages_implicit()). At this time these are bootengine and intel-microcode.

krnowak · 2022-08-01T12:48:16Z

Image build seems to have failed with (see the output):

[2022-07-27T16:26:01.414Z] INFO    build_image: Inserting additional SLSA file
[2022-07-27T16:26:10.159Z] INFO    build_image: Found sys-kernel/bootengine-0.0.38-r13::coreos at /build/arm64-usr/var/lib/portage/pkgs/sys-kernel/bootengine-0.0.38-r13.tbz2
[2022-07-27T16:26:10.159Z] qtbz2: ignoring parent with unknown repo in profile /mnt/host/source/src/third_party/coreos-overlay/profiles/coreos/amd64: 
[2022-07-27T16:26:10.159Z] qtbz2: ignoring parent with unknown repo in profile /mnt/host/source/src/third_party/coreos-overlay/profiles/coreos/amd64/sdk: 
[2022-07-27T16:26:10.159Z] tar: ./usr/share/SLSA: Not found in archive
[2022-07-27T16:26:10.160Z] tar: Exiting with failure status due to previous errors
[2022-07-27T16:26:10.160Z] ERROR   build_image: script called: build_image '--board=arm64-usr' '--group=developer' '--output_root=/home/sdk/build/images' '--only_store_compressed' '--torcx_root=/home/sdk/build/torcx' 'prodtar' 'container'

jepio · 2022-08-02T15:22:37Z

It failed because binary packages from 3305.0.0 were used... I'll try again.
New attempt here: http://jenkins.infra.kinvolk.io:8080/job/container/job/sdk/167/cldsv/

krnowak

The job succeeded, I think it's good to take it in. After resolving the conflict, that is. :)

Sync MAINTAINERS.md

jepio requested review from t-lo and a team July 13, 2022 14:24

jepio mentioned this pull request Jul 13, 2022

Record SLSA provenance data flatcar-archive/coreos-overlay#2027

Merged

1 task

jepio force-pushed the slsa-provenance-by-default branch 3 times, most recently from ae72a1a to f143828 Compare July 20, 2022 08:46

jepio force-pushed the slsa-provenance-by-default branch from 4988742 to 53e0691 Compare July 26, 2022 11:35

jepio force-pushed the slsa-provenance-by-default branch 3 times, most recently from c20e270 to c935bbb Compare July 26, 2022 13:09

jepio force-pushed the slsa-provenance-by-default branch from c935bbb to a5a8897 Compare July 27, 2022 08:01

jepio force-pushed the slsa-provenance-by-default branch 2 times, most recently from 371805a to 62b51ee Compare July 27, 2022 08:28

jepio added 5 commits July 27, 2022 12:59

enable GENERATE_SLSA_PROVENANCE for board ROOT

7c87bb6

This needs to be done in setup_board for ROOT=/build/$BOARD, but also in toolchain_util because basic toolchains packages are built through catalyst.

jepio force-pushed the slsa-provenance-by-default branch from 62b51ee to 658f79f Compare July 27, 2022 11:03

jepio force-pushed the slsa-provenance-by-default branch from 658f79f to bdd464d Compare August 2, 2022 15:08

krnowak reviewed Aug 5, 2022

View reviewed changes

krnowak approved these changes Aug 5, 2022

View reviewed changes

jepio force-pushed the slsa-provenance-by-default branch from bdd464d to 3f39f48 Compare August 15, 2022 11:38

jepio merged commit dc21dda into main Aug 15, 2022

jepio deleted the slsa-provenance-by-default branch August 15, 2022 11:39

jepio mentioned this pull request Aug 15, 2022

[RFE] Add supply chain provenance to Flatcar releases flatcar/Flatcar#782

Closed

t-lo pushed a commit that referenced this pull request Apr 17, 2023

Merge pull request #378 from flatcar/sync-maintainers

eaee1bd

Sync MAINTAINERS.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

generate SLSA provenance by default #378

generate SLSA provenance by default #378

jepio commented Jul 13, 2022

jepio commented Jul 13, 2022

pothos commented Jul 13, 2022 •

edited

Loading

jepio commented Jul 14, 2022

jepio commented Jul 14, 2022

krnowak commented Jul 14, 2022

jepio commented Jul 14, 2022

pothos commented Jul 15, 2022

jepio commented Jul 20, 2022

t-lo commented Jul 26, 2022

t-lo commented Jul 26, 2022

jepio commented Jul 26, 2022

t-lo commented Jul 26, 2022

jepio commented Jul 26, 2022

t-lo commented Jul 27, 2022 •

edited

Loading

t-lo commented Jul 27, 2022

jepio commented Jul 27, 2022

krnowak commented Aug 1, 2022

jepio commented Aug 2, 2022 •

edited

Loading

krnowak left a comment

generate SLSA provenance by default #378

generate SLSA provenance by default #378

Conversation

jepio commented Jul 13, 2022

generate SLSA provenance by default

How to use

Testing done

jepio commented Jul 13, 2022

pothos commented Jul 13, 2022 • edited Loading

jepio commented Jul 14, 2022

jepio commented Jul 14, 2022

krnowak commented Jul 14, 2022

jepio commented Jul 14, 2022

pothos commented Jul 15, 2022

jepio commented Jul 20, 2022

t-lo commented Jul 26, 2022

t-lo commented Jul 26, 2022

jepio commented Jul 26, 2022

t-lo commented Jul 26, 2022

jepio commented Jul 26, 2022

t-lo commented Jul 27, 2022 • edited Loading

t-lo commented Jul 27, 2022

jepio commented Jul 27, 2022

krnowak commented Aug 1, 2022

jepio commented Aug 2, 2022 • edited Loading

krnowak left a comment

Choose a reason for hiding this comment

pothos commented Jul 13, 2022 •

edited

Loading

t-lo commented Jul 27, 2022 •

edited

Loading

jepio commented Aug 2, 2022 •

edited

Loading