prepare foundations for Secure Boot and image resigning #2505
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zmrow
reviewed
Oct 21, 2022
webern
reviewed
Oct 22, 2022
There was a problem hiding this comment.
This is pretty big and I'm probably only 20% of the way through it. I wanted to save my work, sorry for what will be multiple reviews:
nits
- This commit comment
buildsys: refactor manifest parser into cratemisled me a little since what you are doing is making abuildsyslibrary crate that exposes amanifestmodule. - I wonder if "hint" is the best name for
publish_image_size_hint_gib. Maybepublish_image_minimum_size_giborpublish_image_desired_size_gib.
webern
reviewed
Oct 24, 2022
There was a problem hiding this comment.
I need to go through and self resolve or delete some of these things as I was figuring stuff out as I went along. I think the main, potentially useful feedback is that image-features are always assumed to be boolean (might we want other value types in the future) and I think maybe you assume that the presence of the key in the toml file means true (although I might have filled a filter).
jpculp
reviewed
Oct 24, 2022
arnaldo2792
reviewed
Oct 26, 2022
webern
approved these changes
Oct 27, 2022
arnaldo2792
approved these changes
Oct 27, 2022
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
The larger goal of this change is to make it easier to archive the contents of a specific build, to allow it to be saved and restored at a later point in time or on a different machine. Now that `buildsys` can track and clean up symlinks from builds, move all symlink generation into the image build process. Since `buildsys` can also keep track of artifacts in subdirectories, switch to versioned directories to store the output of image builds. Previously, "latest" was a directory containing friendly symlinks to the most recent set of images. Now it is a symlink to the most recent versioned directory, and the friendly symlinks are created during the image build. Signed-off-by: Ben Cressey <bcressey@amazon.com>
This allows metadata about the variant to be parsed by other tools in a consistent way. In particular, `pubsys` will need access to fields related to published image sizes and whether the Secure Boot feature is enabled. Signed-off-by: Ben Cressey <bcressey@amazon.com>
bcressey
force-pushed
the
secureboot-foundations
branch
from
3500ae4
to
9612c3a
Compare
Allow manifests to specify a hint for the published image sizes, for the common case where additional storage space is desired on the data volume. A helper function is provided to calculate the sizes for the OS and data volumes, based on the image layout and the publish size hint. Signed-off-by: Ben Cressey <bcressey@amazon.com>
The larger goal of this change is to make image builds more hermetic, rather than relying on additional scripts to run afterwards to create files and symlinks. That allows the artifacts in the directory to be archived and extracted elsewhere more easily. The main historical blocker for generating OVAs along with VMDKs was that the published image sizes were specified through environment variables rather than in the variant manifest. Now that the manifest provides these values, they can be passed through to image builds. The bulk of this change consists of moving the OVA logic around, and dropping the unnecessary size checks. As part of this change, some OVF template fields are also renamed to standardize on the "os volume" designation for the primary full disk image. Previously this was called the "root volume" in places, which can lead to ambiguity because there is also a "root image" used for in-place updates. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Switch pubsys over to use the variant manifest to determine published AMI volume sizes, rather than relying on environment variables. This aligns AMIs and OVAs around the variant manifest as the source of truth for properties related to publication, which is a necessary step for Secure Boot support. As part of this migration, some fields and variables are renamed to standardize on the "os volume" designation for the primary full disk image. Previously this was called the "root volume" in places, which can lead to ambiguity because there is also a "root image" used for in-place updates. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Presently there is only one image feature flag, indicating whether GRUB has the `search` module built-in along with a config file that sets the "private" variable. For Secure Boot, we will end up with additional image-level feature flags. One will be for GRUB, to indicate whether to enforce a signed config file, and the other will be to enable UEFI Secure Boot. Since not all of these flags will necessarily be tied to GRUB, take this opportunity to migrate to a generic "image features" structure to track the flags before any more are added. Signed-off-by: Ben Cressey <bcressey@amazon.com>
This allows package builds to depend on image feature flags, which are set as build conditionals in the spec file. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Move the steps to handle migrations and the kernel archive out of the shared repobuild stage and into the respective stages. This lets the stage be reused by other targets without waiting for the additional inputs to be available. Signed-off-by: Ben Cressey <bcressey@amazon.com>
bcressey
force-pushed
the
secureboot-foundations
branch
from
9612c3a
to
5f8934f
Compare
webern
approved these changes
Nov 2, 2022
jpculp
approved these changes
Nov 2, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
#2486, #2501
Description of changes:
The commits here divide into three rough groups, though they're all related to the larger goals of being able to resign images and to add support for Secure Boot.
First up is adding support to
buildsysto track symlinks and subdirectories, and then moving the bulk of symlink creation into thebuild-varianttask. Images and symlinks are now placed into a versioned directory, rather than all together in a single directory per variant/arch combination. The ultimate goal here is to be able to cleanly archive and later unpack the full contents of a specific build, without worrying about excluding artifacts from other builds or recreating the (surprisingly fiddly and prolific) symlinks.There's still "latest" but it's now a symlink to the most recent versioned directory (which contains the symlinks alongside the artifacts) rather than a directory (which contains symlinks to artifacts in the parent directory). Scripts that rely on paths to symlinks, as well as tasks in
Makefile.toml, should still work. However, any process that relies on moving images into the "expected" directory, such as the GovCloud AMI registration workflow, will break and will need to be updated.The second set of changes turns the manifest parser into a crate, and makes the variant manifest the new source of truth for published image sizes.
buildsysandpubsysare then updated to consume this data. This allows thebuild-varianttask to generate OVAs directly, rather than relying on follow-up tasks inMakefile.toml, which again is in pursuit of the goal for a single task that's responsible for generating all artifacts and symlinks.What's lost here is the ability to override AMI sizes via environment variables at the
cargo make amistep, or to do the same for OVAs during thecargo makestep. I didn't feel this was worth the implementation effort, or the cost of duplicate logic, since for OVAs to support this it would be necessary for both thebuildandresigntasks to track the environment variables and the desired image layout, just to override a property that can now be set in the variant manifest.Next, the
grub-featuresproperty in the variant manifest is upgraded to the more generalimage-featuresproperty to store feature flags. Some of these will be GRUB-related (grub-set-private-var,grub-signed-config) and some will pertain to the image as a whole (uefi-secure-boot). Features are also passed into package builds so that they can be used for build conditionals.Finally there are a couple of minor fixes. The
repobuildstage in theDockerfileis streamlined and the parts of it that are specific tokmod-kitandmigrationsare moved into those stages to better separate concerns.aws-devandvmware-devreceive the ability to configure boot settings, andvmware-devmoves to the newer "unified" layout. Both are done under the general theory thatdevvariants are a laboratory for experimenting with new features and functionality, where backwards compatibility can be cheerfully ignored.Testing done:
Built lots of OVAs and AMIs. Verified that the images and symlinks were created correctly and that related tasks (
ami,repo,upload-ova) still found the expected inputs and produced the expected results in terms of published disk sizes.Confirmed that the boot settings feature works on
aws-devandvmware-dev.Here's a visual aid for how the artifacts and symlinks are created, before and after this change.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.