Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for Secure Boot #3097

Merged
merged 18 commits into from
Jul 13, 2023

Conversation

bcressey
Copy link
Contributor

@bcressey bcressey commented May 10, 2023

Issue number:

#2501

Description of changes:
The first set of commits brings GRUB up-to-date to fix any known flaws in Secure Boot support, adds shim as the initial EFI bootloader, and prepares the shim and grub binaries for modification in the final image build step.

packages: update grub
grub: move embedded public key into dedicated section
grub: add SBAT info
grub: add modules for GPG signature verification
tools: add script to replace the public key embedded in GRUB
packages: build shim
shim: embed placeholder for vendor cert

The second set of commits introduces the concept of a Secure Boot profile - all of the private keys, certificates, and configuration files necessary for Secure Boot signing - and automates the creation of a local profile. It also adds an image feature flag for Secure Boot support, along with OVF templates for VMware that include placeholders for EFI variables.

build: add scripts to create Secure Boot profiles
build: auto-create a local Secure Boot profile
build: pass Secure Boot profile secrets to builds
build: add Secure Boot OVF templates
build: add image feature flag for Secure Boot

The third set of commits extends buildsys (mostly rpm2img) and pubsys to do the required signing and registration for images, OVAs, and AMIs. This is where all of the above actually pays off.

build: sign artifacts for Secure Boot
pubsys: register AMIs with Secure Boot support

The fourth set of commits is developer-oriented. It enables the feature flag for *-dev variants, documents the new tools for generating Secure Boot profiles, and extends start-local-vm to support firmware overrides for testing the functionality.

start-local-vm: add options to override firmware
docs: document tools to generate Secure Boot Keys
variants: enable Secure Boot for dev builds

Testing done:
Verified that Secure Boot enabled images worked on:

  • AWS (for Nitro-based instance types)
  • VMware (in VMC-A)
  • metal (under QEMU)

Verified that older instance types could be launched from the same AMIs, and fell back to BIOS boot as expected.

Performed negative testing as well with Secure Boot enabled:

  • firmware refuses to boot shim if not signed with a trusted key
  • shim refuses to boot grub if not signed with a trusted key
  • grub refuses to load grub.cfg if the detached signature is invalid or from an untrusted key
  • grub refuses to load the Linux kernel if not signed with a trusted key

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@bcressey
Copy link
Contributor Author

^ force push fixes lint check failures 😿

packages/shim/shim.spec Show resolved Hide resolved
sbkeys/generate-aws-sbkeys Show resolved Hide resolved
tools/buildsys/src/builder.rs Show resolved Hide resolved
sbkeys/README.md Show resolved Hide resolved
sbkeys/README.md Outdated Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
.gitignore Show resolved Hide resolved
sbkeys/generate-aws-sbkeys Show resolved Hide resolved
sbkeys/README.md Outdated Show resolved Hide resolved
CODE_SIGN_KEY="arn:aws:kms:us-west-2:999999999999:key/bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
CONFIG_SIGN_KEY="arn:aws:kms:us-west-2:999999999999:key/cccccccc-cccc-cccc-cccc-cccccccccccc"

./generate-aws-sbkeys \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should pubsys be able to do this instead of a script? If so, wouldn't we want it to be possible to set and forget this by pointing Infra.toml at the right resources?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should pubsys be able to do this instead of a script? If so, wouldn't we want it to be possible to set and forget this by pointing Infra.toml at the right resources?

Conceptually, this script is somewhat closer to what we were trying to do with infrasys, where we create assets that can later be referenced in Infra.toml.

But I didn't see a good reason to try to standardize an "aws" shape and a "local" shape in Infra.toml, since the two profile types are very different and require different handling in rpm2img for signing to actually work.

rpm2img eventually does channel them into a mostly-shared code path, which requires correct NSS and GPG configurations so that pesign and gpg can "just work" later on. Even there, different arguments are required for the actual CLI invocation when signing.

What would help is if there were some sort of virtual smartcard support for local keys generated by OpenSSL. Then we could use the PKCS#11 code path consistently, and pubsys-setup or some other helper could have "configure smartcard for signing" as its contract, and rpm2img could just rely on a correctly-configured smartcard interface with standard key and token names.

AFAICT we'd have to write our own PKCS#11 provider with a "local" backend for that to work - like aws-kms-pkcs11 does for "aws" - which was more than I wanted to tackle here. With that provider in place, we'd need something like these scripts to generate the required files. The output would just look more like this:

❯ tree aws local
aws
├── config-sign.key
├── kms-sign.json
└── efi-certs.json
local
├── config-sign.key
├── local-sign.json
└── efi-certs.json

We could shift some of what the call to virt-fw-vars does into an implementation where pubsys-setup reads cert information from efi-certs.json and creating efi-vars.json.

I'm happy to backlog an issue for it; I could see it being useful for OOTBs for the same reason that it's useful that tuftool can abstract a bit over "local keys" vs. "KMS keys".

Makefile.toml Show resolved Hide resolved
variants/vmware-dev/template.ovf Show resolved Hide resolved
packages/grub/grub.spec Show resolved Hide resolved
tools/replace-grub-pubkey Outdated Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
sbkeys/generate-aws-sbkeys Show resolved Hide resolved
tools/start-local-vm Show resolved Hide resolved
tools/rpm2img Show resolved Hide resolved
tools/rpm2img Show resolved Hide resolved
Copy link
Member

@markusboehme markusboehme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry it took a while to get to, and get through! I needed to rebuild some context, but am happy to see the Secure Boot feature so polished! ✨

packages/grub/grub.spec Show resolved Hide resolved
packages/grub/grub.spec Outdated Show resolved Hide resolved
packages/grub/grub.spec Show resolved Hide resolved
tools/replace-grub-pubkey Outdated Show resolved Hide resolved
packages/shim/Cargo.toml Outdated Show resolved Hide resolved
Makefile.toml Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
tools/rpm2img Show resolved Hide resolved
PROVISIONING-METAL.md Show resolved Hide resolved
@bcressey
Copy link
Contributor Author

bcressey commented Jul 6, 2023

⬆️ force push:

I'll follow up with a different push for any functional changes or fixes.

@bcressey
Copy link
Contributor Author

bcressey commented Jul 7, 2023

⬆️ force push addresses most of the feedback.

@bcressey
Copy link
Contributor Author

bcressey commented Jul 7, 2023

⬆️ force push for whitespace fix

Copy link
Contributor

@foersleo foersleo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me, just the two minor nits that are not blocking anything.
I can right now however not fully grasp why the linter gets thrown off on the build jobs.

packages/shim/Cargo.toml Outdated Show resolved Hide resolved
@bcressey
Copy link
Contributor Author

bcressey commented Jul 7, 2023

I can right now however not fully grasp why the linter gets thrown off on the build jobs.

I noticed those failures just now and I'm equally mystified. I'll try the clippy run locally and see if I can repro.

@bcressey
Copy link
Contributor Author

bcressey commented Jul 7, 2023

⬆️ force push to document IAM permissions for generate-aws-sbkeys

@bcressey
Copy link
Contributor Author

bcressey commented Jul 7, 2023

⬆️ force push to fix the edition in the shim package

@bcressey
Copy link
Contributor Author

bcressey commented Jul 7, 2023

⬆️ force push rebases on develop with no other changes. I think I need the changes from #3228 on this branch to appease clippy, which seems to be confused by the newly cached artifacts.

Copy link
Member

@markusboehme markusboehme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔐 🐧

Copy link
Contributor

@stmcginnis stmcginnis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great overall. Just one minor thing with using a dependency of clap instead of clap itself for one of the arguments that I think we should clean up.

packages/shim/Cargo.toml Show resolved Hide resolved
tools/pubsys/src/aws/ami/mod.rs Outdated Show resolved Hide resolved
@bcressey
Copy link
Contributor Author

⬆️ force push makes a defensive copy of the firmware code on aarch64 also, to fix an error when no code or variables are specified:

❯ ./tools/start-local-vm --arch aarch64 --variant metal-dev --inject-file net.toml
...
qemu-system-aarch64: device requires 67108864 bytes, block backend provides 2097152 bytes

❯ ls -latr /usr/share/edk2/aarch64/
total 206404
-rw-r--r--. 1 root root 67108864 Feb 17 21:43 vars-template-pflash.raw
-rw-r--r--. 1 root root   786432 Feb 17 21:43 QEMU_VARS.fd
-rw-r--r--. 1 root root 67108864 Feb 17 21:43 QEMU_EFI-pflash.raw
-rw-r--r--. 1 root root  2097152 Feb 17 21:43 QEMU_EFI.fd
-rw-r--r--. 1 root root 67108864 Feb 17 21:44 QEMU_EFI-silent-pflash.raw
-rw-r--r--. 1 root root  2097152 Feb 17 21:44 QEMU_EFI.silent.fd
-rw-r--r--. 1 root root   806912 Feb 17 21:46 vars-template-pflash.qcow2
-rw-r--r--. 1 root root  2117632 Feb 17 21:46 QEMU_EFI-silent-pflash.qcow2
-rw-r--r--. 1 root root  2117632 Feb 17 21:46 QEMU_EFI-pflash.qcow2

I could also change the script to default to QEMU_EFI-silent-pflash.raw and vars-template-pflash.raw, but this approach seems more consistent with the x86_64 case.

@bcressey
Copy link
Contributor Author

⬆️ force push removes %changelog from shim.spec

@markusboehme
Copy link
Member

Recent pushes still look good to me. 👍 Restarted a CI build that failed due to timing out on the network.

@bcressey
Copy link
Contributor Author

⬆️ force push rebases on b038795, prior to xfs feature merge, so the next rebase diff will better show the conflict resolution.

bcressey and others added 18 commits July 13, 2023 17:15
Signed-off-by: Ben Cressey <bcressey@amazon.com>
For the EFI build, the Secure Boot config signing key is embedded into
the GRUB image so that GRUB can verify the signature of any files it
loads, e.g. its configuration. By default, the public key is embedded
into a section alongside various executable GRUB modules. This set of
patches instead moves the key into its own dedicated section in the
image, where it can easily be replaced with tools such as objcopy or dd
to reuse a set of binary artifacts with different Secure Boot keys.

Signed-off-by: Markus Boehme <markubo@amazon.com>
shim expects to find an SBAT section in GRUB, and will not continue
booting if it is missing.

Bottlerocket's build of GRUB is downstream from Amazon Linux, which
is downstream from Fedora and RHEL. However, most of GRUB's modules
are not included in the BIOS and EFI images, and the configuration
file built into the image disables loading any other modules. Hence
it's not a given that vulnerabilities in either upstream should lead
to SBAT-based revocations.

Bottlerocket carries quite a few out-of-tree patches which add the
`gpt` and `gptprio` modules, and consequently has an unenviable, but
useful, claim to its own vendor entry. Clearly the vast majority of
GRUB development happens elsewhere, and the use of a distro-specific
vendor entry is not meant to imply otherwise.

There are no current plans for Bottlerocket to participate in the
wider ecosystem of Secure Boot for Linux distributions, by way of a
Microsoft-signed shim, so the choice of SBAT metadata is not relevant
elsewhere.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Build the GRUB EFI image with the modules needed for GPG signature
verification, and embed a large placeholder public key so it can be
replaced in the final stage of the build.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
The newly added replace-grub-pubkey script allows to replace the public
key embedded in a GRUB image. After replacing the key, the image will be
properly signed again.

Example invocation:

    ./replace-grub-pubkey grubx64.efi new-grub.pubkey ~/bottlerocket/sbkeys/local/

The script assumes that new-grub.pubkey is a public key file in GPG
format containing a single public key.

Signed-off-by: Markus Boehme <markubo@amazon.com>
shim will now be used as the first-stage bootloader on EFI platforms,
and will handle part of the Secure Boot verification chain when that
feature is enabled.

Adjust the macros used by the grub package so that shim runs first,
and finds GRUB with the expected filename.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Add a large placeholder certificate to the shim binary to facilitate
replacement in the final stage of the build.

Note that shim is used as the first-stage UEFI bootloader for all
variants, but only enforces Secure Boot if the relevant EFI variables
are set, and will not use the vendor certificate otherwise.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
For Secure Boot, various certificates, keys, and configuration files
are needed to sign binaries and register images. Provide two scripts
to simplify the process of generating the correct artifacts.

The "local" version of the script is only meant for non-production
use, for example by individual developers or by automated CI testing,
where the variant builds must support the feature but do not need to
be maintained indefinitely.

The "aws" version of the script expects various AWS resources such as
private CAs and managed keys to be available. This can be costly and
only makes sense for official builds of supported variants.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Ensure that at least one Secure Boot profile is always available, for
cases where the variant has Secure Boot enabled.

This uses the "local" version of the script, since locally generated
keys cost nothing and are guaranteed to be available. This is similar
to the locally generated keys used by default for TUF repositories,
in that neither is suitable for long-term production use.

The individual Secure Boot profile and the directory where profiles
are stored can both be overridden, so that profiles can be stored in
a different location, such as another Git repository.

Profiles are checked for completeness, to allow the expected files to
evolve over time, for example to add support for a new platform that
expects EFI variables in a different format.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
`buildsys` needs to pass through files from the Secure Boot keys
profile in order to sign artifacts during the variant build step.

Those files might include an `aws-kms-pkcs11` configuration that uses
KMS for signing, which requires network access to be enabled for
variant builds.

On an EC2 instance, credentials from IMDS will be used automatically
by `aws-kms-pkcs11`, but otherwise they need to come from environment
variables or an AWS CLI configuration file.

Accordingly, `buildsys` now passes the most important of these AWS
environment variables as additional secrets.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Secure Boot is incompatible with existing variants, so a feature flag
is required to conditionally enable it.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
To support Secure Boot on VMware ESXi, the OVF templates need to be
configured for UEFI boot, and contain placeholders for EFI variables
to be populated at build time.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
When Secure Boot is enabled for the variant, ensure that EFI binaries
and the GRUB configuration are signed with the expected keys, and
replace the placeholder certificates and keys with the right ones.
When building an OVA, populate the template with the EFI variable
data.

Because GRUB will have an embedded GPG public key in this mode, it
will automatically verify the detached signature for every file read.
This is the desired behavior for `grub.cfg`, which contains sensitive
parameters like the dm-verity root hash.

However, it's redundant with the EFI binary verification performed by
the shim verifier. It would also prevent reading the "initrd" that
contains the Boot Config data for kernel command line parameters,
because this is generated on the local system and cannot be signed by
a trusted key.

Since this is the only remaining file to read that is not an EFI
binary, it's OK to disable signature checking inside the verified
`grub.cfg`.

Have GRUB reboot if the kernel cannot be loaded, and backstop this by
blocking edits to menu entries with an empty superusers group. This
prevents runtime modifications to the expected configuration, which
could otherwise be used to alter the dm-verity root hash.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Markus Boehme <markubo@amazon.com>
If Secure Boot is enabled for the variant, AMIs should be registered
with the UEFI boot mode along with the relevant EFI variable payload.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Secure Boot can't be added in a backwards-compatible way to existing
variants without breaking the downgrade path, but will be the default
for all newer variants.

To ensure the functionality is exercised, set the Secure Boot feature
flag in the *-dev variants, which are not intended for production use
and do not support downgrades.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Both new options are meant to help with Secure Boot testing.

The `firmware-code` option allows a custom edk2 build to be used, for
example with aarch64 where the AAVMF build packaged by Fedora doesn't
have the Secure Boot feature available.

The `firmware-vars` option populates the initial firmware variables
from a different file, and sets it to the correct size for aarch64.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Document the goal and chain of trust for Bottlerocket's Secure Boot
implementation.

Mention the Lockdown feature, since it's now enabled for all current
variants with the exception of `*-nvidia`.

Update the metal provisioning guide with a section on how to enable
Secure Boot.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
@bcressey
Copy link
Contributor Author

⬆️ force push rebases on develop with the XFS feature conflicts fixed.

Copy link
Contributor

@yeazelm yeazelm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@bcressey bcressey merged commit 68450ed into bottlerocket-os:develop Jul 13, 2023
38 checks passed
@bcressey bcressey deleted the secureboot-shenanigans branch July 13, 2023 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants