Experience with disk/directory encryption

[nandra]

Hello,

AFAIU from nvidia documentation NX (and other platform) have implemented SecureBoot functionality so FW produced by authority which owns private key only can be booted. I would like to move to step further to protect also IP for some firmware parts.

Was thinking about encrypting whole disk (or at least some directories which contains sensitive material). I was looking into meta-encrypted-storage which seems to provide kind of functionality. Would like to ask if anyone successfully integrate such functionality or maybe have some other ideas how to achieve this? Thanks a lot.

marek

[madisongh]

In my test distro I have a jetson-tx2-cboot machine configuration that I use for testing the secure boot and encryption features I've implemented on the TX2, which should be easy to make work with Xavier. I don't use a TPM, but the keystore trusted application I implemented provides a mechanism for having a per-device storage encryption passphrase that you can use for LUKS-encrypted partitions.

[nandra]

@madisongh thanks a lot for pointing me to this. I shortly checked and planning to add support for nx. Still have some questions:

• signing_server class point to localhost:port (do I need to install something locally?)
• it's not clear to me how password for disk encryption is created (I saw systemd unit which read it from keystore)
  do I need to fuse password manually to fuses ?
• can have more passwords to decrypt filesystem?

Thanks.

[madisongh]

I have an implementation of a signing server here.

The README for the Keystore TA outlines the process. The base passphrase is stored in the encrypted keyblob that you flash onto the device, which is decrypted at boot time (then hashed with the unique ID of the SoC).

You can have multiple passphrases for unlocking the filesystem. I think LUKS allows up to eight.

[nandra]

OK thanks a lot for info. Just last questio before I dig in I saw in repo eks.img (in bootfiles recipe). Which passphrase is used in this blob? Thanks

[madisongh]

Which passphrase is used in this blob?

You mean what's the plaintext of the passphrase? I don't remember any more. Once I verified that the mechanism worked, I promptly forgot what I set it to.

[nandra]

OK thanks a lot. I'll create new one (some dummy for testing purposes). I think I understand concept how LUKS decryption work what is still not clear to me where is cryptsetup called when creating root partition? It is done in some script? Many thanks.

[madisongh]

For that, I flash a special installer image that uses the scripts here to set up the encrypted partitions (which are named with the prefix crypt- in the partition layout XML file).

[outbackdingo]

Hi, Im seeing this and reading it, I looked at the distro, are you saying this in fact does secure-boot ? I also have an Xavier and am looking at secure-boot and full disk encryption. There isnt a lot of information on the test-distro so id assume its just for "testing changes", but it seems there is a bit more features enabled in it ... I did get an image built, curious what is in the image and what actually happens when flashed, and by far you have the knowledge in your head 👍

[madisongh]

are you saying this in fact does secure-boot ?

To be more accurate, the jetson-tx2-cboot machine configuration in test-distro (which I have now migrated to a newer distro forked off of the new OE4T demo distro, where the machine is called jetson-tx2-sb) builds images for my secure-boot-enabled TX2. Features for that machine are in support of creating a product that can pass a security review. There is a lot under the hood there, and more documentation would be better, but I haven't really had time to work on it. It is a complex topic.

Basically, though, the relevant features are:

• Creating the initial chain of trust by using the secure boot support - the L4T documentation covers the basics here, and I went with their recommended PKC (signing) + SBK (encryption) approach.
• LUKS for encrypting filesystems.
• A TrustZone application that leverages the encrypted keyblob (EKB) support NVIDIA provides for generating the passphrase for the LUKS-encrypted filesystems, along with a tool to create the EKB and a tool to retrieve the passphrase from the trusted app at runtime.
• A modified initramfs for retrieving the the passphrase to unlock the rootfs at boot time. Initially, this was a simple shell script, but I recently updated it to use systemd in the initramfs instead.
• Kernel configuration to require that all kernel modules be signed, plus a small patch to work around an issue I found with the Jetson hardware encryption engine driver.
• A system installer script that is used during initial flashing to encrypt and format the filesystems, then load the actual rootfs image. The script can also be used for system updates with a basic A/B rootfs selection, although the specific flash layout I use is also compatible with using Mender for system updates.
• A tool to assist with programming of the boot-related partitions during system installation, based on a BUP payload, since it's likely that the boot partitions will need different contents than what would be flashed initially for the installation.
• A tool for examining (and, to a limited degree, programming) the fuses via the sysfs entries exposed by the Jetson fuse driver.

Build-related features are:

• No secrets stored with the build metadata, nor available to the developer directly as files - all signing/encryption happens externally to the build process. (One exception to this is the EKB image, but that is pre-encrypted outside the build process.)
• Hooks for calling out to the signing/encryption service for the boot-related binaries (i.e., the BUP payloads for the Jetson bootloader update system), kernel modules, and Mender artifacts (when using Mender).
• A simple digital signing server that retrieves the necessary secrets and runs the signing commands for the above-mentioned items.

The images also include packages for testing the BSP and meta-tegra itself, and some features I've worked on for other projects, like using the PMIC watchdog timer and A/B failover since those projects have been headless, non-interactive devices that need to be running 24/7 with no provision for re-flashing via the USB OTG port.

There's more to building a secure product, of course, and whether any of the above will be useful to you will depend on what your product is, what other hardware and software you incorporate into it, how it will be used, how secure you need it to be, etc.

[dwalkes]

Thanks for porting the secureboot machine over to tegra-demo-distro @madisongh and thanks for starting/contributing to this thread @nandra and @outbackdingo

I just wanted to mention we are also really interested in Secureboot on Xavier NX and would love to coordiate work on and/or help fund efforts others are doing in this area. Perhaps we could use one of the communication mechanisms in #426 to get organized on this. Please weigh in on communcation preferences there if you haven't already and I'd like to post an update there next week.

[nandra]

Thanks @dwalkes . I'm interested in sharing effort of Secureboot on NX. I posted my response on pool few days ago ;).

[ichergui]

Hi guys,

Very interesting subject for me, I remember my talk with @madisongh in this issue about this particular configuration for platform Jetson TX2.

It depends on what we want to do (secure our device) and we have also to adapt with the design (SOFT + HW) of the Jetsons platform.

If we want to have a completed chain of trust, here is an interesting link

There is different option:

• Chain of trust with rootfs signed.
• Chain of trust with rootfs encrypted as @madisongh done via Jentson TX2 CBOOT configuration.
• Chain of trust with rootfs signed and encrypted.

Nvidia pushed new documentation/sample about the SE (Secure Engine) from L4T R32.4.2.

Please let me know if I can help.
It will be with a pleasure.

[outbackdingo]

i think theres a bit of confusion and some issues between tegra-demo-distro and
tegra-test-distro as tegra-test-demo README points to

Clone this repository:

$ git clone https://github.com/OE4T/tegra-demo-distro.git

where i think you really want us to be cloning $ git clone https://github.com/madisongh/tegra-test-distro.git

which README also states

Clone this repository:

$ git clone https://github.com/OE4T/tegra-demo-distro.git

anyway i cloned https://github.com/madisongh/tegra-test-distro.git and I am working from that, correct me if i am wrong!

[madisongh]

The tegra-test-distro repo under my username is a fork of OE4T/tegra-demo-distro which I'm using for my own testing and staging of changes for upstream.

[outbackdingo]

while perusing through this distro, i see a few things, firstly let me state, I have a Xavier NX devkit, with a fully encrypted rootfs via zymkey / zymbit module. It however doesnt include secure-boot processes, so im looking to move towards this process you have created. I also notice that you utilized mender, Honestly I think that meta-rauc, aktualizer and ota-updates-community edition from advancedtelmatica https://github.com/advancedtelematic/ota-community-edition would be far better suited for this, though im sure a bit of work to integrate fully. Lastly i see you mention tergra-test-disro and jetson-tx2-sb, where it appears you state " builds images for my secure-boot-enabled TX2" and later you mention flash a special installer image that uses the scripts here to set up the encrypted partitions, so im guessing its a matter of moving / duplicating the jetson-tx2-sb to the Xavier, and figuring out how to flash the installer image for the luks, am i on the right track logically? Im sure ive over simplified it. Just wanting to clarify the pieces for all of us wanting to do this on the Xavier.

[outbackdingo]

so... jumping in with 2 feet..... Ive copied/duplicated the work for jetson-tx2-sb to jetson-xavier-nx-devkit-sb,
i can bitbake demo-image-full -k with the single failure mentioned below.

I created a diff pasted below, i have 3 curious questions, one the build failed with exit 7 from 'curl --silent --fail -X POST "$@" "http://127.0.0.1:9999/$endpoint"' in tegra-test-distro/layers/meta-tegra/recipes-kernel/linux/linux-tegra_4.9.bb:do_package, So ill assume its looking for a local digsigserver deployment for signing ? correct ?

Next is is simply made a copy of flash_jetson-tx2-sb_custom.xml to custom-flash-layout/flash_jetson-xavier-nx-devkit-sb_custom.xml which contains the

cp /var/home/dingo/tegra-test-distro/layers/meta-testdistro/recipes-bsp/tegra-binaries/custom-flash-layout/flash_jetson-tx2-sb_custom.xml /var/home/dingo/tegra-test-distro/layers/meta-testdistro/recipes-bsp/tegra-binaries/custom-flash-layout/flash_jetson-xavier-nx-devkit-sb_custom.xml

and

cp layers/meta-testdistro/conf/machine/jetson-tx2-sb.conf layers/meta-testdistro/conf/machine/jetson-xavier-nx-devkit-sb.conf

and

cp -R ./meta-testdistro/recipes-bsp/tegra-binaries/bootfiles/jetson-tx2-sb ./meta-testdistro/recipes-bsp/tegra-binaries/bootfiles/jetson-xavier-nx-devkit-sb
where exists ls layers/meta-testdistro/recipes-bsp/tegra-binaries/bootfiles/jetson-xavier-nx-devkit-sb
eks.img

so the questions are what / how to generate this eks.img or will this one in three i copied actually work ?
and is the for the jetson-tx2-sb also useable for the jet-xavier--nx-devkit-sb

lastly my diff for review before i make a pull request that requires many more changes, id like to get a complete build and test it before i submit the pull request.

---

diff --git a/layers/meta-testdistro/recipes-bsp/tegra-binaries/bootfiles_1.0.bb b/layers/meta-testdistro/recipes-bsp/tegra-binaries/bootfiles_1.0.bb
index 3e32f64..db3f83b 100644
--- a/layers/meta-testdistro/recipes-bsp/tegra-binaries/bootfiles_1.0.bb
+++ b/layers/meta-testdistro/recipes-bsp/tegra-binaries/bootfiles_1.0.bb
@@ -3,7 +3,7 @@ LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

INHIBIT_DEFAULT_DEPS = "1"
-COMPATIBLE_MACHINE = "(jetson-tx2-sb)"
+COMPATIBLE_MACHINE = "(jetson-tx2-sb|jetson-xavier-nx-devkit-sb)"

SRC_URI = "file://eks.img"

diff --git a/layers/meta-testdistro/recipes-bsp/tegra-binaries/custom-flash-layout_1.0.bb b/layers/meta-testdistro/recipes-bsp/tegra-binaries/custom-flash-layout_1.0.bb
index 965d926..1af63e8 100644
--- a/layers/meta-testdistro/recipes-bsp/tegra-binaries/custom-flash-layout_1.0.bb
+++ b/layers/meta-testdistro/recipes-bsp/tegra-binaries/custom-flash-layout_1.0.bb
@@ -3,7 +3,7 @@ LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

INHIBIT_DEFAULT_DEPS = "1"
-COMPATIBLE_MACHINE = "(jetson-tx2-sb)"
+COMPATIBLE_MACHINE = "(jetson-tx2-sb|jetson-xavier-nx-devkit-sb)"

SRC_URI = "file://flash_${MACHINE}_custom.xml"

diff --git a/layers/meta-testdistro/recipes-bsp/tegra-binaries/tegra-bootfiles_32.4.3.bbappend b/layers/meta-testdistro/recipes-bsp/tegra-binaries/tegra-bootfiles_32.4.3.bbappend
index a7b4dba..0242137 100644
--- a/layers/meta-testdistro/recipes-bsp/tegra-binaries/tegra-bootfiles_32.4.3.bbappend
+++ b/layers/meta-testdistro/recipes-bsp/tegra-binaries/tegra-bootfiles_32.4.3.bbappend
@@ -1,10 +1,18 @@
EXTRADEPS = ""
EXTRADEPS_jetson-tx2-sb = "custom-flash-layout bootfiles"
+EXTRADEPS_jetson-xavier-nx-devkit-sb = "custom-flash-layout bootfiles"
DEPENDS += "${EXTRADEPS}"
PARTITION_FILE_jetson-tx2-sb = "${STAGING_DATADIR}/custom-flash-layout/${PARTITION_LAYOUT_TEMPLATE}"
+PARTITION_FILE_jetson-xavier-nx-devkit-sb = "${STAGING_DATADIR}/custom-flash-layout/${PARTITION_LAYOUT_TEMPLATE}"
MENDER_PARTITION_FILE_jetson-tx2-sb = "${STAGING_DATADIR}/custom-flash-layout/${PARTITION_LAYOUT_TEMPLATE}"
+MENDER_PARTITION_FILE_jetson-xavier-nx-devkit-sb = "${STAGING_DATADIR}/custom-flash-layout/${PARTITION_LAYOUT_TEMPLATE}"

do_install_append_jetson-tx2-sb() {
rm ${D}${datadir}/tegraflash/eks.img
install -m 0644 ${STAGING_DATADIR}/bootfiles/eks.img ${D}${datadir}/tegraflash/
}
+
+do_install_append_jetson-xavier-nx-devkit-sb() {

• rm ${D}${datadir}/tegraflash/eks.img
• install -m 0644 ${STAGING_DATADIR}/bootfiles/eks.img ${D}${datadir}/tegraflash/
  +}
  diff --git a/layers/meta-testdistro/recipes-testdistro/packagegroups/packagegroup-demo-base.bbappend b/layers/meta-testdistro/recipes-testdistro/packagegroups/packagegroup-demo-base.bbappend
  index 7c107ba..81befb0 100644
  --- a/layers/meta-testdistro/recipes-testdistro/packagegroups/packagegroup-demo-base.bbappend
  +++ b/layers/meta-testdistro/recipes-testdistro/packagegroups/packagegroup-demo-base.bbappend
  @@ -10,6 +10,10 @@ RDEPENDS_${PN}_append_jetson-tx2-sb = "
  tegra-sysinstall-tools
  "

+RDEPENDS_${PN}_append_jetson-xavier-nx-devkit-sb = " \

• tegra-sysinstall-tools
  +"

RRECOMMENDS_${PN}_append = "
kernel-module-hid-logitech-hidpp
kernel-module-hid-logitech-dj
diff --git a/repos/meta-tegra b/repos/meta-tegra
--- a/repos/meta-tegra
+++ b/repos/meta-tegra
@@ -1 +1 @@
-Subproject commit 1a4e3cf
+Subproject commit 1a4e3cf-dirty

[nandra]

assume its looking for a local digsigserver deployment for signing ? correct ?

lease see this reply: #421 (comment)

[outbackdingo]

assume its looking for a local digsigserver deployment for signing ? correct ?

lease see this reply: #421 (comment)

@nandra yeah i figured as much, just clarifying i was correct, trying to deploy that now locally on the build box... Thanks

[madisongh]

I also notice that you utilized mender, Honestly I think that meta-rauc, aktualizer and ota-updates-community edition from advancedtelmatica https://github.com/advancedtelematic/ota-community-edition would be far better suited for this, though im sure a bit of work to integrate fully.

The project I'm working on for my day job happened to be using Mender. I don't know the details of other OTA update systems, but hopefully it won't be too hard to adapt some of the underlying features for them.

am i on the right track logically?

Sounds that way.

[outbackdingo]

@nandra you making any further progress? Im still working through the signing server, a bit busy lately!

[nandra]

you making any further progress?

@outbackdingo waiting on final word from @dwalkes about some common discussion and sharing efforts and common understanding ;). @dwalkes do you have any output from pool? Thanks

[dwalkes]

@nandra I'll have something posted there by Monday.

[outbackdingo]

okay, i am very close to having a generated image, called jetson-xavier-nx-devkit-sb, having an issue with kernel-bup-payload, any ideas??

[2020-10-06 14:08:45 -0400] - (sanic.access)[INFO][127.0.0.1:43168]: POST http://127.0.0.1:9999/sign/tegra 500 13
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: DTBFILE=tegra186-quill-p3310-1000-c03-00-base.dtb
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: ODMDATA=0x1090000
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: LNXFILE=boot.img
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: BOARDID=3310
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: FAB=C04
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: fuselevel=fuselevel_production
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: localbootfile=boot.img
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: boardcfg=
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: CHIPREV=0
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: BOARDSKU=
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: BOARDREV=
[2020-10-06 14:15:06 -0400] [22558] [INFO] manifest line: BUPGENSPECS=fab=B00 fab=B02 fab=C04 fab=D00 fab=D01
[2020-10-06 14:15:06 -0400] [22558] [DEBUG] Setting: FAB=B00
[2020-10-06 14:15:06 -0400] [22558] [INFO] Running: ['tegra186-flash-helper', '--bup', '-u', '/tmp/tmpb7x9sv6j/rsa_priv.pem', 'flash.xml.in', 'tegra186-quill-p3310-1000-c03-00-base.dtb', 'jetson-xavier-nx-devkit-sb.cfg', '0x1090000', 'boot.img']
[2020-10-06 14:15:06 -0400] [22558] [WARNING] signing error, stdout:
stderr: ERR: flash variable set not defined

[2020-10-06 14:15:06 -0400] - (sanic.access)[INFO][127.0.0.1:43586]: POST http://127.0.0.1:9999/sign/tegra 500 13

[ichergui]

Hi @outbackdingo

Did you follow the README that you can find here ?

Could you check flashvars file if exists, the helper try to source it but it seems that the variable FLASHVARS needed is empty.

I think you have to copy the file from meta tegra to the BSP folder:

$ cd /path/to/meta-tegra/recipes-bsp/tegra-binaries/tegra-flashvars/jetson-xavier-nx/
$ sudo install -m 0644  flashvars \
  /opt/nvidia/L4T-32.2.3-tegra186/Linux_for_Tegra/bootloader/flashvars

@madisongh could you please confirm that ?

[outbackdingo]

Yeah followed the doc to the letter really, there is no mention of flashvars anywhere in it, and no its doesnt exist in my bootloader folder so ill move it there, and see how we go, you did also mention it looked like it was trying to sign for a TX2-based (tegra186) machine, rather than a Xavier-based machine (tegra194). That might be part of the problem. so next question is in what file are these variables set, remember i basically made a copy of the TX work for the Xavier, highly possible i missed something

[outbackdingo]

something appears off here? using L4t-32.4.3-tegra186, flashvars file is 0 byte

ls -al ../layers/meta-tegra/recipes-bsp/tegra-binaries/tegra-flashvars
total 40
drwxr-xr-x 10 dingo dingo 4096 Oct 4 13:41 .
drwxr-xr-x 11 dingo dingo 4096 Oct 4 13:41 ..
-rw-r--r-- 1 dingo dingo 0 Oct 4 13:41 flashvars
drwxr-xr-x 2 dingo dingo 4096 Oct 4 13:41 jetson-nano-emmc
drwxr-xr-x 2 dingo dingo 4096 Oct 4 13:41 jetson-nano-qspi-sd
drwxr-xr-x 2 dingo dingo 4096 Oct 4 13:41 jetson-tx2
drwxr-xr-x 2 dingo dingo 4096 Oct 4 13:41 jetson-tx2-4gb
drwxr-xr-x 2 dingo dingo 4096 Oct 4 13:41 jetson-tx2i
drwxr-xr-x 2 dingo dingo 4096 Oct 4 13:41 jetson-xavier
drwxr-xr-x 2 dingo dingo 4096 Oct 4 13:41 jetson-xavier-8gb
drwxr-xr-x 2 dingo dingo 4096 Oct 4 13:41 xavier-nx

[ichergui]

Hi @outbackdingo

You should use the flashvars from here:

$ cd ${META_TEGRA_PATH}
$ ls -la recipes-bsp/tegra-binaries/tegra-flashvars/xavier-nx 
total 12
drwxrwxr-x  2 ilieschergui ilieschergui 4096 Oct  7 11:09 .
drwxrwxr-x 10 ilieschergui ilieschergui 4096 Oct  7 11:09 ..
-rw-rw-r--  1 ilieschergui ilieschergui  843 Oct  7 11:09 flashvars
$    

This is an issue in the config file xavier-nx.inc
I send a PR to @madisongh, please take a look here: #433

[madisongh]

@outbackdingo As I mentioned over on this issue, it looks like you're basing your modified machine configuration on a TX2, rather than a Xavier NX.

From your comment above...

Next is is simply made a copy of flash_jetson-tx2-sb_custom.xml to custom-flash-layout/flash_jetson-xavier-nx-devkit-sb_custom.xml which contains the

That XML file is for Jetson-TX2. You need to start with the default flash layout file for the Xavier NX dev kit and modify that similarly to the differences between jetson-tx2-sb_custom.xml and the original flash layout file for the TX2.

cp layers/meta-testdistro/conf/machine/jetson-tx2-sb.conf layers/meta-testdistro/conf/machine/jetson-xavier-nx-devkit-sb.conf

Here is the root of the problem you're seeing with the signing requests - that config file requires jetson-tx2.conf.

None of this is as straightforward as a simple copy-paste from the existing files for the TX2, unfortunately. You need to compare the differences between the jetson-tx2-sb modifications and the base jetson-tx2 configuration and files, then apply similar (but not identical, due to differences between TX2 and Xavier) modifications from the base jetson-xavier-nx-devkit configuration and files to create a secureboot-enabled setup for that platform.

It might be easier to make the changes one step at a time, too. For instance, you could start with manual signing of the bootloaders and the normal flash layout. If you can successfully boot your secured XNX devkit with the manually signed image, then move on to the next steps - either automating the signing using digsigserver, or adding the filesystem encryption setup, then add Mender. An incremental approach will make it easier to identify issues as you go along.

[outbackdingo]

Here is the root of the problem you're seeing with the signing requests - that config file requires jetson-tx2.conf.

This is confusing in of itself as I can even find an example of jetson-tx2.conf anywhere in the layers.

And i agree maybe easier, longer learning curve, I actually think im close, once i can get past the flash layout and this configuration, so im feeling I cant be that far away from at least kernel signing, Ive already seen modules signing function, and as Im also trying to contribute working code, collaboratively instead of being a bull in a china shop, as usual i just need to know where to look, or what to look for, not necessarily have the work done for me, as we all know youve succeded greatly at a ton of heavy lifting. We all want this and my ultimate goal is to deliver, once i figure out the "missing" bits....

[ichergui]

Hi guys @nandra @outbackdingo

There was updates in repo tegra-test-distro concerning secure boot support with RootFS and other partitions (DATA, Logs and extra) encrypted for different jetson platforms (Jetson TX2 and Xavier NX).

Did you guys try it ?

Please let me know if you need help otherwise I think we can close this issue.