RFC: extend support to all Jetsons

[madisongh]

Note that I still consider this a work in progress, but I thought I'd get these changes up for review for some early feedback.

This patch series extends Mender support to all of the current Jetson development kits: TX1, TX2, Xavier, and Nano.

Key changes:

1. Supporting the non-U-Boot platforms (Xavier and TX2, where U-Boot is optional)
2. Supporting Nano dev kits with SPI flash+SDcard, with U-Boot environment residing in SPI flash.
3. Reworking the interface with nv_update_engine to eliminate the need for an ArtifactRebootEnter script.
4. For TX2s with U-Boot, adding a mechanism to handle the possibility that the U-Boot environment location changes with an upgrade.

I've done some testing both manually and OTA on TX1, TX2 with and without U-Boot, Nano, and Xavier. Xavier has some issues that are probably not Mender-specific - this is the first time I've actually tried using the NVIDIA bootloader update support on that platform, and it looks like something may need fixing in meta-tegra. Other platforms look OK so far.

During testing I ran into some issues that required fixes in meta-tegra, so if you play with these changes, make sure you're up-to-date with the zeus branch there.

A couple of things in particular I'd like feedback on:

• I'm not 100% sure I've got the U-boot variable migration scripts completely right, particularly for the rollback case. I'm willing to drop them altogether, since they're handling what is probably an uncommon use case.
• I know @Ecordonnier already has a pending PR for the warrior branch for Nano support, and that uses the SDcard for U-Boot environment storage... I can see why, since it was harder to coax the Mender stuff into having it reside in the SPI flash. If that goes in, I can rework these changes to also use the SDcard for the environment (or make it an option).
• I ran into some issues with the tegra-state-scripts recipe not propagating changes, particularly when I was deleting scripts. I was thinking of moving the installation and deployment of the scripts to recipes that need them, rather than having this separate recipe, which I think would improve the reliability.

[Ecordonnier]

Nice changes! I am back from holiday on the 6th and should have some time to review the branch.

Regarding the u-boot environment being in the SD-card in my PR, the reason I chose the SD-card instead of the SPI flash is that currently, in the company I am working for, only the SD card is flashed to provision a new Jetson Nano, and the SPI flash is left in its factory state. AFAIU having the environment in the SPI flash would add one more step to the provisioning process without any advantage for our use-case.

In the past, as I worked on projects having different machines/models using the same module but a different carrier-board, we solved this by using some identification resistor on the carrier-board connected to some GPIO of the module so that the software on the module could recognize the carrier-board, rather than by storing the machine ID in the SPI flash on the carrier-board. This has the advantage that it is more robust than storing the machine ID in software.
However maybe you have different use-cases in mind for which such an approach would not be feasible or less easy than storing the ID in the SPI flash.

[dwalkes]

@madisongh Thanks for sharing! This is great! I just read through, haven't tried the changes myself yet.

I'm not 100% sure I've got the U-boot variable migration scripts completely right, particularly for the rollback case. I'm willing to drop them altogether, since they're handling what is probably an uncommon use case.

Heroic efforts to handle this! I'd be less worried about them not working properly as I would be that they could somehow overwrite an area they shouldn't overwrite, although I can't think of a specific scenario where this would happen or would cause problems. If anything I'd say keep them in git and just remove from do_compile_append with a comment saying they aren't fully tested.

I know @Ecordonnier already has a pending PR for the warrior branch for Nano support, and that uses the SDcard for U-Boot environment storage... I can see why, since it was harder to coax the Mender stuff into having it reside in the SPI flash. If that goes in, I can rework these changes to also use the SDcard for the environment (or make it an option).

I don't have a strong opinion on this one, but since it's useful for @Ecordonnier to have this option it seems like it would be worth keeping it.

I ran into some issues with the tegra-state-scripts recipe not propagating changes, particularly when I was deleting scripts. I was thinking of moving the installation and deployment of the scripts to recipes that need them, rather than having this separate recipe, which I think would improve the reliability.

Fine with me.

[dwalkes]

I just read through, haven't tried the changes myself yet.

I tried today, using cboot-prebuilt configuration on TX2 and tegraflash to flash the new image.

I scp'd in the .mender file to the /tmp directory, then ran this command

mender -install /tmp/core-image-base-jetson-tx2.mender -log-file /data/mender-install.log -log-level debug

This completed successfully.
I see this output from nvbootctrl after the command is run

root@jetson-tx2:~# nvbootctrl get-current-slot
0
root@jetson-tx2:~# nvbootctrl dump-slots-info
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 14,             suffix: _a,             retry_count: 7,             boot_successful: 1
slot: 1,             priority: 15,             suffix: _b,             retry_count: 7,             boot_successful: 0
root@jetson-tx2:~#

Here's mender-install.log

I then rebooted, here's an example reboot log:
reboot-after-update.log

At this point I'm still on /dev/mmcblk0p1 for /. I was expecting to go to the APP2 partition for the rootfs in mmcblk0p30.

I also noticed I'm still on slot 0 after the reboot, was expecting to go to slot 1

root@jetson-tx2:~# nvbootctrl get-current-slot
0

and I noticed the retry counter went to 0 on slot 1

root@jetson-tx2:~# nvbootctrl dump-slots-info
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 14,             suffix: _a,             retry_count: 7,             boot_successful: 1
slot: 1,             priority: 15,             suffix: _b,             retry_count: 0,             boot_successful: 0
root@jetson-tx2:~#

I don't understand how the root filesystem is supposed to be setup in the cboot case. I see that fw_printenv and fw_setenv u-boot-fw-utils-fake files just use a /var/lib/mender/upgrade_available marker file which doesn't look like it will have anything to do with rootfs partition selection. I can see logic related to finding the other boot partition in redundant-boot-install-script but I don't see how this gets reflected in cboot or in the boot process, and I don't see anything about setting the application partition for the rootfs in the Bootloader Update section of the nvidia documentation based on nv_update_engine --install

Any pointers are appreciated. This is the first time I've tried to use cboot only as bootloader so I'm probably missing something obvious.

[madisongh]

From the reboot log, it looks like there was something wrong with the BUP payload:

[ 1338.562920] reboot: Restarting system
[0000.200] C> ERROR: Highest Layer Module = 0x40, Lowest Layer Module = 0x40,
Aux Info = 0x0, Reason = 0xd

which caused the bootloader to revert back to slot 0.... the slots info you see reflect the unsuccessful boot.

The boot-time rootfs selection for the cboot case is driven by the boot slot. cboot passes the current boot slot as a kernel command line parameter, and the init script in the tegra-minimal-initramfs initrd uses that to decide which rootfs to mount. My posted changes may be missing something there, though, since the script in meta-tegra by default expects the slot 1 rootfs partition to be labeled APP_b instead of APP2.

[Ecordonnier]

I checked the code and didn't find any issue in the logic. Also see my comment, basically since my branch and yours are not compatible I think only your branch should get merged.

Feel free to take over the README.md change of my branch as well as update the wiki entry for jetson-nano once you send a PR to the upstream repository.

Regarding the u-boot environment I still think it would be better to have it in the sd-card (one step less for provisioning, meaning reduced cost of provisioning and reduced risk of error).

[madisongh]

@dwalkes I ran a bunch of tests today after reproducing the issue you're seeing. Looks like it might be related to the issue I was seeing on my Xavier. I don't see the issue with the TX2 I have set up for secure boot, though - bootloader updates work fine there (although my setup for that machine is a bit different from the default).

Bootloader updates also work fine on both Xavier and TX2/cboot when I don't include meta-mender-core/meta-mender-tegra in my layers. And they appear to work fine when using TX2/U-Boot with the mender stuff.

I'm not sure what's going on yet. I'm going to run some comparisons of the built artifacts from working vs. non-working builds and see if that will point me in the right direction.

[dwalkes]

Thanks @madisongh I spent a bit of time on it last night mostly just understanding more about how bup creation works in tegra. I was planning to take out mender layers as a next step which it looks like you already did. I should have more time to spend on it this weekend if you haven't solved by then. Were you able to find out what Reason = 0xd means in the MB1 error log? All I could find was this post and the resolution there doesn't completely make sense to me.

[madisongh]

I tracked down the problem and will push updates to address that and the other issues reported here shortly.

[madisongh]

So "shortly" turned out to take longer than I expected, but I've pushed changes for:

• Fixing the issue with non-U-boot devices -- the problem was with a discrepancy in setting the size of the APP partition between the real image and the initrafms image (which is where BUP payloads get generated for the cboot devices). The MB1 Boot Control Table used by the first-stage bootloader includes pointers to boot partitions located in the eMMC as direct offsets from the start of the eMMC (no reading of the GPT). Since the boot partitions come after the APP partition, sizing it differently in the two contexts caused the problem.
• Removing the state scripts that did U-Boot environment variable migration, per earlier discussion.
• Adding in a setting for allowing SDcard-resident U-Boot environment on Nanos.

I also did some fault injection testing and made additional changes to the state scripts to better handle update failures and rollbacks. In particular, there are checks to make sure that we don't let the Tegra A/B boot slot get out of sync with Mender's for different failure scenarios.

I'll run some further tests this weekend, but this latest should be in reasonable shape.

[dwalkes]

Thanks @madisongh!

I can confirm this fixes issues I noticed with cboot TX2 related updates. I've also tested TX2 u-boot, Nano with and without TEGRA_MENDER_UBOOT_ENV_IN_SPIFLASH and rollback in each scenario, all worked for me. I don't have an Xavier platform to test with but based on the relatively small number of changes from TX2 I'm not expecting issues with that platform.

I started a conversation on mender hub to get some feedback on the fake fw_printenv/fw_setenv approach here.

You are of course welcome to add your name to the maintainer list if you like but I'm guessing you have enough on your plate with meta-tegra. If you did want to maintain this it might make more sense to make it a part of meta-tegra. If not I'll plan to continue to maintain as I can through meta-mender-community.

Thanks for sharing your expertise and massive improvements with these contributions, very much appreciated!

[madisongh]

I've pushed a handful a fixes for problems that cropped up in my testing this weekend. With these changes, Jetson-TX1 now updates properly, and the one system I have with a custom flash layout that hard-codes the value for ROOTFSPART_SIZE also works. Retested across all my devices this morning with these changes, all check out.

Let me know if you'd like me to rebase these changes and either re-push to this PR or open up a new one with the cleaned-up history.

[dwalkes]

Thanks @madisongh

Let me know if you'd like me to rebase these changes and either re-push to this PR or open up a new one with the cleaned-up history.

No need as far as I'm concerned. Assuming you want to keep these changes in meta-mender-community I think we need to wait for them to add a zeus branch before submitting a PR there. Depending on what happens with mendersoftware#119 I'll rebase these changes on top of warrior as needed.

[madisongh]

You are of course welcome to add your name to the maintainer list if you like but I'm guessing you have enough on your plate with meta-tegra. If you did want to maintain this it might make more sense to make it a part of meta-tegra. If not I'll plan to continue to maintain as I can through meta-mender-community.

Let's keep it separate, although I'm open to making changes in meta-tegra to simplify the integration where it makes sense to do so. As you say, I already have my hands full, so I'm fine with handing off the ongoing maintenance, but feel free to reach out if you need something.

[madisongh]

FYI, I have updates for L4T R32.3.1 on this branch. Flash layouts changed quite a bit between R32.2.x and R32.3.x, so OTA upgrades between them aren't going to be possible without a bunch of work (and even then, maybe not).

[dwalkes]

Thanks @madisongh

Flash layouts changed quite a bit between R32.2.x and R32.3.x, so OTA upgrades between them aren't going to be possible without a bunch of work (and even then, maybe not).

Would you suggest I just make the zeus branch for the meta-mender-mender community project target zeus-l4t-r32.3.1 for people who don't have specific requirements, assuming 32.3.x is most likely to be what's closest to matching flash layouts going forward?

[madisongh]

Would you suggest I just make the zeus branch for the meta-mender-mender community project target zeus-l4t-r32.3.1 for people who don't have specific requirements, assuming 32.3.x is most likely to be what's closest to matching flash layouts going forward?

Might be a good idea. Another approach which I'm going to try is to use the 32.3.1 layouts with the 32.2.x-based zeus branch, to see if that will work. But going with 32.3.1 as the base has the added advantage that bootloader update support looks better in that version.

[dwalkes]

I've opened #11 to track what I plan to merge on the zeus branch, targeting l4t r32.3.1 and reverting previous merges for warrior support before applying the changes on @madisongh zeus-l4t-r32.3.1 branch

[madisongh]

Flash layouts changed quite a bit between R32.2.x and R32.3.x, so OTA upgrades between them aren't going to be possible without a bunch of work (and even then, maybe not).

I've been able to make this work by using an R32.3.1-based flash layout XML file for my R32.2.x builds.