Mender delta update issues with demo distro

[vinotion]

Hi there!

Not sure if this is the right place, but I'm asking here anyway, as perhaps someone here can help me track down the issue.

We're currently trying to integrate Mender delta updates in our Yocto distro (which was modeled on tegra-demo-distro). However, we'er running into problems.

I have an issue thread running on Mender Hub: https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/9

We started out finding this issue on our custom distro, but it turns out a vanilla build of tegra-demo-distro / demo-image-base has the same problem. We followed this documentation to integrate Mender delta updates into the Yocto build.

Does this ring a bell for anyone? Any help is appreciated, thanks very much in advance! Also, if there's a better place to discuss this, let me know.

[madisongh]

Which branch are you based off of? The original Mender integration wasn't compatible with delta updates, as the rootfs would get modified during an upgrade. Some changes went in to address that, so recent dunfell and later branches should be compatible.

[vinotion]

OK, thanks.

I'm rebuilding demo-image-base now with the latest changes. I'll check back in when this is tested.

[dwalkes]

I think this was the last round of delta update fixes which were upstreamed, FYI mendersoftware/meta-mender-community#221

[vinotion]

Unfortunately I'm still having the same issues. The Git branch for dunfell is at 9b9df014a0d2b58b4e5e83a877be8850dc30a5d5.

What I tested is:

1. Take the vanilla tegra-demo-distro,
2. Add Mender delta update support (as per this documentation),
3. Build demo-image-base for distro tegrademo-mender for machine jetson-tx2-devkit,
4. Name this version 'A'; flash it onto a TX2 development board, copy the full rootfs .mender file to A.mender,
5. Install and commit A.mender to both partitions such that the right provides values are available,
6. Make a slight modification to tegra-demo-distro (I added a line of comment to layers/meta-tegrademo/recipes-core/base-files/base-files/tegrademo-mender/fstab),
7. Rebake demo-image-base,
8. Copy the full rootfs .mender file to B.mender,
9. Generate a delta artifact using mender-binary-delta-generator -o A2B.mender A.mender B.mender,
10. Try to install the delta artifact on the device: mender install A2B.mender.

This last step fails:

root@jetson-tx2-devkit:~# mender install A2B.mender 
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p33 
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 14524928...
INFO[0000] No public key was provided for authenticating the artifact 
.....................................................          -  86 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect 
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent 
INFO[0000] Update module output: Failed to apply the delta, err: 1 
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: demo-image-base-jetson-tx2-devkit.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: broken pipe 
ERRO[0000] Update module terminated abnormally: exit status 1: Payload: can not install Payload: demo-image-base-jetson-tx2-devkit.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: broken pipe

I'm not sure where to look next really.

[vinotion]

I think this was the last round of delta update fixes which were upstreamed, FYI mendersoftware/meta-mender-community#221

Hi Dan!

I missed your reply by minutes. I checked, and the meta-mender-community layer is at 87b5c03c8e1c0c2cb27ecfe35c9665bbec2f55e6, which has this fix (according to the Git log).

[dwalkes]

I'm not sure where to look next really

I see your thread at https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/9 - that was going to be one of my suggestions.

I've never put the time in to try to get delta updates working, but I know we've had contributions from at elast one user in the past who has succesfully used it. You might try to match whatever they were likely using based on the timeframes, or perhaps they will chime in here.

[vinotion]

Hi all,

I've progressed sorting out this issue on the Mender Hub. The main cause seems to be the fact that the checksum of the read-only root filesystem is different from what Mender expects. We already tried disabling metadata_csum on the root filesystem creation, but this did not help.

But what's more: there is even a difference between the two root partitions right after a fresh flash (see this post). Does anyone have a clue why this may be the case?

Again, thanks in advance!

[dwalkes]

Wow sorry this is turning into such a project. I would never have guessed this would have been such a project given it's been reported as working with a few minor changes in the past.
As I recall, the issues we had in the past by this were all related to work done in the commit script. Adding the read only to the mount should have prevented this from being a possibilty I would assume.
I think what I'd try next is writing a script which dumps out the md5 of every file in the filesystem and compares to the filesystem dumped out of the artifact. Or maybe just start by the list of files and looking for differences in files between artifact and rootfs.

[vinotion]

Thanks for the suggestions, very much appreciated! We will definitely look into this.

Do you (or anyone else) have a first idea as to why after a fresh install the two root filesystems have a different hash sum values? I would think a clean flash of a device would result in two bit-level identical read-only root filesystems. It's probably also the cause of why the whole delta updates integration is failing in the first place, but this comment by Kristian made me unsure about how the initial post-flash state is set up.

[vinotion]

Right. A first result of the per-file SHA256 diff shows different contents of /etc/machine-id (part of systemd). This might be the root cause.

There are a lot of layers/recipes that do something with this file (including Mender-specific ones). Looking into it now.

[vinotion]

We've managed to fix the issue.

As I already alluded to in my previous message, the difference was in /etc/machine-id. It turns out the kernel didn't mount / as read-only by default, so we had to add that. This will make sure the machine ID writer will not change the root filesystem.

However, that's not the whole story. There's more to /etc/machine-id than meets the eye. Especially when it is used in conjunction with read-only file systems and overlay mounts for /etc. The order in which things happen is very important, see this PR for more info. The short summary: it's kinda broken in systemd but not going to be fixed.

Thanks everyone for helping out! 😃

We can post a PR for adding the ro kernel argument for tegra-demo-distro, is that desired?

[madisongh]

We can post a PR for adding the ro kernel argument for tegra-demo-distro, is that desired?

At least open an issue for this. I think it's already there for the u-boot platforms, so we'll just need to figure out the right way to put it in place for cboot platforms when building an image with read-only-rootfs in IMAGE_FEATURES.

Thanks! Glad you were able to track this down.

[dwalkes]

Great work tracking this down @vinotion !
I remember this has come up before - see the discussion at OE4T/meta-mender-community#8 (comment) if you hadn't seen that one already. I think OE4T/meta-tegra#527 is related too.

[vinotion]

A bit late, but here we go: #214