sdm 10.0 and earlier incorrect /boot/firmware handling

[gitbls]

A fairly serious bug has been uncovered in sdm with respect to Bookworm. Bookworm IMGs customized with 'apt upgrade' and kernel 6.1.58 gets installed, that is, were updated to 6.1.58-1+rpt2 kernel during an IMG customize, are not really running the 6.1.58 kernel.

If the system was booted and runnng when the upgrade to 6.1.58 was done, there is no problem.

IDENTFICATION

You can identify problematic systems that are booted and running by:

• Determine the latest kernel version that is installed: sudo apt search linux-image | grep installed
• Determine what kernel is running: uname -r

If the version of the kernel running does NOT match the latest installed kernel, the system needs to be repaired.

REPAIRING THE RUNNING SYSTEM

• Install sdm V10.1 or later on your host:
  curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash
• Determine which kernels are installed: sudo apt search linux-image | grep 6.1.58 | grep installed
• Update each installed kernel:
  sudo apt install --reinstall linux-image 6.1.0-rpi6-rpi-v8 linux-headers-6.1.0-rpi6-rpi-v8

Note that on 64-bit systems you should update both the -v8 and -2712 kernels.

REPAIRING SDM

• Install sdm V10.1 or later on your host:
  curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash
• Delete and recreate any affected customized IMGs, or alternatively use the above REPAIRING THE RUNNING SYSTEM while doing a --explore on the IMG with sdm 10.1

REPAIRING CUSTOMIZED IMGs

First update sdm on the host per above, then:

• sdm --explore /path/to/img.img using sdm V10.1 or later
• Determine which kernels are installed: sudo apt search linux-image | grep 6.1.58 | grep installed
• apt install --reinstall linux-image 6.1.0-rpi6-rpi-v8 linux-headers-6.1.0-rpi6-rpi-v8

WHAT WAS BROKEN

Long story short: sdm was not updated correctly for /boot/firmware in Bookworm

[rosesyrett]

I've got a bookworm image that I've customised that I've just tried to repair... At the moment I'm seeing the following outputs:

root@sdm:/# apt search linux-image | grep 6.1.58 | grep installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

root@sdm:/# apt search linux-image | grep installed              

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

linux-headers-6.1.0-rpi4-rpi-2712/stable,now 1:6.1.54-1+rpt2 arm64 [installed,automatic]
linux-headers-6.1.0-rpi4-rpi-v8/stable,now 1:6.1.54-1+rpt2 arm64 [installed,automatic]
linux-image-6.1.0-rpi4-rpi-2712/stable,now 1:6.1.54-1+rpt2 arm64 [installed,automatic]
linux-image-6.1.0-rpi4-rpi-v8/stable,now 1:6.1.54-1+rpt2 arm64 [installed,automatic]
root@sdm:/# apt install --reinstall linux-image 6.1.0-rpi6-rpi-v8 linux-headers-6.1.0-rpi6-rpi-v8
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'raspberrypi-kernel' instead of 'linux-image'
E: Unable to locate package 6.1.0-rpi6-rpi-v8
E: Couldn't find any package by glob '6.1.0-rpi6-rpi-v8'

Is there something about the instructions that I'm missing?

Thanks!

Note: I know my image needs to be repaired because:

root@sdm:/# sudo apt search linux-image | grep installed
sudo: unable to resolve host sdm: Temporary failure in name resolution

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

linux-headers-6.1.0-rpi4-rpi-2712/stable,now 1:6.1.54-1+rpt2 arm64 [installed,automatic]
linux-headers-6.1.0-rpi4-rpi-v8/stable,now 1:6.1.54-1+rpt2 arm64 [installed,automatic]
linux-image-6.1.0-rpi4-rpi-2712/stable,now 1:6.1.54-1+rpt2 arm64 [installed,automatic]
linux-image-6.1.0-rpi4-rpi-v8/stable,now 1:6.1.54-1+rpt2 arm64 [installed,automatic]
root@sdm:/# uname -r
6.2.0-37-generic

[gitbls]

This is very strange: Note, selecting 'raspberrypi-kernel' instead of 'linux-image'
I've never seen that. I'd need to see /etc/sdm/history and /etc/sdm/apt.log from the IMG to have any hope of sorting out what's going on.

I'm also quite puzzled by the 2nd part, there hasn't been any kernel with 6.2.0-37 that I'm aware of. Where did that come from?

If possible, I'd encourage you to rebuild the IMG using sdm V10.1. If that's not possible, this is going to require much more investigation and would be much easier if you could email me the above-mentioned files at the email address in the sdm README. Thx.

[Jackbennett]

This is a neat project but I had the same issue initially v10 and then from v10.1 on a fresh download, had to fix it in a pinch so
going back to the bullseye images worked.

This was being customized from a container on the RPI and mounting an SD card via USB. So if there's any scripts assuming the running linux is like the SD card that's not going to line up.

[gitbls]

This is a neat project but I had the same issue initially v10 and then from v10.1 on a fresh download, had to fix it in a pinch so going back to the bullseye images worked.

This was being customized from a container on the RPI and mounting an SD card via USB. So if there's any scripts assuming the running linux is like the SD card that's not going to line up.

Please explain what you mean by "So if there's any scripts assuming the running linux is like the SD card that's not going to line up."

Thx

[gitbls]

Inspired by the feedback here, I took yet another look into this issue. It appears that for some reason, when doing a customize, the linux-image and linux-headers packages are not updated UNLESS you add --apt-dist-upgrade to your --customize command line.

Without --apt-dist-upgrade the system boots using the original bookworm kernel, 6.1.54, and that can be updated using standard update/upgrade once the system is up and running.

I didn't notice this before because my sdm customization script always uses --apt-dist-upgrade. My recommendation is that you do so as well.

[gitbls]

@rosesyrett Can you please test the following on that IMG with V10.1?

• sdm --explore into the IMG
• apt update
• apt dist-upgrade
• `apt search linux-image | grep 6.1.58 | grep installed

The apt search will include the most recent updates, so you should see the newly-installed packages in the output

[Jackbennett]

here, I took yet another look into this issue. It appears that for some reason, when doing a customize, the linux-image and linux-headers packages are not updated UNLES

Thanks for the reply, please disregard because I don't think it adds anything. but from my console uname -r is a talos container kernel not physical system. That may not matter anyway.

[gitbls]

here, I took yet another look into this issue. It appears that for some reason, when doing a customize, the linux-image and linux-headers packages are not updated UNLES

Thanks for the reply, please disregard because I don't think it adds anything. but from my console uname -r is a talos container kernel not physical system. That may not matter anyway.

I might be misunderstanding what you said, but in the sdm container -uname shows the host system, not the OS in the container that we're customizing. I could tell by looking at the files in /boot and /boot/firmware.

[gitbls]

Closing this issue as it's ancient history now