darp6: Failed Firmware update

[npmenard]

Yesterday I run a firmware update on my Darter Pro running Archlinux using system76-firmware-cli from the system76-firmware-daemon-git AUR package.
When scheduling the firmware update the only issue I ran into was that the script couldn't remove Boot entry 0001 (expected since it doesn't exist, my Archlinux boot on entry 0002) Rebooting and running the update worked without any issues, at the end the updater reported that the boot entry was removed and then the system rebooted.

This is when the issues started, at reboot the system booted the firmware updater again which then throw a Device Error and asked to press a key to reboot. For all subsequent reboot the BIOS would again boot the firmware updater ignoring the ESC and SPACE keys. At this point it was impossible to select another boot entry or boot from an USB key or access any BIOS menu.
I tried to unplugged battery and CMOS battery without any result, I removed one RAM module without any result finally I removed the SSD. Without the SSD the BIOS would cycle through the boot entries (first the firmware updater then Archlinux) and complain that it couldn't find the EFI executable. After cycling through the entries I was able to boot from a USB key (with PopOS) but still could not interact with the BIOS menu via ESC or SPACE key. In POPOs I used efibootmgr to remove the 1776 entry as to avoid booting into it, the program didn't report any errors but after rebooting the BIOS would still boot first into the firmware updater. I also tried to add boot entry (just for the sake of it) in this case efibootmgr reports a Could not prepare Boot variable: Input/output error and refuse to add the entry

I plugged the SSD drive to another machine to remove the firmware updater EFI executable once back in the Darter the system would boot again Archlinux (after complaining about the firmware updater missing) But after I replaced the second memory stick back into the machine, the BIOS would happily load grub the the boot process would get stuck at loading ramdisk. and Thus would not boot, the same happened with my PopOS USB key and my archlinux USB key.

I read about issue #98 and tried the darp6 USB image that was linked, however I ended up with

Finally I booted into my PopOs usb key and build the firmware myself using firmware open, I first tried to boot into the usb.img generated but it would give me a DeviceError. I was however able to flash the BIOS using the flash script (but i didn't update the EC) The BIOS was successfully updated and I was able to reboot into it (2020-09-17_f10af76), the ESC and SPACE started to work again BUT I still can't boot into anything when both memory stick are in, and I still cannot add an entry using efibootmgr. The only way to boot my Archlinux is to drop in an EFI shell and manually load grubx64.efi
EDIT: after the BIOS update all boot entries were removed

This was a long post but I wanted to give as much information as possible.

Summary:

1. Updated BIOS firmware using the cli tool under archlinux
2. After update ESC and SPACE stopped working and the system would continuously boot into the updater
3. Removing 1 RAM stick, SSD and disconnecting CMOS battery and battery allowed me to boot into a USB key, but the ESC and SPACE key still didn't work.
4. Putting the RAM stick back would not let me boot beyond the ramdisk stage of the boot process. So going forward I removed it
5. Tried solution fronm Issue on DarterPro-6 / Galaga Pro #98 that would not work either
6. Rebuild a BIOS image and updated it using the flashing tool (under a PopOS live) worked. ESC and SPACE worked again as expected, however I am still not able to manipulate boot entry using efibootmgr
7. If I put the second RAM stick back would still not let me boot beyond the ramdisk stage.
8. Right now my system works with one RAM stick out and if I manually boot to my system using an EFI shell.

Things I will try:

1. Cycle the RAM sticks and see if the boot issue happens regardless of order or position.
2. Flash the EC

[ryanolton]

Darter Pro (darp6) updated today with 2020-09-03_9c310f0 and I have the exact same issue as shown in the screenshot above.

[jackpot51]

I believe I know the issue, based on the Device Error message mentioned.

In testing this update, we flashed back and forth between firmware versions a number of times on a number of Darter Pros, with no issue. But I believe that updating without clearing NVRAM between the very first Darter Pro firmware and the latest Darter Pro firmware is not functioning correctly. Updating without clearing NVRAM between the previous version and the latest version is. So devices that have kept their NVRAM over a number of updates may be unable to write NVRAM variables.

I will provide a new usb.img that will fix this, and the firmware updater will detect this condition and clear the NVRAM if necessary.

[jackpot51]

Here are the steps to follow to clear your NVRAM, which should fix this issue:

• Download darp6-clear-nvram.zip
• Extract darp6-clear-nvram.img from darp6-clear-nvram.zip and verify with sha256sum:

71d6c7ecf6276ca0d4a3f5dae53a52cb6f0f254035a17388466aecd6515511db  darp6-clear-nvram.img

• Use etcher or popsicle to image that to a USB flash drive, preferably USB 2.0
• Remove all internal drives and all but one memory stick from the broken system
• Disconnect AC, main battery, and CMOS battery for 30 seconds, then reconnect them
• Turn on your broken system with the drive plugged in, and the AC adapter plugged in
• Press ESC at the System76 logo to get to the firmware menu. Go to the boot menu.
• Boot from the USB drive if it shows up in the firmware boot menu. If not, please take a screenshot of the firmware boot menu with the drive plugged in, and note the drive make and model.
• The previous firmware will be installed by the firmware updater and the system will prompt to shut down
• When booting up the next time, it will have a blank screen for about 30 seconds. This is normal.
• After 30 seconds, the System76 logo should appear. Press ESC, go to the boot menu and select your internal drive
• Boot your OS and check for stability
• If at any point any instructions don't work please make a comment with a screenshot

[jackpot51]

A user downloading the firmware update from this point onwards will have their NVRAM cleared, possibly fixing the issue. If the update was downloaded before this comment was made, the instructions in #127 (comment) should be used.

[ryanolton]

@jackpot51 I have followed the steps detailed above and am now at a Firmware Updater screen that reads:

App error: NotFound

Thoughts?

[jackpot51]

Could you please give me a screenshot?

[ryanolton]

[jackpot51]

Thanks, what did you use to flash the USB stick?

[ryanolton]

Popsicle/USB Flasher, I think there was an update to this app today (maybe?).

[jackpot51]

That should have been fine then. Normally, there would be some output before an error like this. Since there is none, I can guess where it crashed.

Here is one that avoids reading and setting the boot override:
darp6-clear-nvram-no-override.zip

shasum256 is as follows:

0edc4e7ae48baad6e005a197b68f852bd77e2ca05d8d5ea806e618c4fe5eb31e  darp6-clear-nvram-no-override.img

[mdeneen]

Hi! I attempted to do this, wrote the contents of the iso file to a sandisk cruiser USB memory stick using dd and followed the procedure as described.

No USB entry shows up in the boot menu regardless of the port used. I let it boot from the default device and received this:

Thinking that maybe it wiped nvram, I inserted the memory module and the nvme storage but received the same error that I started with.

Is dd okay to use? I tried balena etcher in windows 10 as well.

[ryanolton]

This version allowed the Firmware Updater to complete, however, now I'm back to my machine not progressing past the screen that prompts you to Press ESC for Boot/Options/Settings, or SPACE for Pop!_OS Recovery.

When I attempt to flash the machine again using this latest file, it tells me that * All updates applied successfully *. When I hit ESC and select my boot drive, the screen goes dark and never boots.

Do I need to repeat any steps - battery disconnect, etc. - in order to complete this process?

[leviport]

@ryanolton Does selecting your normal boot device from the boot menu also stick at the logo screen?

@mdeneen Did you use the most recent USB image that was posted here? #127 (comment)

dd should be fine, yeah