Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.32 BIOS update for t14s possibly causes the laptop to be bricked #129

Closed
espidev opened this issue Jun 7, 2021 · 29 comments
Closed

1.32 BIOS update for t14s possibly causes the laptop to be bricked #129

espidev opened this issue Jun 7, 2021 · 29 comments
Assignees
Labels
bug Something isn't working

Comments

@espidev
Copy link

espidev commented Jun 7, 2021

I updated my laptop yesterday (on Arch, through Discover's fwupd backend) and after a reboot it seems to be bricked.

On the first reboot, it showed a black screen for a long time, and then showed some slightly stretched words in white text on black (that I unfortunately can't remember). I do remember that it did not say anything along the lines of doing background upgrades, and it just froze like that for about half an hour. I then decided to force reboot, and now the screen is just always black, with the power button glowing on.

I've tried to reset power through the reset pin hole, and also disconnecting the CMOS battery, but nothing seems to have helped.

It's possible I may have done something wrong? The update has been out for 2 weeks and I haven't see any other complaints...

Description I remember seeing in Discover:

https://download.lenovo.com/pccbbs/mobiles/r1cuj63w.txt

@hughsie hughsie added the bug Something isn't working label Jun 7, 2021
@hughsie
Copy link
Member

hughsie commented Jun 7, 2021

Can you try disconnecting all the batteries (there might be multiple internal/external ones) and waiting 30 minutes, then reconnecting the batteries? @mrhpearson might have some more ideas. So sorry to hear this, I'm really disappointed this happened.

@espidev
Copy link
Author

espidev commented Jun 7, 2021

Sure, I'll give it a try, thanks.

I disconnected the CMOS and the main internal battery (I don't see any others)

@espidev
Copy link
Author

espidev commented Jun 7, 2021

Unfortunately nothing seems to have changed, both having the battery disconnected (and on AC power) and having it reconnected.

@mrhpearson
Copy link
Collaborator

Really sorry to hear that - and like Richard very disappointed too.

Richard - there seem to be a lot of issues around this update for the AMD platforms. I'm seeing failed updates (albeit not bricks) raised on the forum. Should we pull this to prevent users hitting the issue while I work with the firmware team to understand what's going on?

Note - I tested the update myself on the T14 AMD with no issues - but there's obviously something else going wrong.

@espidev - please contact Lenovo support about your system in the meantime.

@espidev
Copy link
Author

espidev commented Jun 7, 2021

@espidev - please contact Lenovo support about your system in the meantime.

I will, thanks!

@fkru
Copy link

fkru commented Jun 7, 2021

Really sorry to hear that - and like Richard very disappointed too.

Richard - there seem to be a lot of issues around this update for the AMD platforms. I'm seeing failed updates (albeit not bricks) raised on the forum. Should we pull this to prevent users hitting the issue while I work with the firmware team to understand what's going on?

Note - I tested the update myself on the T14 AMD with no issues - but there's obviously something else going wrong.

Do you have secure boot disabled? Thx.

@The-Compiler
Copy link

FWIW I've had a similar experience on my T14 AMD (see #74 (comment)), but thankfully without the "stretched text" and/or bricking part. 😅

I contacted Lenovo Premier support when the hang happened since I didn't want to interrupt a potentially still running upgrade, and they recommended to restart it in the following way:

  • Detach any power sources
  • Press the reset pinhole 4 times
  • Press the reset pinhole again and hold for 30s
  • Try turning the laptop on via the power button

Your situation might be entirely different and I'm sure Lenovo support can give you a better answer for your case, but perhaps something worth trying anyways?

@espidev
Copy link
Author

espidev commented Jun 7, 2021

Do you have secure boot disabled? Thx.

Not sure if you were replying to me, but I do have it disabled.

Your situation might be entirely different and I'm sure Lenovo support can give you a better answer for your case, but perhaps something worth trying anyways?

Thanks! Doesn't seem to work for me though :c

I've filed a support ticket with Lenovo already, I guess I'll have to see if the depot can get it fixed

@fkru
Copy link

fkru commented Jun 7, 2021

Do you have secure boot disabled? Thx.
Not sure if you were replying to me, but I do have it disabled.

@espidev: My question was meant for @mrhpearson :-)

@mgelde
Copy link

mgelde commented Jun 9, 2021

I'm having issues with this update, too. My P14s Gen 1 (AMD) is not bricked, but it's not updating either. In may case I saw:

  1. some rebooting (not sure, but I think it was twice)
  2. black screen, followed after some secs by the word 'Reboot' or 'restart' (not quite sure which)
  3. then blackness again with no end in sight. i power cycled eventually (with a very bad feeling in my gut) and crossed my fingers...
    Never been so happy to see the grub boot menu.

But I don't dare to try again after reading this here.

IMO it would be super helpful if:

  1. The hint before applying the update was clearer. It says (SIC) "In case user update the UEFI BIOS v1.32 (R1bET63W) , need to load default configuration in BIOS setup.(for enable DMAr function support)."
    No idea what DMAr is (or if that's another typo) but I don't find anything meaningful. I cannot reset to factory defaults because my system won't boot (secure boot) and I can't apply the update w/o running my OS. Resetting during the update just messed up my EFI vars and I had to rest them from a live environment. So this could definitely be clearer. Just enabling Update support from user software (or whatever the "Allow Windows to update UEFI" menu entry is called) led to the behavior described above.
  2. In the same vein: Please outline what the update should look like. For instance: "You will see a black screen with the word Reboot. Then please reboot" (or possibly "WHATEVER YOU DO, DON'T REBOOT").
    I'd definitely feel better if I had some knowledge about whether what I was seeing and doing is expected or not.

(Take what you think is helpful and ignore the rest.)

@mgelde
Copy link

mgelde commented Jun 11, 2021

FWIW: I installed the update using a bootable stick (geteltorito workflow) and that worked with secure boot disabled (so at least for p14s it's not an incompatibility issue per se). The description inside the official image is somewhat more comprehensive (it also contains the line i quoted in my previous post; didn't realize it was from there) and explains that before resetting to defaults, the "Enable OS specific defaults" option must be enabled. I am not sure that I had that box checked before trying the fwupd update. So that might be the difference between people who successfully update and those who don't.

Also: The screen stayed dark for quite some time for me as well; so long, that I got worried. but then the fans spun up and some sort of status screen with progress info appeared.

The OP reported stretched words and waiting for ~30 min. I neither saw stretched text, not did it take 30 minutes before that progress screen appeared. more like two or three. whatever went wrong there, from what I saw during the update and from what the OP wrote, the problem likely occurred long before the forced reboot.

@fkru
Copy link

fkru commented Jun 11, 2021

FWIW: I installed the update using a bootable stick (geteltorito workflow) and that worked with secure boot disabled (so at least for p14s it's not an incompatibility issue per se). The description inside the official image is somewhat more comprehensive (it also contains the line i quoted in my previous post; didn't realize it was from there) and explains that before resetting to defaults, the "Enable OS specific defaults" option must be enabled. I am not sure that I had that box checked before trying the fwupd update. So that might be the difference between people who successfully update and those who don't.

This is interesting news, thank you, which is not obvious from the notes provided by Lenovo via LVFS. Incidentally, after 10 years with DELL, where I never had any problems with firmware updates on Linux machines, I am really disappointed with the crap that Lenovo's firmware team provides. Given the issues and complaints, e.g., discussed at https://forums.lenovo.com/t5/Other-Linux-Discussions/bd-p/Special_Interest_Linux, it seems that they are completely overwhelmed.

@michaellass
Copy link

michaellass commented Jun 13, 2021

I wish I had read this report before attempting the update. In the end I think I got lucky but I'm not entirely sure. Here's how the update worked out for me on my T14s AMD (secure boot and OS defaults disabled in BIOS):

  1. I tried the update via fwupdmgr. Got a black screen that was not going away. Finally pressed the power button and turned the machine on again. It seemed to work fine but was still on 1.29. This behavior is also described in 1.32 BIOS update for T14s not successfull, falls back to 1.29  #131.
  2. Loaded factory defaults in BIOS (which removed Linux from the boot manager) and re-attempted the BIOS update, but this time from Windows. Again I got the black screen for a while but then the update continued. However, the update got stuck at "BIOS Self Healing backup progressing ... 11%".
  3. Since it would not continue, I pressed Ctrl+Alt+Del. After a couple of seconds, the machine rebooted and on next boot continued the BIOS backup. This time it got up to 19%. Rebooted again, went up to 36% and so on. After a couple more reboots the process actually finished.
  4. Now the machine would not boot anymore. It got stuck at the Lenovo screen. Trying to enter the BIOS settings, it just showed "Entering BIOS Setup" but nothing happened.
  5. Turned the machine off and on again. Now I could enter the BIOS settings again and it showed version 1.32. I loaded factory defaults again and now also Windows would boot again.
  6. Windows behaved weirdly. The device manager showed non-functional USB devices because the hub could not be reset or something like that. Also the fan of the machine seemed to behave differently (slowly ramping up on every boot).
  7. Turned off the machine, pulled the power and pressed the reset button on the bottom of the device for 20 seconds.

Finally, the machine seems to work normal again. Windows has no non-functional devices anymore. With a Linux system on a USB stick I managed to restore the UEFI boot entry and Linux now runs again as well.

All in all a quite frightening experience. I am also not entirely sure in what state the machine now is. Are all components properly updated? Did the BIOS backup actually succeed after dozens of attempts or will the next broken BIOS finally brick the device?

I highly recommend pulling this update until those issues are resolved.

@jashank
Copy link

jashank commented Jun 14, 2021

Oh boy, I just got bit by this on a T14s G1 AMD (MT 20UH). Updated from 0.1.25 to 0.1.32 yesterday --- unusually, that update was entirely uneventful (modulo some stuffing around to get it to start --- fwupdtool helpfully claims EFULL on a path on the EFISP, if, say, efivarfs was full because pstore had filled it).

All seemed okay until a suspend/resume cycle today caused my display panel to cycle through its self-test (red, green, blue, black, grey, white, 5x5 black/white checkerboard) --- the rest of the system was still running happily --- and this self-test didn't go away across suspend/resume, a hibernate/resume cycle, a full power cycle ... continued while the system was powered off (albeit not backlit) ... but went away while reseating the LCD connector until the system was powered on again.

Very terrifying, especially five minutes before a meeting.

@The-Compiler's reset-five-times trick seems to have brought my laptop back to life.

I'm firmly in favour of yanking the update. I'll be annoying my Lenovo support rep later today too.

(Update: I've spoken to a Lenovo support rep who claims that, because I have recovered from the issue and have already run diagnostics that don't show any issues, and because they haven't ever heard of any issues with this BIOS update, there's no issue and the BIOS update is perfectly fine.)

@mgelde
Copy link

mgelde commented Jun 14, 2021

(Update: I've spoken to a Lenovo support rep who claims that, because I have recovered from the issue and have already run diagnostics that don't show any issues, and because they haven't ever heard of any issues with this BIOS update, there's no issue and the BIOS update is perfectly fine.)

You and some of the other posters with bricked or otherwise problematic updates could open a thread describing your symptoms here: https://forums.lenovo.com/t5/Other-Linux-Discussions/bd-p/Special_Interest_Linux

I have seen Lenovo reps respond to problems voiced on that forum. This forum is also what lead me to the update in the first place (I was experiencing various problems with the machine that this update fixes). In any case, what you folks describe seems to be less of a fwupd problem and rather a Lenovo problem. There seems to be the additional problem that the fwupd process fails for some people (myself included), while working for others.

@fkru
Copy link

fkru commented Jun 14, 2021

(Update: I've spoken to a Lenovo support rep who claims that, because I have recovered from the issue and have already run diagnostics that don't show any issues, and because they haven't ever heard of any issues with this BIOS update, there's no issue and the BIOS update is perfectly fine.)

You and some of the other posters with bricked or otherwise problematic updates could open a thread describing your symptoms here: https://forums.lenovo.com/t5/Other-Linux-Discussions/bd-p/Special_Interest_Linux

Thanks. Mark RH Pearson is Lenovo's lead technical engineer for the linux PC team (https://forums.lenovo.com/user/viewprofilepage/user-id/1942528). According to his today's comment on bricked systems due to the update to FW 1.32 at https://forums.lenovo.com/t5/Other-Linux-Discussions/T14-AMD-battery-drain-in-standby-Linux/m-p/5037674?page=37, he states that "I have had a few other reports of the same and we're looking at those with urgency but so far we've been unable to reproduce." Maybe those of you who are hit by this issue should leave a comment there.

@mrhpearson
Copy link
Collaborator

Thanks for the note above @fkru. As a note, I'm personally happy to keep the firmware update issues here - the Lenovo forum tends to go into power management, camera, tsc, audio issues and can take a bit of wading through. Here I get to focus on just the one issue which helps.

I'm trying to go through fwupd and Lenovo forums and make sense of the reports. Having secure boot enabled seems to be a key factor. I've seen lots of 'black screen need pin hole hard resets to recover ' cases when secure boot is enabled - do we have any of those with secure boot disabled?
I think there have been some 'just doesn't update' with secure boot disabled, which still sucks but isn't anywhere near as scary.

I'm not sure for the two bricked systems I'm aware of if SB was on or not (maybe it's three systems - but it does get confusing with all the cross-posting)

This issue has been escalated to the firmware team, with a summary of all the cases above and those from ticket #79 - We're very aware of the issue. As noted in my above forum post though - we've not been able reproduce it ourselves. I'm going to give it another go shortly...and slightly nervously.

Mark

@fkru
Copy link

fkru commented Jun 14, 2021

@mrhpearson: Thanks, Mark, for the speedy reply and the clarification. I appreciate it.

@michaellass
Copy link

michaellass commented Jun 14, 2021

@mrhpearson:

I'm trying to go through fwupd and Lenovo forums and make sense of the reports. Having secure boot enabled seems to be a key factor. I've seen lots of 'black screen need pin hole hard resets to recover ' cases when secure boot is enabled - do we have any of those with secure boot disabled?

In my case secure boot was disabled as far as I can remember. However, issued from Linux via fwupd I just got the "black screen, no update" issue. The hickups started when trying to update from Windows as described in #129 (comment)

Adding an afterthought: It could of course be that the first attempt via fwupd already brought the machine into some weird state, causing the subsequent attempt from Windows to make things even worse.

@laur89
Copy link

laur89 commented Jun 20, 2021

Failed sys update from 1.29 to 1.32 for P14s AMD. On reboot it showed the firmware update screen, after which it restarted and the "BIOS self healing backup progressing" screen was shown. Everything appeared to recover nicely with no intervention by me. Haven't tried subsequent reboots though, as some user reports have noted this might fail. Suspends have worked without issues.

Post-reboot report as follows:

$ fwupdmgr get-devices
20Y1CTO1WW
│
/-/
├─System Firmware:
│ │   Device ID:          3f10cbb75c39627fc0d14eddd274374207a5c753
│ │   Current version:    0.1.32
│ │   Vendor:             LENOVO (DMI:LENOVO)
│ │   GUIDs:              01453b71-da0c-4832-9f4f-e378245339c7
│ │                       230c8b18-8d9b-53ec-838b-6cfc0383493a
│ │                       38ad8832-d322-5d73-aff7-77b8ae0f1a6f
│ │   Device Flags:       • Internal device
│ │                       • Updatable
│ │                       • System requires external power source
│ │                       • Supported on remote server
│ │                       • Needs a reboot after installation
│ │                       • Cryptographic hash verification is available
│ │                       • Device is usable for the duration of the update
/-/
Devices that were not updated correctly:

 • System Firmware (0.1.29 → 0.1.32)

Target:                  https://fwupd.org/lvfs/firmware/report
Payload:                 {
                           "ReportVersion" : 2,
                           "MachineId" : "9e479629875723cc5c9d61f756b28de6dab429773a705ad1d463827ea39b3ad1",
                           "Metadata" : {
                             "DistroId" : "debian",
                             "DistroVersion" : "11"
                           },
                           "Reports" : [
                             {
                               "Checksum" : "115e76aad628d4c30944c60f6c42fa367d33f651",
                               "ChecksumDevice" : [
                                 "8eb768c259edd4318ebd60c6d09a5cd09bcde4fc"
                               ],
                               "UpdateState" : 3,
                               "UpdateError" : "failed to run update on reboot",
                               "Guid" : [
                                 "01453b71-da0c-4832-9f4f-e378245339c7"
                               ],
                               "Plugin" : "uefi_capsule",
                               "VersionOld" : "0.1.29",
                               "VersionNew" : "0.1.32",
                               "Flags" : 574619947,
                               "Created" : 1623582324,
                               "Modified" : 1623582446,
                               "Metadata" : {
                                 "RuntimeVersion(org.freedesktop.gusb)" : "0.3.5",
                                 "HostFamily" : "ThinkPad P14s Gen 1",
                                 "BootMgrDesc" : "legacy",
                                 "CompileVersion(com.redhat.fwupdate)" : "12",
                                 "CompileVersion(org.freedesktop.fwupd)" : "1.5.7",
                                 "SecureBoot" : "Disabled",
                                 "LinuxLockdown" : "none",
                                 "EspPath" : "/boot/efi",
                                 "KernelVersion" : "5.10.0-7-amd64",
                                 "UEFIUXCapsule" : "Enabled",
                                 "LastAttemptStatus" : "0x0",
                                 "EfivarNvramUsed" : "31390",
                                 "RuntimeVersion(org.kernel)" : "5.10.0-7-amd64",
                                 "MissingCapsuleHeader" : "False",
                                 "RuntimeVersion(com.dell.libsmbios)" : "2.4",
                                 "CompileVersion(org.freedesktop.gusb)" : "0.3.5",
                                 "DistroId" : "debian",
                                 "CpuArchitecture" : "x86_64",
                                 "BootTime" : "1623261476",
                                 "HostVendor" : "LENOVO",
                                 "LastAttemptVersion" : "0x0",
                                 "RuntimeVersion(org.freedesktop.appstream-glib)" : "0.7.14",
                                 "KernelCmdline" : "vsyscall=emulate",
                                 "HostProduct" : "20Y1CTO1WW",
                                 "RuntimeVersion(com.redhat.fwupdate)" : "12",
                                 "TpmFamily" : "2.0",
                                 "HostSku" : "LENOVO_MT_20Y1_BU_Think_FM_ThinkPad P14s Gen 1",
                                 "RuntimeVersion(org.freedesktop.fwupd)" : "1.5.7",
                                 "DistroVersion" : "11"
                               }
                             }
                           ]
                         }

After uploading the report it stated

Update failure is a known issue, visit this URL for more information: https://github.com/fwupd/fwupd/wiki/LVFS-Triaged-Issue:-Failed-to-run-update-on-reboot

a) Is this the case?

b) Also note System Firmware is reported being on version 1.32 - does this mean update actually succeeded?

@mrhpearson
Copy link
Collaborator

As an update the issue has apparently been fixed and you shouldn't see the same in the 1.34 update that has been released. It worked in my testing.

Recommend disabling secure boot during updates until the secure boot issue that was introduced in shim is resolved

Any objections if I close the ticket?

@fkru
Copy link

fkru commented Jul 29, 2021

As an update the issue has apparently been fixed and you shouldn't see the same in the 1.34 update that has been released. It worked in my testing.

Recommend disabling secure boot during updates until the secure boot issue that was introduced in shim is resolved

Regarding the shim issue, is there any solution in sight? Thx.

@mrhpearson
Copy link
Collaborator

Fix for shim has been delivered: rhboot/shim@87fbf42

How it gets rolled out in your distro of choice may vary. If it's useful the Ubuntu notes are here: https://bugs.launchpad.net/ubuntu/+source/shim/+bug/1929471
I suspect Fedora and Arch have it already but haven't confirmed.

Mark

@fkru
Copy link

fkru commented Jul 31, 2021

Fix for shim has been delivered: rhboot/shim@87fbf42

I am puzzled, this fix is part of shim 15.3 released on 23 March. I am with shim 15.4 and firmware update via lvfs stopped working some time ago when secure boot is enabled. So, the culprit is something else.

@michaellass
Copy link

Fix for shim has been delivered: rhboot/shim@87fbf42

I am puzzled, this fix is part of shim 15.3 released on 23 March. I am with shim 15.4 and firmware update via lvfs stopped working some time ago when secure boot is enabled. So, the culprit is something else.

As far as I can see, that commit was never merged and the corresponding pull request was closed. The patch that was actually merged seems to be this one: rhboot/shim@4d64389

That patch is pretty recent and not included in 15.3 or 15.4. A good place to track the current shim issues seems to be rhboot/shim-review#165.

It also looks to me like that particular shim issue is not related to secure boot, so that would explain why the update also fails for people with secure boot disabled. Maybe it all just depends on what shim version people have installed.

@LukeIT11
Copy link

Hi Guys,

I had this same issue with T14. I had ugraded the memory and done updates etc and noticed the screen would go black and only power light would stay on. I took out the installed memory and the unit powered on fine and BIOS update continued and was able to successfully install. I then put the memory back in and the unit would again not boot black screen with power light.

@hughsie
Copy link
Member

hughsie commented Jan 25, 2023

I had ugraded the memory

So the T14 worked with the added memory and the old BIOS, but doesn't work with the added memory and the new BIOS?

@mrhpearson
Copy link
Collaborator

For the memory issue - I think that's likely unrelated to this original ticket.

I have a ticket open with the FW team about an issue with the T14 G3 with user installed memory after BIOS version 1.05 (for my reference - LO-2111). We've been unable to reproduce in-house - it seems to be very memory independent. I've been working with a customer and the FW team testing trial BIOS images to try and figure out what change caused the problem. It's quite a slow task as it relies on time from both the customer to test and the FW team to generate trial BIOS images - we're narrowing down on the likely culprit but not there yet.

If you think that's the issue you're seeing and you're willing to help debug let me know and we can communicate off thread.

@lwbt
Copy link

lwbt commented Jun 1, 2024

I took out the installed memory and the unit powered on

Thanks, I was wondering if the device would boot without memory. It took some time for me to remember that some of the RAM was soldered directly to the mainboard.

So the T14 worked with the added memory and the old BIOS, but doesn't work with the added memory and the new BIOS?

I can only answer for my P14s Gen 1. I took out the after market memory (Teamgroup, 32GB if it matters). Let the procedure finish, go to re BIOS setup screen and configure my desired state again. Shut down, reboot and not its working fine again.

TPM-based FDE is broken and I need to install one of the OSes, but I was prepared for such cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

No branches or pull requests