Surface Go hangs on boot with 5.8.14-surface kernel.

[pmsjt]

Still boots fine when I select 5.8.12-surface.

[pmsjt]

Same exact kernel (5.8.14-surface) boots fine on my Book2. Of these 2 devices, this issue only seems to affect the Go1.

[kitakar5525]

There is a report that SGO2 won't boot after applying commit gregkh/linux@953fc77 (added on upstream v5.8.13) (linux-surface/linux-surface#91 (comment)). So, the commit seems to be really causing issue on the SGO series...

I saw the journal log in the comment, but I couldn't find what is wrong.

You can get a kernel log of previous boot by journalctl --dmesg --output=short-monotonic --no-hostname --boot=-1. Can you also attach the output here?

1. boot with 5.8.14-surface
2. restart and boot with bootable kernel (e.g., 5.8.12-surface)
3. get the kernel log of previous boot

[pmsjt]

journalctl --dmesg --output=short-monotonic --no-hostname --boot=-1

Thanks @kitakar5525

I am attaching a few files. The TXT file is the output of the command you asked for (log from previous boot session). The two images help tell the story.

The machine boots for a bit and then freezes at the point captured by the first photo (usbhid). It lingers there for a long time. At least 5 minutes. At that point some timeout expires and boot continues to progress a bit more till it hits the point captured by the second picture (gtk-common-themes). I left the machine in this state for a very long time and it does not progress any longer. This seems to be the final resting place.

last_boot_dmesg.txt

[pmsjt]

Another datapoint: If I add the "nomodeset" kernel option, the boot process still does that long pause after enumerating the USB devices, but after it resumes from that point, it continues to boot and eventually completed.

So, "nomodeset" doesn't make all the symptoms go away, but it does have a positive effect.

[kbingham]

[    2.799587] kernel: usbhid: USB HID core driver
[  122.286040] kernel: EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)

Hard to be sure here, 120 seconds (a 2 minute timeout?) from the USB HID driver to the point it was able to mount the filesystem...

Can't easily remove USBHID, and we can't remove the filesystems so not easy to bisect those... but I think this needs more information.

To really identify this issue, we should bisect the kernels between 5.8.12 and 5.8.14 - that would identify/confirm the exact commit causing the issue....

@pmsjt Is that something you're able to do? It requires building the kernels manually, and using 'git bisect' to decide which kernel commit to build - test, and iterate until the bisection states explicitly which commit causes the fault.

Or if there really is some confidence that this commit is causing the problems, try reverting it (git revert 21653a4) to see if the issue is resolved with it gone.

[kitakar5525]

In addition to what kbingham mentioned: It looks like the Ubuntu config sets usbhid driver as module. So, something might be revealed by not loading the module. To do so, add module_blacklist=usbhid to your bootloader.

[B-Woody]

I have Microsoft Surface Go 2 ( with Intel Pentium Gold CPU ) that's running Fedora 32. It worked flawlessly until the update to kernel to 5.8.13-200 was installed. When I try to boot kernel 5.8.13-300 the system freezes very early in the boot process with the last line of output being displayed:

clocksource: Switched to clocksource tsc

I checked on the Surface Linux subreddit and another user with the Surface Go 2 confirmed that they had the same issue on kernel 5.8.13 but in Arch Linux. This would lead me to believe that it's a kernel issue and not Fedora specific.

I saw this commit on the kernel 5.8.13 changelog that has something to do with clocksource drivers and I think it could be related ( but it might be a coincidence )

https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.8.13

commit 510c51ff61e3313426f4103dc12766156335a8db
Author: Tony Lindgren <tony@atomide.com>
Date:   Mon Aug 17 12:24:28 2020 +0300

    clocksource/drivers/timer-ti-dm: Do reset before enable
    
    commit 164805157f3c6834670afbaff563353c773131f1 upstream.

I'm willing to test any potential fixes on my hardware if that helps.

Kernel 5.8.12 continues to work fine if I manually select it from GRUB so I'm doing that in the meantime.

[kbingham]

It's best not to speculate based on what the last message is. By the fact that, that message has printed, that one has been successful. Oftentimes, the message you need to debug is in the driver that's hung.

I'd still maintain we need someone to do a bisect on this issue and determine the exact commit causing the regression, and report that upstream to the linux-stable maintainers.

If someone can do that but needs support doing that, please ask.

[B-Woody]

Or if there really is some confidence that this commit is causing the problems, try reverting it (git revert 21653a4) to see if the issue is resolved with it gone.

If I get some time on the weekend I may be able to use my Surface Go 2 to at least give this a try and/or check if the issue occurs on a 5.9 kernel

[tml89]

Got the same issue with sgo 1 as b-woody.

[pmsjt]

In addition to what kbingham mentioned: It looks like the Ubuntu config sets usbhid driver as module. So, something might be revealed by not loading the module. To do so, add module_blacklist=usbhid to your bootloader.

That doesn't seem to solve either problem. Maybe USBHID is just a read herring and the real problem is affecting what gets printed after and not before.

There seem to have been quite a few developments since I last checked the thread. At the moment, which is the commit with the most suspicion (and the best candidate for a git revert + rebuild) ?

Thanks.

[kitakar5525]

In addition to what kbingham mentioned: It looks like the Ubuntu config sets usbhid driver as module. So, something might be revealed by not loading the module. To do so, add module_blacklist=usbhid to your bootloader.

That doesn't seem to solve either problem. Maybe USBHID is just a read herring and the real problem is affecting what gets printed after and not before.

I see...

There seem to have been quite a few developments since I last checked the thread. At the moment, which is the commit with the most suspicion (and the best candidate for a git revert + rebuild) ?

The first candidate to revert I think is i2c: core: Call i2c_acpi_install_space_handler() before i2c_acpi_register_devices()

• for v5.9-rc6 and later, 21653a4
• for v5.8.13 and later, 953fc77

But if I assume that this commit is the cause, I don't have any idea why this commit causes an issue on the SGO series. If reverting this helped, I'd suggest reporting this to the Bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=1842039) referenced by the commit.

The next candidate to revert B-Woody mentioned is clocksource/drivers/timer-ti-dm: Do reset before enable

• for v5.9-rc7 and later, 1648051
• for v5.8.13 and later, 510c51f

Ideally, as kbingham mentioned, doing git bisect is the best idea though.
(from #63 (comment))

To really identify this issue, we should bisect the kernels between 5.8.12 and 5.8.14 - that would identify/confirm the exact commit causing the issue....

[jwrdegoede]

Author of the 'i2c: core: Call i2c_acpi_install_space_handler() before i2c_acpi_register_devices()' commit here, ugh I certainly don't hope that it is the root cause. But with ACPI being ACPI and with how horribly Microsoft (of all "people") abuses ACPI in their surface line I would not be surprised if it is the root cause.

I do not expect the 'clocksource/drivers/timer-ti-dm: Do reset before enable' commit to have any impact, that changes code for the TI OMAP series chips.

Can someone provide me an acpidump of the Microsoft Surface Go 2 please ? Then I can check if the i2c-acpi commit is expected to have any influence there.

[kbingham]

Hi Hans ;-) Thanks for jumping over! Did you find this thread from somewhere ?

ACPI dumps for this device are at :
https://github.com/linux-surface/acpidumps/tree/master/surface_go

[kitakar5525]

and acpidump for Surface Go 2 is here:
https://github.com/linux-surface/acpidumps/tree/master/surface_go_2