[Xavier NX Devkit SD-CARD] Occasional cboot panic and halt on 32.6.1 after reboot #891
Comments
|
That it's triggering an exception like that is probably a bug - maybe not properly handling the case where no boot device is found? Considering how complicated the t19x boot sequence has become as they've tacked on support for NVME, extlinux, etc., CBO, etc., it wouldn't surprise me that some infrequently-exercised code path is broken. Still, bugs happen, and cboot should have a watchdog timer enabled by default, unless you've configured otherwise. I don't remember the default timeout off the top of my head, but it's on the order of minutes. |
|
Thanks Matt, I left it for about 5 minutes and nothing happened but I'll reproduce this again and leave it for longer. I was reading in the docs that the PMIC WDT is disabled by default from odmdata on the Xavier NX and that's why I was asking. I'll get back with an update. |
|
@madisongh Unfortunately the device did not restart after 40 minutes in this state, perhaps that watchdog is not enabled by default? |
|
Sure enough, the default ODMDATA ( |
|
Hi @madisongh , unfortunately the PMIC WDT doesn't appear to work with systemd, the device reboots as soon as it finishes booting. We're using systemd to kick the watchdog with RuntimeWatchdogSec set to 10s. This doesn't happen with PMIC WDT disabled in the previous ODMDATA though. |
|
Ah, right, you may need to modify your device tree and/or kernel config and/or systemd configuration to switch to using the PMIC WDT from Linux, too. I believe the driver is enabled by default in the default kernel config (CONFIG_MAX77620_WATCHDOG), but it may not be enabled by default in the device tree. Even if it is, if there are multiple watchdog devices, systemd uses |
|
@madisongh yes, the config is enabled and from what I see the plugin manager also enables it in the device tree based on the selected odmdata. There are no other watchdog devices created (only watchdog and watchdog0) and if I remove the watchdog.conf file systemd no longer opens the device and the board doesn't reboot. If however i do a cat on /dev/watchdog the board is rebooted immediately. |
|
Now that you mention it, I vaguely remember running into a similar problem on the TX2 a while back. IIRC there was a bug somewhere that kept the SoC internal WDT around even when I chose the PMIC WDT. Looking at my kernel config, I think the solution was to just disable the support for the other WDTs there - here's my I don't remember now whether the bug was in cboot or the kernel, but there could be something similar going on here. |
|
Many thanks @madisongh! indeed, disabling the other tegra watchdog modules made the max77620-watchdog work with systemd. |
|
Hi @madisongh, unfortunately enabling the PMIC WDT did not solve the original cboot problem, I managed to reproduce the issue once more and left the board running for 30 minutes, the pmic did not reset it though. The easiest way to reproduce the watchdog not resetting the device is to flash the NX SD with that odmdata and then remove the sd-card or any other medium from which it could boot before powering it on. I guess that maybe cboot doesn't actually start this watchdog? I see it's overriding some nodes in the dtb but a not sure if they have anything to do with this: |
|
Hmm. I took a closer look at the cboot code, and sure enough, it disables the WDT as part of the kernel handoff. I don't know if that's a recent change, or whether I was misremembering how it worked, but that surprised me. :( Also, it turns out that cboot always uses the internal WDT, even if you've chosen the PMIC WDT via ODMDATA. Some additional patches to cboot will be needed to addresses these shortcomings. |
|
Thanks again for looking into this @madisongh ! I see that tegrabl_reset() makes the boot sequence restart from MB1, at least on the NX devkit, and am wondering if resetting instead of halting is a good idea. I'll try to reproduce the problem to see if cboot is able to load the kernel and dtb from the raw partitions after reset with this: |
|
Yes, looks like doing the tegrabl_reset() after the panic caused by sd-card partitions read failure makes the nx boot normally: |
|
@acostach Great, that looks like a much simpler way to solve the problem you're seeing. Please leave this issue open, though, as I will try and solve the problems with cboot's WDT handling, when I get a chance. |
Not sure if this has been happening with older L4Ts but I've noticed this sporadic panic in cboot 32.6.1, happens occasionally after rebooting the device multiple times:
I guess the sd-card is sometimes in a bad state so I'm wondering if it would it be possible to configure cboot to simply reboot in this case instead of halting? Thank you
The text was updated successfully, but these errors were encountered: