[26.04_linux-nvidia] NVIDIA: SAUCE: watchdog: sbsa_gwdt: stop the watchdog across the whole system-sleep transition - #568
Conversation
…e system-sleep transition The driver stops a running watchdog in its own device suspend callback and restarts it in its resume callback. That leaves the watchdog armed, with nobody refreshing it, for the entire early part of suspend entry: userspace freeze, kernel thread freeze, and every device suspend callback that runs before this device's own. The same window exists at the tail end of resume. When the watchdog is running from boot (early_enable=1, previously force_enable=1 downstream; 10 s default timeout) and any device stalls its suspend callback past the timeout, the watchdog resets the system in the middle of suspend entry. On N1x (Yukon) this fired on about 7% of suspend attempts in a randomized stress run (9 resets in 124 suspends; the serial console shows the board dropping into the boot ROM mid-entry with the watchdog reset status set in NONRST_REG2). Two elimination runs confirm the mechanism: the identical stress matrix with the parameter off produced zero resets in 118 suspends, and with this patch applied and the watchdog force-enabled, zero resets in 198 suspends across four runs (the baseline rate predicts about 14). Stop the watchdog from a PM notifier at the *_PREPARE events, before tasks are frozen and device callbacks run, and restart it at the PM_POST_* events, after everything has resumed. The existing dev_pm_ops are kept: their stop/start are idempotent register writes and they preserve the current behaviour if the notifier registration ever fails. This is also upstream-relevant as a companion to the early_enable parameter: the armed-during-entry window exists for any system running the SBSA watchdog from boot. Signed-off-by: David Cemin <dcemin@nvidia.com>
BaseOS Kernel ReviewWarning
|
PR Validation ReportPatchscan ✅ No Missing FixesAll cherry-picked commits checked — no missing upstream fixes found. PR Lint ✅ All checks passedDetailsChecking 1 commits... Cherry-pick digest: ┌──────────────┬──────────────────────────────────────────────────────────────────┬────────────┬─────────┬───────────────────────────┐ │ Local │ Referenced upstream / Patch subject │ Patch-ID │ Subject │ SoB chain │ ├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤ │ 4c4bd0ced295 │ [SAUCE] watchdog: sbsa_gwdt: stop the watchdog across the whole │ N/A │ N/A │ dcemin │ └──────────────┴──────────────────────────────────────────────────────────────────┴────────────┴─────────┴───────────────────────────┘ Lint: all checks passed. |
|
@dcemin-nv I created the Launchpad bug for this change: Could you please add this link to the PR description? Also we need below information to track these patches: |
|
|
Tracking info: internal NVbug 6611666 (watchdog initiated reset during suspend entry on the N1x stress runs). Upstream plan: the patch is prepared against current mainline and is being posted to linux-watchdog (Wim Van Sebroeck, Guenter Roeck cc linux-watchdog@vger.kernel.org); I will add the lore link here once it is archived. |
3 participants
BugLink: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia/+bug/2166302
Fix for a watchdog initiated system reset during suspend entry, found and root caused on the N1x laptop program.
The driver stops a running watchdog only in its device suspend callback, which leaves it armed with nobody refreshing it through userspace freeze, kernel thread freeze, and every device suspend callback that runs earlier (same window at the tail of resume). With the watchdog running from boot and a 10 second timeout, any slow device suspend resets the system mid entry. On N1x this fired on about 7 percent of suspend attempts in randomized stress runs (9 resets in 124 suspends); two elimination runs confirm the mechanism (0 resets in 118 with the watchdog off, 0 in 198 with this fix and the watchdog force enabled, where the baseline rate predicts about 14).
The fix moves the stop to a PM notifier at the PREPARE events and the restart to PM_POST, keeping the existing dev_pm_ops as an idempotent fallback.
Validation: board proven on Yukon (the 198 suspend elimination matrix plus a 200 cycle confirmation run on the early_enable rename with zero watchdog resets); shipping in the N1x FastOS kernel.
Notes for review: