[BUG][TGL] failed to ready HDA controller gctl 0x0 (sof_probe_work failed) #2571

sathya-nujella · 2020-11-20T06:58:19Z

DUT: TGL Chrome DUT with Max98373_RT5682 I2S
BIOS: Coreboot
Kernel: Google - Kernel 5.4
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/refs/heads/chromeos-5.4

Issue:
Sound card creation failed.

TEST SCENARIO:
During reboot tests, randomly seeing this issue.
Able to reproduce the issue (no consistent method, just reboot some times catch this issue).

From logs:
1) error from dmesg:
[ 8.912119] sof-audio-pci 0000:00:1f.3: error: failed to ready HDA controller gctl 0x0
[ 8.922070] sof-audio-pci 0000:00:1f.3: error: get caps error
[ 8.929600] sof-audio-pci 0000:00:1f.3: display power disable
[ 8.929622] sof-audio-pci 0000:00:1f.3: error: failed to probe DSP -5
[ 8.937423] sof-audio-pci 0000:00:1f.3: error: sof_probe_work failed err: -5

lspci -vvvv //1f.3 part
00:1f.3 Multimedia audio controller: Intel Corporation Device a0c8 (rev 20)
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 16
Region 0: Memory at 7ff48000 (64-bit, non-prefetchable) [size=16K]
Region 4: Memory at 7fe00000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Vendor Specific Information: Len=14 <?>
Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel driver in use: sof-audio-pci
Kernel modules: snd_hda_intel, snd_sof_pci

kv2019i · 2020-11-20T08:39:48Z

There are a couple of past bugs with similar fingerprint:
#1399
#1556
#1735

Most have such low reproduce rates, that no exact patch could be pointed to for making the problem go away.

One concrete thing to try is this:
#1735 (comment)

kv2019i · 2020-11-20T16:43:50Z

Findings today:

with snd-hda-intel, bug is harder to trigger (on same system)
when bug occurs, unloading and reloading does not help
when bug occurs, unloading and reloading snd-hda-intel instead of SOF, does not help

Yet to be tested:

try to reproduce with force-nocodec mode
unloading driver always before reboot
increase delay to wait HDA to come from reset

plbossart · 2020-11-20T16:59:31Z

Right, we faced this before and never reached a conclusion as to what cause the capability parsing to fail. I had a discussion with hardware folks and there was no smoking gun.

A work-around was added in upstream commit f09e9c7 ('ASoC: SOF: Intel: hda-ctrl: add reset cycle before parsing capabilities')

@sathya-nujella can you check if this was back-ported and try to revert it if yes?

ranj063 · 2020-11-20T17:01:09Z

Right, we faced this before and never reached a conclusion as to what cause the capability parsing to fail. I had a discussion with hardware folks and there was no smoking gun.

A work-around was added in upstream commit f09e9c7 ('ASoC: SOF: Intel: hda-ctrl: add reset cycle before parsing capabilities')

@sathya-nujella can you check if this was back-ported and try to revert it if yes?

@plbossart I already suggested this last night and it didnt help

kv2019i · 2020-11-20T17:21:45Z

More tests:

try to reproduce with force-nocodec mode -> still happens
increase delay to wait HDA to come from reset -> done, did not help

Ongoing:

unloading driver always before reboot -> under progress, @sathya-nujella running, no errors yet but very limited iterations

plbossart · 2020-11-20T17:25:30Z

@ranj063 if the unloading driver before reboot solves the issue, wouldn't this point to a .shutdown() callback that's not doing the same thing as a .remove()?

ranj063 · 2020-11-20T17:41:26Z

@ranj063 if the unloading driver before reboot solves the issue, wouldn't this point to a .shutdown() callback that's not doing the same thing as a .remove()?

@plbossart I'd think so. I'll work with @sathya-nujella this morning to try it out.

sathya-nujella · 2020-11-22T05:27:23Z

Thank you @ranj063 . In today's debug with Ranjani's support tried, warm reboot tests.
With her suggestion, found out that after reverting this SOF patch, issue not reproduced in 100-iterations.
As it is random issue, I will continue more tests. But clearly reverting that SOF patch shows improvement.

Patch reverted: thesofproject/sof@75561a9

SOF issue reported based on this: thesofproject/sof#3633

mengdonglin · 2020-11-23T10:25:44Z

Let's track the issue in thesofproject/sof#3633 atm

Invoke hda_dsp_shutdown() as the .shutdown() callback, this will help to perform needed operation on TGL platforms before shutting down or rebooting the system. Fixes: thesofproject#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com>

Invoke hda_dsp_shutdown() as the .shutdown() callback, this will help to perform needed operation on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com>

mengdonglin · 2020-11-30T01:47:20Z

Reopen since we also need fix in kernel driver

keyonjie · 2020-11-30T08:31:02Z

We have a driver PR aim to address this: #2581

Unfortunately, it can't fix the issue thoroughly, there is still a tricky point to hit the issue: if we perform system reboot at the point that the WoV is switching the DSP from D0 to D0I3(e.g. at around 5 Seconds(+-0.5S) after the WoV is switched on), we can hit this within 10 iterations.

Invoke hda_dsp_remove() as the .shutdown() callback, this will help to perform shutdown the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com>

keyonjie · 2020-12-21T01:16:58Z

the PR #2600 will fix this issue also. @ranj063 FYI.

Invoke hda_dsp_remove() as the .shutdown() callback, this will help to perform shutdown the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com>

keyonjie · 2020-12-28T02:36:50Z

@sathya-nujella PR merged, can you check if we can close this?

sathya-nujella · 2020-12-28T15:31:37Z

@sathya-nujella PR merged, can you check if we can close this?

Hi @keyonjie ,
Original reported issue was fixed.
But later, we also observed the same error prints in this scenario as we updated in email,
i.e. "wov tests via cmd line + reboot".

For this, as you and @ranj063 suggested, we ran tests with including linux PR's (both of them included): #2600 and #2581.
It passed 500+ cycles after including fixes from PR's: #2581 and #2600.
We will continue tests one more day and update on this.

jairaj-arava · 2020-12-28T21:36:44Z

We have verified the speaker and headset playback too with the above PR's and seems the functionality is good.

Invoke hda_dsp_remove() as the .shutdown() callback, this will help to perform shutdown the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: #2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com>

Invoke hda_dsp_remove() as the .shutdown() callback, this will help to perform shutdown the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com> Reviewed-by: Bard Liao <bard.liao@intel.com> Reviewed-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

Invoke hda_dsp_remove() as the .shutdown() callback, this will help to perform shutdown the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: #2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com> Reviewed-by: Bard Liao <bard.liao@intel.com> Reviewed-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

Invoke hda_dsp_remove() as the .shutdown() callback. This will help to perform shutdown of the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com> Reviewed-by: Bard Liao <bard.liao@intel.com> Reviewed-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>

Invoke hda_dsp_remove() as the .shutdown() callback. This will help to perform shutdown of the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject/linux#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com> Reviewed-by: Bard Liao <bard.liao@intel.com> Reviewed-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> Link: https://lore.kernel.org/r/20210113152617.4048541-4-kai.vehmanen@linux.intel.com Signed-off-by: Mark Brown <broonie@kernel.org>

Invoke hda_dsp_remove() as the .shutdown() callback, this will help to perform shutdown the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com> Reviewed-by: Bard Liao <bard.liao@intel.com> Reviewed-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>

…back Invoke hda_dsp_remove() as the .shutdown() callback. This will help to perform shutdown of the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject/linux#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com> Reviewed-by: Bard Liao <bard.liao@intel.com> Reviewed-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> Link: https://lore.kernel.org/r/20210113152617.4048541-4-kai.vehmanen@linux.intel.com Signed-off-by: Mark Brown <broonie@kernel.org> (cherry picked from commit 44a4cfa git://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git for-next) BUG=b:175915544 TEST=verified the audio Signed-off-by: Jairaj Arava<jairaj.arava@intel.com> Signed-off-by: Sathyanarayana Nujella <sathyanarayana.nujella@intel.com> Change-Id: Ifb3a8dd1c8982dff7766063071020b9d826e0d95 Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/2628182 Reviewed-by: Sean Paul <seanpaul@chromium.org> Reviewed-by: Curtis Malainey <cujomalainey@chromium.org> Tested-by: Jairaj Arava <jairaj.arava@intel.corp-partner.google.com> Commit-Queue: Curtis Malainey <cujomalainey@chromium.org> (cherry picked from commit 3916179) Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/2643522 Tested-by: Curtis Malainey <cujomalainey@chromium.org>

BugLink: https://bugs.launchpad.net/bugs/1919930 Invoke hda_dsp_remove() as the .shutdown() callback. This will help to perform shutdown of the DSP safely on TGL platforms before shutting down or rebooting the system. BugLink: thesofproject#2571 Signed-off-by: Keyon Jie <yang.jie@linux.intel.com> Reviewed-by: Bard Liao <bard.liao@intel.com> Reviewed-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com> Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com> Link: https://lore.kernel.org/r/20210113152617.4048541-4-kai.vehmanen@linux.intel.com Signed-off-by: Mark Brown <broonie@kernel.org> (cherry picked from commit 44a4cfa) Signed-off-by: Hui Wang <hui.wang@canonical.com> Acked-by: Timo Aaltonen <tjaalton@ubuntu.com> Acked-by: Tim Gardner <tim.gardner@canonical.com> Signed-off-by: Tim Gardner <tim.gardner@canonical.com>

sathya-nujella added P1 Blocker bugs or important features TGL Applies to Tiger Lake platform labels Nov 20, 2020

kv2019i added bug Something isn't working Chrome Related to Chrome integration labels Nov 20, 2020

sathya-nujella mentioned this issue Nov 22, 2020

[BUG] [TGL] HDA controller reset fails in reboot tests thesofproject/sof#3633

Closed

mengdonglin closed this as completed Nov 23, 2020

keyonjie mentioned this issue Nov 25, 2020

ASoC: SOF: sof-pci-dev: add .shutdown callback #2581

Merged

mengdonglin reopened this Nov 30, 2020

keyonjie mentioned this issue Dec 7, 2020

ASoC: SOF: add mutex to protect the dsp_power_state access #2600

Merged

bardliao closed this as completed in #2581 Dec 29, 2020

sathya-nujella mentioned this issue Aug 3, 2021

TGL; sound card binding fails with error "failed to ready HDA controller" #3078

Closed

mengdonglin added the HDA reset failure HD-Audio controller failed to reset label Aug 23, 2021

ranj063 mentioned this issue Dec 7, 2022

[BUG] failed to ready HDA controller gctl 0x0 when suspend-resume-with-capture on MTL platforms #4077

Closed

rfvirgil mentioned this issue Sep 14, 2023

UpXtreme TGL + 6.3 kernel: failed to ready HDA controller gctl 0x0 #4589

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG][TGL] failed to ready HDA controller gctl 0x0 (sof_probe_work failed) #2571

[BUG][TGL] failed to ready HDA controller gctl 0x0 (sof_probe_work failed) #2571

sathya-nujella commented Nov 20, 2020

kv2019i commented Nov 20, 2020

kv2019i commented Nov 20, 2020

plbossart commented Nov 20, 2020

ranj063 commented Nov 20, 2020

kv2019i commented Nov 20, 2020

plbossart commented Nov 20, 2020

ranj063 commented Nov 20, 2020

sathya-nujella commented Nov 22, 2020

mengdonglin commented Nov 23, 2020

mengdonglin commented Nov 30, 2020

keyonjie commented Nov 30, 2020 •

edited

Loading

keyonjie commented Dec 21, 2020

keyonjie commented Dec 28, 2020

sathya-nujella commented Dec 28, 2020

jairaj-arava commented Dec 28, 2020

[BUG][TGL] failed to ready HDA controller gctl 0x0 (sof_probe_work failed) #2571

[BUG][TGL] failed to ready HDA controller gctl 0x0 (sof_probe_work failed) #2571

Comments

sathya-nujella commented Nov 20, 2020

kv2019i commented Nov 20, 2020

kv2019i commented Nov 20, 2020

plbossart commented Nov 20, 2020

ranj063 commented Nov 20, 2020

kv2019i commented Nov 20, 2020

plbossart commented Nov 20, 2020

ranj063 commented Nov 20, 2020

sathya-nujella commented Nov 22, 2020

mengdonglin commented Nov 23, 2020

mengdonglin commented Nov 30, 2020

keyonjie commented Nov 30, 2020 • edited Loading

keyonjie commented Dec 21, 2020

keyonjie commented Dec 28, 2020

sathya-nujella commented Dec 28, 2020

jairaj-arava commented Dec 28, 2020

keyonjie commented Nov 30, 2020 •

edited

Loading