OCC sensor can not be activated #58

Kenthliu · 2016-03-17T08:46:00Z

We found some board can not get OCC sensor.
Hwmon driver did not appear in /sys/class/hwmon
Attachment is the hconsole log and journalctrl log.
OCCfail.txt

Linux version 4.3.6-openbmc-20160222-1 (openpower@openpower-VirtualBox) (gcc version 4.9.3 (GCC) ) #1 Tue Mar 8 14:03:29 CST 2016

[941]: Installing OCC device
------------[ cut here ]------------
WARNING: CPU: 0 PID: 1072 at /home/openpower/openbmc/build/tmp/work-shared/barreleye/kernel-source/fs/sysfs/dir.c:31 sysfs_warn_dup+0x50/0x70()
sysfs: cannot create duplicate filename '/devices/platform/ahb/ahb:apb/1e78a000.i2c/i2c-3/i2c-3/3-0050/hwmon/hwmon3/freq1_input'
Modules linked in:
CPU: 0 PID: 1072 Comm: sh Not tainted 4.3.6-openbmc-20160222-1 #1
Hardware name: ASpeed SoC
[<c000f28c>] (unwind_backtrace) from [<c000cf0c>] (show_stack+0x10/0x14)
[<c000cf0c>] (show_stack) from [<c0016734>] (warn_slowpath_common+0x84/0xac)
[<c0016734>] (warn_slowpath_common) from [<c0016788>] (warn_slowpath_fmt+0x2c/0x3c)
[<c0016788>] (warn_slowpath_fmt) from [<c00dd914>] (sysfs_warn_dup+0x50/0x70)
[<c00dd914>] (sysfs_warn_dup) from [<c00dd674>] (sysfs_add_file_mode_ns+0xfc/0x180)
[<c00dd674>] (sysfs_add_file_mode_ns) from [<c00ddf80>] (internal_create_group+0x18c/0x24c)
[<c00ddf80>] (internal_create_group) from [<c0295650>] (set_occ_online+0x19c/0x35c)
[<c0295650>] (set_occ_online) from [<c00dcb28>] (kernfs_fop_write+0x128/0x188)
[<c00dcb28>] (kernfs_fop_write) from [<c008caf4>] (__vfs_write+0x20/0xd0)
[<c008caf4>] (__vfs_write) from [<c008d1a8>] (vfs_write+0xa8/0x130)
[<c008d1a8>] (vfs_write) from [<c008d8a0>] (SyS_write+0x40/0x80)
[<c008d8a0>] (SyS_write) from [<c000a280>] (ret_fast_syscall+0x0/0x38)
---[ end trace 31651e1056550963 ]---
occ-i2c 3-0050: error create freq sysfs entry

The text was updated successfully, but these errors were encountered:

nkskjames · 2016-03-17T14:46:45Z

It sounds like the hardware is definitely bad, so I'm calling this an error handling enhancement as opposed to a bug. The hwmon driver should have a better error message.

adamliyi · 2016-03-18T03:06:49Z

I'd propose a workaround just for testing (still now clear about what is the root cause):
See attach.

power8_occ_i2c.c.patch.txt

diff --git a/drivers/hwmon/power8_occ_i2c.c b/drivers/hwmon/power8_occ_i2c.c
index c9c70d1..c5f843a 100644
--- a/drivers/hwmon/power8_occ_i2c.c
+++ b/drivers/hwmon/power8_occ_i2c.c
@@ -1317,6 +1317,12 @@ static ssize_t set_occ_online(struct device *dev,
            return PTR_ERR(data->hwmon_dev);

        err = occ_create_hwmon_attribute(dev);
+       /* Workaround: For some reason, a sysfs entry already existed.
+        * occ_create_hwmon_attribute() will do clean up.
+        * Try to create again. */
+       if (err == -EEXIST) {
+           err = occ_create_hwmon_attribute(dev);
+       }
        if (err) {
            hwmon_device_unregister(data->hwmon_dev);
            return err;
@@ -1325,7 +1331,7 @@ static ssize_t set_occ_online(struct device *dev,
        dev_dbg(dev, "%s: sensor '%s'\n",
            dev_name(data->hwmon_dev), data->client->name);
    } else if (val == 0) {
-       if (data->occ_online == 0)
+       if (data->occ_online == 0 && !data->hwmon_dev)
            return count;

        occ_remove_sysfs_files(data->hwmon_dev);

shenki · 2016-03-18T03:36:09Z

Have you attempted to work out why the sysfs entry already exists?

shenki · 2016-03-18T03:37:24Z

This is a bug in the occ driver that needs to be resolved.

adamliyi · 2016-03-18T08:44:41Z

@shenki , Thanks and I now know how to paste code in comments correctly :-).
The problem is that I cannot reproduce this issue on Barreleye - make it hard to debug. According to @Kenthliu, the issue happens when poweron host. I tried "obmcutil poweron" several times, but no such warning appears. And I just proposed this workaround for @Kenthliu to test on his Barreleye - see if OCC driver can still be used even with such warning. Next step we need to find out the root cause for duplicated occ sysfs attributes.

Kenthliu · 2016-03-21T03:49:23Z

Hi @adamliyi and @shenki
I saw different symptom in different system.
I ran the version that add @adamliyi 's patch. But OCC sensor still did not show up.
I suppose it's the same fail to previous one but it seems not.

adamliyi · 2016-03-21T06:27:00Z

@Kenthliu , the error message in your last comment looks like an issue of the OCC itself, not BMC.

Kenthliu · 2016-03-22T07:26:28Z

I tried @adamliyi solution. It will fix this issue. Here is the log. I found another error message in that system, it seems another issue,please help to check it. @nkskjames
occputty2.txt

shenki · 2016-03-22T07:34:08Z

@Kenthliu you need to open a new bug with this error, it's not related to the original bug.

@williamspatrick how do you want these to be tracked? Should we get people to report the issue against open-power/hostboot?

adamliyi · 2016-03-22T08:17:22Z

Submitted a pull request to fix this issue: openbmc/linux#65
@Kenthliu has tested on his 12-core Barreleye and the "duplicated filename" warning disappears.
This fix is not good. It is still hard-coded the sensor number for 12-core CPU.
I will try to create the occ hwmon sysfs entries dynamically based on sensor number polled from OCC. But I am not sure I can catch the next release window.

…upport more sensors This patch fixes issue: openbmc/skeleton#58 The hwmon sys attributes are created using statically defined arrays. Some POWER CPU has 10-core, while some POWER CPU has 12-core. The more cores, the more OCC sensors. E.g, for 12-core CPU, there will be 28 temperature sensors. The statically defined array will overflow in this case. This is a temporary fix. Will need to generate the hwmon sysfs attributes dynamically. Signed-off-by: Yi Li <adamliyi@msn.com>

williamspatrick · 2016-03-22T15:12:23Z

Errors reported by "0xE500" correspond to the diagnostics component, so
they are not typically a software issue. This one in particular is a
memory ECC UE:

Mba.prf.err.C: PRDR_ERROR_SIGNATURE ( 0xffff0010, "", "Maintenance UE")

On Tue, Mar 22, 2016 at 12:26:28AM -0700, KenLiu wrote:

I tried @adamliyi solution. It will fix this issue. Here is the log. I found another error message in that system, it seems another issue,please help to check it. @nkskjames
occputty2.txt

117.60312|================================================
117.60313|Error reported by unknown (0xE500)
117.60313|
117.60313| ModuleId 0x0b unknown
117.60314| ReasonCode 0xe504 unknown
117.60314| UserData1 unknown : 0x000d000a00000404
117.60315| UserData2 unknown : 0xffff001000000000
117.60315|User Data Section 0, type UD
117.60315| Subsection type 0x06
117.60316| ComponentId errl (0x0100)
117.60316| CALLOUT
117.60316| HW CALLOUT
117.60316| Reporting CPU ID: 8
117.60317| Called out entity:
117.60317|User Data Section 1, type UD
117.60317| Subsection type 0x33
117.60318| ComponentId unknown (0xe500)
117.60318|User Data Section 2, type UD
117.60318| Subsection type 0x01
117.60319| ComponentId unknown (0xe500)
117.60319| STRING
117.60319|
117.60319|User Data Section 3, type UD
117.60320| Subsection type 0x15
117.60320| ComponentId hb-trace (0x3100)
117.60320|User Data Section 4, type UD
117.60321| Subsection type 0x03
117.60321| ComponentId errl (0x0100)
117.60321|User Data Section 5, type UD
117.60322| Subsection type 0x01
117.60322| ComponentId errl (0x0100)
117.60322| STRING
117.60322| Hostboot Build ID:
117.60323|User Data Section 6, type UD
117.60323| Subsection type 0x04
117.60323| ComponentId errl (0x0100)
117.60324|================================================

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#58 (comment)

Patrick Williams

Kenthliu · 2016-03-23T08:14:22Z

@williamspatrick So you think this is a another HW issue that memory has something wrong? I think this is related to the SEL log we saw as attachment.
Dimm16 log.txt

Kenthliu · 2016-03-23T09:14:12Z

@adamliyi image-bmc_occ_0322_final works fine. This issue can be closed now. For the DIMM issue, I will open another issue if needed. Thanks.

nkskjames · 2016-04-29T11:47:09Z

@Kenthliu I would like to close this, but looks like you need to open another issue against the DIMM fail. I will wait to close until other issue is opened so we don't forget.

Kenthliu · 2016-05-03T07:22:42Z

@nkskjames I open the issue at #71. But the symptom could not 100% reproduce in every system.

Kenthliu · 2016-05-11T11:53:45Z

This issue happened again in 0.7. Please help to notice.
occfail2.txt

adamliyi · 2016-05-11T13:44:44Z

obmc_v0.7 does not include my temporary fix in: openbmc/linux#65. So this bug appears again.

This patch is a linux-4.4 port of previous patch: openbmc@f8087df It is created as the backup workaround in order to catch 5/19 release, for the blocking occ issue: openbmc/skeleton#58 I am working on a new fix to dynamically create those sysfs attributes. If the new fix cannot catch 5/19 release, we can use this patch temporarily. When the new fix come out, this patch can be replaced. Signed-off-by: Yi Li <adamliyi@msn.com>

This patch fixes issue: openbmc/skeleton#58. OCC sensor number varies for different platforms. The patch creates hwmon sysfs attributes dynamically, using sensor information get from OCC. Previously the sysfs attributes are created using statically defined data structures. Signed-off-by: Yi Li <adamliyi@msn.com>

This patch fixes issue: openbmc/skeleton#58. OCC sensor number varies for different platforms. The patch creates hwmon sysfs attributes dynamically, using sensor information get from OCC. Previously the sysfs attributes are created using statically defined data structures. Signed-off-by: Yi Li <adamliyi@msn.com> Signed-off-by: Joel Stanley <joel@jms.id.au>

williamspatrick · 2016-05-26T03:26:29Z

I believe this is resolved. Please re-open if observed in latest tag.

This patch fixes issue: openbmc/skeleton#58. OCC sensor number varies for different platforms. The patch creates hwmon sysfs attributes dynamically, using sensor information get from OCC. Previously the sysfs attributes are created using statically defined data structures. Signed-off-by: Yi Li <adamliyi@msn.com> Signed-off-by: Joel Stanley <joel@jms.id.au>

nkskjames added the enhancement label Mar 17, 2016

nkskjames added bug and removed enhancement labels Mar 21, 2016

adamliyi mentioned this issue Mar 22, 2016

hwmon: power8_occ_i2c: Add more hwmon sysfs attribute definition to s… openbmc/linux#65

Closed

adamliyi mentioned this issue May 16, 2016

hwmon: power8_occ_i2c: Add more hwmon sysfs attribute definition openbmc/linux#81

Closed

adamliyi mentioned this issue May 20, 2016

hwmon: power8_occ_i2c: create hwmon sysfs attributes dynamically openbmc/linux#84

Closed

williamspatrick closed this as completed May 26, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCC sensor can not be activated #58

OCC sensor can not be activated #58

Kenthliu commented Mar 17, 2016

nkskjames commented Mar 17, 2016

adamliyi commented Mar 18, 2016

shenki commented Mar 18, 2016

shenki commented Mar 18, 2016

adamliyi commented Mar 18, 2016

Kenthliu commented Mar 21, 2016

adamliyi commented Mar 21, 2016

Kenthliu commented Mar 22, 2016

shenki commented Mar 22, 2016

adamliyi commented Mar 22, 2016

williamspatrick commented Mar 22, 2016

Kenthliu commented Mar 23, 2016

Kenthliu commented Mar 23, 2016

nkskjames commented Apr 29, 2016

Kenthliu commented May 3, 2016

Kenthliu commented May 11, 2016

adamliyi commented May 11, 2016

williamspatrick commented May 26, 2016

OCC sensor can not be activated #58

OCC sensor can not be activated #58

Comments

Kenthliu commented Mar 17, 2016

nkskjames commented Mar 17, 2016

adamliyi commented Mar 18, 2016

shenki commented Mar 18, 2016

shenki commented Mar 18, 2016

adamliyi commented Mar 18, 2016

Kenthliu commented Mar 21, 2016

adamliyi commented Mar 21, 2016

Kenthliu commented Mar 22, 2016

shenki commented Mar 22, 2016

adamliyi commented Mar 22, 2016

williamspatrick commented Mar 22, 2016

Kenthliu commented Mar 23, 2016

Kenthliu commented Mar 23, 2016

nkskjames commented Apr 29, 2016

Kenthliu commented May 3, 2016

Kenthliu commented May 11, 2016

adamliyi commented May 11, 2016

williamspatrick commented May 26, 2016