OCPBUGS-100376: fibre_channel: fix crash when attempting to dereference invalid count…

[jan--f]

…ers (prometheus#3573)

The statistics counters of a disabled FC host cannot be read from sysfs and any read attempt fails with errno ENOENT. Therefore, all statistics counters returned by procfs are nil in such a case. This results in a crash on s390x Linux when a zFCP host is disabled with chzdev -d <host id>.

Crash stacktrace in a GDB

$ gdb /root/node_exporter/node_exporter
...
Thread 4 "node_exporter" received signal SIGSEGV, Segmentation fault. [Switching to LWP 35579]
0x00000000007be86c in github.com/prometheus/node_exporter/collector.(*fibrechannelCollector).Update (c=0xc00012fef0, ch=0xc0002fc8c0, ~r0=...)
at /root/node_exporter/collector/fibrechannel_linux.go:133
133 c.pushCounter(ch, "dumped_frames_total", *host.Counters.DumpedFrames, *host.Name)
(gdb) bt
at /root/node_exporter/collector/fibrechannel_linux.go:133
(gdb) p host
$1 = {Name = 0xc00038faf0, Speed = 0xc00038fb00, PortState = 0xc00038fb10, PortType = 0xc00038fb30, SymbolicName = 0xc00038fba0, NodeName = 0xc00038fb40, PortID = 0xc00038fb50,
PortName = 0xc00038fb60, FabricName = 0xc00038fb80, DevLossTMO = 0xc00038fb90, SupportedClasses = 0xc00038fbb0, SupportedSpeeds = 0xc00038fbd0, Counters = 0xc0001cb2d0}
(gdb) p host.Name
$2 = (string *) 0xc00038faf0
(gdb) p *host.Name
$3 = "host0"
(gdb) p *host.Counters.DumpedFrames
❌️ Cannot access memory at address 0x0
(gdb) p *host.Counters
$4 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0, RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0, InvalidTXWordCount = 0x0,
LinkFailureCount = 0x0, LossOfSyncCount = 0x0, LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb) bt
at /root/node_exporter/collector/fibrechannel_linux.go:133
name="fibrechannel", c=..., ch=0xc0002fc8c0, logger=0xc0002104a0)
at /root/node_exporter/collector/collector.go:160
(gdb) p *host.Counters
$5 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0,
RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0,
InvalidTXWordCount = 0x0, LinkFailureCount = 0x0, LossOfSyncCount = 0x0,
LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb)

(cherry picked from commit 366f908)

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@jan--f: This pull request references Jira Issue OCPBUGS-100376, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

…ers (prometheus#3573)

The statistics counters of a disabled FC host cannot be read from sysfs and any read attempt fails with errno ENOENT. Therefore, all statistics counters returned by procfs are nil in such a case. This results in a crash on s390x Linux when a zFCP host is disabled with chzdev -d <host id>.

Crash stacktrace in a GDB

$ gdb /root/node_exporter/node_exporter
...
Thread 4 "node_exporter" received signal SIGSEGV, Segmentation fault. [Switching to LWP 35579]
0x00000000007be86c in github.com/prometheus/node_exporter/collector.(*fibrechannelCollector).Update (c=0xc00012fef0, ch=0xc0002fc8c0, ~r0=...)
at /root/node_exporter/collector/fibrechannel_linux.go:133
133 c.pushCounter(ch, "dumped_frames_total", *host.Counters.DumpedFrames, *host.Name)
(gdb) bt
at /root/node_exporter/collector/fibrechannel_linux.go:133
(gdb) p host
$1 = {Name = 0xc00038faf0, Speed = 0xc00038fb00, PortState = 0xc00038fb10, PortType = 0xc00038fb30, SymbolicName = 0xc00038fba0, NodeName = 0xc00038fb40, PortID = 0xc00038fb50,
PortName = 0xc00038fb60, FabricName = 0xc00038fb80, DevLossTMO = 0xc00038fb90, SupportedClasses = 0xc00038fbb0, SupportedSpeeds = 0xc00038fbd0, Counters = 0xc0001cb2d0}
(gdb) p host.Name
$2 = (string *) 0xc00038faf0
(gdb) p *host.Name
$3 = "host0"
(gdb) p *host.Counters.DumpedFrames
❌️ Cannot access memory at address 0x0
(gdb) p *host.Counters
$4 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0, RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0, InvalidTXWordCount = 0x0,
LinkFailureCount = 0x0, LossOfSyncCount = 0x0, LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb) bt
at /root/node_exporter/collector/fibrechannel_linux.go:133
name="fibrechannel", c=..., ch=0xc0002fc8c0, logger=0xc0002104a0)
at /root/node_exporter/collector/collector.go:160
(gdb) p *host.Counters
$5 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0,
RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0,
InvalidTXWordCount = 0x0, LinkFailureCount = 0x0, LossOfSyncCount = 0x0,
LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb)

(cherry picked from commit 366f908)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[danielmellado]

/lgtm

[openshift-merge-bot]

Scheduling required tests:
/test e2e-agnostic-cmo
/test e2e-aws
/test e2e-aws-upgrade

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danielmellado, jan--f

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [danielmellado,jan--f]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[simonpasquier]

/verified by unit tests

[openshift-ci-robot]

@simonpasquier: This PR has been marked as verified by unit tests.

Details

In response to this:

/verified by unit tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 6241c74 and 2 for PR HEAD 8a5e11a in total

[openshift-ci]

@jan--f: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@jan--f: Jira Issue Verification Checks: Jira Issue OCPBUGS-100376
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-100376 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

…ers (prometheus#3573)

The statistics counters of a disabled FC host cannot be read from sysfs and any read attempt fails with errno ENOENT. Therefore, all statistics counters returned by procfs are nil in such a case. This results in a crash on s390x Linux when a zFCP host is disabled with chzdev -d <host id>.

Crash stacktrace in a GDB

$ gdb /root/node_exporter/node_exporter
...
Thread 4 "node_exporter" received signal SIGSEGV, Segmentation fault. [Switching to LWP 35579]
0x00000000007be86c in github.com/prometheus/node_exporter/collector.(*fibrechannelCollector).Update (c=0xc00012fef0, ch=0xc0002fc8c0, ~r0=...)
at /root/node_exporter/collector/fibrechannel_linux.go:133
133 c.pushCounter(ch, "dumped_frames_total", *host.Counters.DumpedFrames, *host.Name)
(gdb) bt
at /root/node_exporter/collector/fibrechannel_linux.go:133
(gdb) p host
$1 = {Name = 0xc00038faf0, Speed = 0xc00038fb00, PortState = 0xc00038fb10, PortType = 0xc00038fb30, SymbolicName = 0xc00038fba0, NodeName = 0xc00038fb40, PortID = 0xc00038fb50,
PortName = 0xc00038fb60, FabricName = 0xc00038fb80, DevLossTMO = 0xc00038fb90, SupportedClasses = 0xc00038fbb0, SupportedSpeeds = 0xc00038fbd0, Counters = 0xc0001cb2d0}
(gdb) p host.Name
$2 = (string *) 0xc00038faf0
(gdb) p *host.Name
$3 = "host0"
(gdb) p *host.Counters.DumpedFrames
❌️ Cannot access memory at address 0x0
(gdb) p *host.Counters
$4 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0, RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0, InvalidTXWordCount = 0x0,
LinkFailureCount = 0x0, LossOfSyncCount = 0x0, LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb) bt
at /root/node_exporter/collector/fibrechannel_linux.go:133
name="fibrechannel", c=..., ch=0xc0002fc8c0, logger=0xc0002104a0)
at /root/node_exporter/collector/collector.go:160
(gdb) p *host.Counters
$5 = {DumpedFrames = 0x0, ErrorFrames = 0x0, InvalidCRCCount = 0x0, RXFrames = 0x0,
RXWords = 0x0, TXFrames = 0x0, TXWords = 0x0, SecondsSinceLastReset = 0x0,
InvalidTXWordCount = 0x0, LinkFailureCount = 0x0, LossOfSyncCount = 0x0,
LossOfSignalCount = 0x0, NosCount = 0x0, FCPPacketAborts = 0x0}
(gdb)

(cherry picked from commit 366f908)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.