RFE: Alert on network SoftIRQ backlog or budget drops

[jbainbri]

The file /proc/net/softnet_stat provides statistics on each CPU core's SoftIRQ network receive work. Here's two cores:

$ cat /proc/net/softnet_stat 
00023aee 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
028ad20a 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

If the second column of this contains greater than zero, then net.core.netdev_max_backlog needs to be increased.

If the third column of this contains greater than zero, then net.core.netdev_budget needs to be increased.

Could xsos alert to the fact these values have incremented, and provide the current tunable value?

I'd prefer this up around the ETHTOOL section, as that's where people are looking for drops. Mockup example:

ETHTOOL
  Interface Status:
    eth0  link=up 6000Mb/s full (autoneg=Y)  rx ring 982/1024   drv be2net v4.0.160r / fw 4.9.311.20
  Interface Errors:
    eth0  rx_errors: 679
          rx_tcp_checksum_errs: 679

SOFTIRQ
  Backlog max has been reached, increase net.core.netdev_max_backlog (current value: 1000)
  Budget is not sufficient, increase net.core.netdev_budget (current value: 300)

We don't collect /proc/net/softnet_stat in all versions of sosreport, so xsos would need to check for the presence of the file first.

If the column contains zero, there is no need to report, or an ok status would be fine:

SOFTIRQ
  Backlog max is sufficient (current value: net.core.netdev_max_backlog = 1000)
  Budget is sufficient (current value: net.core.netdev_budget = 300)

[ryran]

For the record Jamie: I did no independent research on this; I simply added exactly what you asked for. I can't think of anyone else I would trust like that...

PS: I added net.core.netdev_max_backlog and net.core.netdev_budget to sysctl output as well.

PPS: I did put it right below ethtool.

[ryran]

For the record Jamie: I did no independent research on this; I simply added exactly what you asked for. I can't think of anyone else I would trust like that...

PS: I added net.core.netdev_max_backlog and net.core.netdev_budget to sysctl output as well.

PPS: I did put it right below ethtool.

[ryran]

@jbainbri : As always, speak up if you have any feedback.

[ryran]

@jbainbri : As always, speak up if you have any feedback.

[ryran]

PS: I just noticed that I didn't add net.core.netdev_max_backlog and net.core.netdev_budget to sysctl .... today. They were already in there, thanks to your RFE from Nov 2013. haha.

[ryran]

PS: I just noticed that I didn't add net.core.netdev_max_backlog and net.core.netdev_budget to sysctl .... today. They were already in there, thanks to your RFE from Nov 2013. haha.

[ryran]

@jbainbri do we have a KCS for this? sroza was talking about it and it made me realize that if we do, I'd like to add it (like how I link the tainted kcs)

EDIT: Ah. https://access.redhat.com/solutions/1241943

[ryran]

@jbainbri do we have a KCS for this? sroza was talking about it and it made me realize that if we do, I'd like to add it (like how I link the tainted kcs)

EDIT: Ah. https://access.redhat.com/solutions/1241943

[mwtzzz]

This should only alert if the counter in the second and third columns of softnet_stat are increasing. Simply being greater than 0 is not sufficient unless the counter has been reset since the last check.

[mwtzzz]

This should only alert if the counter in the second and third columns of softnet_stat are increasing. Simply being greater than 0 is not sufficient unless the counter has been reset since the last check.

[jbainbri]

Except sosreport provides a single snapshot in time, not a repeated collection over time, so that is outside the scope of sosreport and hence xsos as well.

Even an increasing count doesn't matter to most people, as long as they're not also suffering ring buffer drops at the same time. In most cases it's fine for the SoftIRQ to reschedule with more work to complete, as long as all the work is completed before ring buffer exhaustion.

[jbainbri]

Except sosreport provides a single snapshot in time, not a repeated collection over time, so that is outside the scope of sosreport and hence xsos as well.

Even an increasing count doesn't matter to most people, as long as they're not also suffering ring buffer drops at the same time. In most cases it's fine for the SoftIRQ to reschedule with more work to complete, as long as all the work is completed before ring buffer exhaustion.

[mwtzzz]

Both your points make sense.

[mwtzzz]

Both your points make sense.