failed to parse /proc/net/softnet_stat

[FlorinAndrei]

Host operating system: output of uname -a

Linux cscale-82-156.XXXXXXXXX.com 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.0.0-rc.0 (branch: HEAD, revision: ef7c058)
build user: root@b38a2df1a38b
build date: 20200220-12:54:05
go version: go1.13.8

node_exporter command line flags

./node_exporter
--collector.cpu.info
--collector.buddyinfo
--collector.interrupts
--collector.ksmd
--collector.logind
--collector.mountstats
--collector.processes
--collector.qdisc
--collector.schedstat
--collector.systemd
--collector.tcpstat

Are you running node_exporter in Docker?

no

What did you do that produced an error?

launched node_exporter

What did you expect to see?

no errors

What did you see instead?

level=error ts=2020-02-27T22:48:20.264Z caller=collector.go:161 msg="collector failed" name=softnet duration_seconds=0.001846591 err="could not get softnet statistics: failed to parse /proc/net/softnet_stat: 10 columns were detected, but 11 were expected"

This is the contents of the softnet_stat file:

00029620 00000000 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0012f6e7 00000000 0000000e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0028b596 00000000 00000012 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0028d1b4 00000000 00000009 00000000 00000000 00000000 00000000 00000000 00000000 00000000
002f9d70 00000000 0000000d 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0003baa1 00000000 00000003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0008b9c2 00000000 00000004 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00034422 00000000 00000006 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00044ae8 00000000 00000003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00027c94 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002b962 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0003140b 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000306fd 00000000 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00027096 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002f6e0 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002f073 00000000 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000277ba 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002e199 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002d252 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002d4f1 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0304743c 00000000 00000465 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0049bc3a 00000000 0000000e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
001f9954 00000000 00000006 00000000 00000000 00000000 00000000 00000000 00000000 00000000
002ea54d 00000000 00000009 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002c5b6 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002a126 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002c899 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000288f8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000290c8 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00070452 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00027548 00000000 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002603d 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00027321 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00026bbf 00000000 00000003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00027018 00000000 00000002 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000251d5 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00025877 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0002b9ef 00000000 00000004 00000000 00000000 00000000 00000000 00000000 00000000 00000000
000266e5 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00024e27 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

[SuperQ]

Thanks for the bug report, this needs to be fixed in the procfs library. It looks like older kernels have different fields.

[francistan89]

Is there a fix on this issue?

Getting the same issue when running node exporter on Centos 6..

"level=error ts=2020-03-15T19:33:12.156Z caller=collector.go:161 msg="collector failed" name=softnet duration_seconds=0.000105656 err="could not get softnet statistics: failed to parse /proc/net/softnet_stat: 10 columns were detected, but 11 were expected"

Thanks.

[SuperQ]

The fix is upstream, but I'm waiting on one more other fix to be merged and I can cut a release.

As a workaround, you can disable softnet with --no-collector.softnet.

[francistan89]

Thank you. I will try that workaround. Hopefully will get a proper fix soon.

[roy-spark]

The fix is upstream, but I'm waiting on one more other fix to be merged and I can cut a release.

As a workaround, you can disable softnet with --no-collector.softnet.

This workaround doesn't affect my logs.

grafana-main-deployment-node-exporter-1   | ts=2023-06-14T18:52:18.792Z caller=collector.go:169 level=error msg="collector failed" name=arp duration_seconds=9.098e-06 err="could not get ARP entries: error reading arp \"/host/proc/net/arp\": open /host/proc/net/arp: no such file or directory"```

[SuperQ]

@roy-spark That's because that is the arp collector, not softnet.