cpu collector/stat collector,couldn't parse -2049562273 (processes)

[queenns]

Host operating system: output of uname -a

Linux centos 4.14.105-19-0016 #1 SMP Wed Sep 23 19:31:42 CST 2020 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

prom/node-exporter:v0.17.0

node_exporter command line flags

Are you running node_exporter in Docker?

yes,kubernetes

What did you do that produced an error?

What did you expect to see?

What did you see instead?

time="2020-11-09T07:01:26Z" level=error msg="ERROR: cpu collector failed after 0.002500s: couldn't parse -2049041839 (processes): strconv.ParseUint: parsing "-2049041839": invalid syntax" source="collector.go:132"
time="2020-11-09T07:01:26Z" level=error msg="ERROR: stat collector failed after 0.002542s: couldn't parse -2049041839 (processes): strconv.ParseUint: parsing "-2049041839": invalid syntax" source="collector.go:132"

ps -ef
root 4076051 2 0 13:49 ? 00:00:00 [kworker/2:1]
root 4123371 2 0 13:50 ? 00:00:00 [kworker/45:2]
root 4165156 2 0 12:23 ? 00:00:00 [kworker/10:1]

[SuperQ]

What is the contents of /proc/stat?

[queenns]

What is the contents of /proc/stat?

~]# cat /proc/stat
cpu 849282590 7356 398488121 14888068462 187327979 0 136302036 0 0 0
cpu0 17271478 78 9092206 208446164 13199790 0 315335 0 0 0
cpu1 11845656 264 9070064 231229159 262669 0 609782 0 0 0
cpu2 16794734 94 9026998 227372326 1936549 0 1085223 0 0 0
cpu3 11801270 263 9132348 234380261 212849 0 1409866 0 0 0
cpu4 17337333 91 9231364 226505277 1398683 0 2293188 0 0 0
cpu5 13923489 222 8519781 232525607 236401 0 2877218 0 0 0
cpu6 18142770 81 9023909 226398410 1051365 0 2063205 0 0 0
cpu7 13816220 242 8485396 232914740 205308 0 2712633 0 0 0
cpu8 15062041 59 6101576 227718262 5344632 0 3995306 0 0 0
cpu9 9707533 147 5477319 239271643 883628 0 3337380 0 0 0
cpu10 14793980 67 6038709 228883685 5364516 0 1993644 0 0 0
cpu11 10017513 161 5487975 239803204 923321 0 1971290 0 0 0
cpu12 14998823 56 6077902 228775336 5358334 0 1766449 0 0 0
cpu13 10259098 158 5533268 239546008 904624 0 1838705 0 0 0
cpu14 15148767 59 6073244 228667199 5344445 0 1740049 0 0 0
cpu15 10306170 172 5548324 239545160 908222 0 1718558 0 0 0
cpu16 15357910 60 6226015 228111239 5224505 0 2001669 0 0 0
cpu17 10576739 147 5637062 239147160 898431 0 1698490 0 0 0
cpu18 15338818 58 6208036 228427535 5207766 0 1739298 0 0 0
cpu19 10485736 148 5671654 239242928 895215 0 1646816 0 0 0
cpu20 15393550 102 6252443 228476574 5211580 0 1657886 0 0 0
cpu21 10454887 154 5757269 239054349 879561 0 1825589 0 0 0
cpu22 15521330 68 6310524 228219102 5195011 0 1741415 0 0 0
cpu23 10451611 125 5773188 238614017 856593 0 2326448 0 0 0
cpu24 15722663 68 6221010 228186451 5126316 0 1701255 0 0 0
cpu25 10476606 153 5422388 239139426 827972 0 1810762 0 0 0
cpu26 15560295 71 6182790 228483517 5123260 0 1642804 0 0 0
cpu27 10357730 171 5632130 239430318 874158 0 1658552 0 0 0
cpu28 15452599 79 6159778 228637782 5139408 0 1651938 0 0 0
cpu29 10675737 167 5526160 239333022 892415 0 1521467 0 0 0
cpu30 15854777 72 6140211 228024340 5184371 0 1896075 0 0 0
cpu31 10574279 177 5549397 239295411 888874 0 1654636 0 0 0
cpu32 15759874 62 6431880 228056052 4962444 0 1711872 0 0 0
cpu33 10636814 150 5911892 238832149 896788 0 1630436 0 0 0
cpu34 16017157 82 6418202 227623581 4975350 0 1932975 0 0 0
cpu35 10746534 166 5853062 238747513 890738 0 1691784 0 0 0
cpu36 15859476 72 6393242 227263973 5196182 0 2448217 0 0 0
cpu37 10944910 157 5900933 238623184 906386 0 1631123 0 0 0
cpu38 15825826 87 6379660 228140250 4887181 0 1780178 0 0 0
cpu39 10875019 140 5813448 238678420 882597 0 1633756 0 0 0
cpu40 16178587 62 5839503 228545993 4671824 0 1755391 0 0 0
cpu41 10144506 130 5443971 239990987 860198 0 1499539 0 0 0
cpu42 16229988 51 5845983 228410440 4723449 0 1822211 0 0 0
cpu43 10288563 136 5418534 239862356 847979 0 1526535 0 0 0
cpu44 15797766 66 5862668 228103786 4993677 0 2414562 0 0 0
cpu45 10334836 145 5391994 239789089 857188 0 1563753 0 0 0
cpu46 15353644 52 5858741 229372889 4726298 0 1701176 0 0 0
cpu47 9945218 139 5281933 240208183 822899 0 1654739 0 0 0
cpu48 15320123 42 5981440 227424426 5165732 0 2764924 0 0 0
cpu49 10008878 123 5403071 238247997 868940 0 3062064 0 0 0
cpu50 15524131 55 5865555 227204406 5130643 0 2928688 0 0 0
cpu51 10171086 134 5380912 238450979 893596 0 2750064 0 0 0
cpu52 15383202 60 5886474 226832466 5149489 0 3365926 0 0 0
cpu53 10160810 131 5389812 237829546 894334 0 3261991 0 0 0
cpu54 15316619 55 5906945 227485460 5193348 0 2822620 0 0 0
cpu55 10245465 139 5437267 238180352 891598 0 2884970 0 0 0
cpu56 15835854 94 6109214 226356364 5348378 0 2971450 0 0 0
cpu57 10761146 138 5561151 237645103 942642 0 2706255 0 0 0
cpu58 15959933 80 6122960 225537574 5340252 0 3611951 0 0 0
cpu59 10817312 140 5643700 236765195 909682 0 3368072 0 0 0
cpu60 15886140 45 6035390 226628660 5323392 0 2794705 0 0 0
cpu61 10715602 156 5590227 237538339 939987 0 2852212 0 0 0
cpu62 15875742 56 6084514 226383938 5345234 0 3020405 0 0 0
cpu63 10879661 156 5453382 237503166 928753 0 2834538 0 0 0
intr 275605269514 39 10 0 0 434 0 0 0 0 0 0 0 904 0 2568819 0 0 0 0 0 0 0 0 0 0 2039191993 1 1842758020 1 1845702734 1 1858568012 1 2771617697 0 1 3022915601 3012688342 1 1892491216 1 1861527543 1 1835474011 2 1874774953 1 2785092442 1 2818163904 1 1982988119 1 1860833321 2 2160992571 1 1817606450 1 0 131293397 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 341304842671
btime 1602320404
processes -2045677862
procs_running 2
procs_blocked 1
softirq 161801794935 1 3123151663 1122534 3868691281 1284486 0 56707 4118874546 3728711634 0 931014019

[queenns]

What is the contents of /proc/stat?

Is this a system setting problem or an application bug?

[SuperQ]

It looks like a kernel bug or corruption. processes -2045677862 is supposed to be a Number of forks since boot0. Having a negative number shouldn't be possible. Unless you have more than one of these, maybe the easiest option is to reboot the node.

[SuperQ]

There's definitely something wrong with this system. Based on fs/proc/stat.c, the output should never be negative.

[queenns]

Thank you very much for your help, I will try to deal with it first

[roidelapluie]

It looks like a kernel bug or corruption. processes -2045677862 is supposed to be a Number of forks since boot0. Having a negative number shouldn't be possible. Unless you have more than one of these, maybe the easiest option is to reboot the node.

isn't it an issue with int / uint?

[queenns]

Colleagues who understand the kernel found that the process statistics method of this version of the kernel caused problems. Remove this node for system upgrade processing methods.

[MrYueQ]

@SuperQ hi. We have also encountered this problem now. How can i solve it

image: prom/node-exporter:v1.1.2

error msg

level=error ts=2021-06-03T04:36:25.876Z caller=collector.go:161 msg="collector failed" name=stat duration_seconds=0.003621644 err="couldn't parse \"-1276565208\" (processes): strconv.ParseUint: parsing \"-1276565208\": invalid syntax"
level=error ts=2021-06-03T04:36:25.876Z caller=collector.go:161 msg="collector failed" name=cpu duration_seconds=0.003461332 err="couldn't parse \"-1276565208\" (processes): strconv.ParseUint: parsing \"-1276565208\": invalid syntax"

[queenns]

The problem is caused by the kernel

1. Restarting the host can temporarily solve the problem, but a negative number will still appear at a certain time
2. Choose another kernel version

[SuperQ]

Yes, it's still unclear how this is even possible, as the kernel code clearly is formatting the data as %lu unsigned long. Without more system information, it's impossible to know why the OS is corrupted.

[MrYueQ]

Thank you very much for your reply. @queenns