Malformed increse/rate results on data coliding with compaction

[FUSAKLA]

Bug Report

What did you do?
Nothing

What did you expect to see?
Normal increase similar to other instances of application

What did you see instead? Under which circumstances?
I have one instance of Prometheus(1.8.0) scraping data (scrape interval 1m). This instance is federated by two other Prometheis (one 1.8.0[scrape interval 1m] second 2.2.1[scrape interval 20s]).

Out of nothing strange spikes in data appeared in graphs using increase/rate but it happens only for one scrape target. It showed up during weekend and there were no changes during that time. This happens only on the 2.2.1 Prometheus instance.

Those spikes are caused by the rate breaks in monotonicity adjustment because delta shows some weird data changes which are not present in pure data (see picture below). I don't understand where does those changes in delta result coming from? Those data points are more often than 20s which is scrape interval of the queried Prometheus instance.

Also those weird fuzzy data occurrences does correlate with the compaction interval strongly (also the retention cutoff, WAT truncate and so on..) as shown on picture below

This happens only to the one instance of given application and happens to multiple of metrics exposed by this instance. I suspected colliding labels caused by some mis configured relabel config but I found none of those.

The last thing is that the other Prometheus instance federating those data (1.8.0) is ok.

Environment

• System information: Official docker image running in Kubernetes

• Prometheus version: 2.2.1

• Logs:
  There are just logs about running successful compactions no error.

level=info ts=2018-04-16T19:00:00.703999401Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1523894400000 maxt=1523901600000
level=info ts=2018-04-16T19:00:19.733565554Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.189600114s
level=info ts=2018-04-16T19:00:38.447929878Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=18.714279886s
level=info ts=2018-04-16T21:00:00.703968058Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1523901600000 maxt=1523908800000
level=info ts=2018-04-16T21:00:20.371436364Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.103209377s
level=info ts=2018-04-16T21:00:47.461620519Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=27.090109852s
level=info ts=2018-04-16T21:00:48.302710269Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=1523880000000 maxt=1523901600000
level=info ts=2018-04-16T21:01:07.745462365Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=1523836800000 maxt=1523901600000
level=info ts=2018-04-16T23:00:00.703921618Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1523908800000 maxt=1523916000000

[brian-brazil]

Delta is for gauges, not counters. You want increase.

[FUSAKLA]

Ok thank's for the quick reply I know that I'm supposed to use increase not delta

I'm using the increase/rate the delta was used just for debugging. If you look at the first and third figures in first image those are raw counter and increase.

The increase was about 3000(img1 figure1) and result of increase function is 20M(img1 figure3)?

I used the delta just to illustrate what caused the raise in the increase.

[brian-brazil]

Pulling per-instance data via federation is not a good idea. If you believe you have found a bug, you should show it on the original scraping Prometheus and share the raw data for a single time series with the problem.

[FUSAKLA]

Yes, point taken. I know it's not suggested due to the staleness and performance issues as you state. I'll try to reproduce this on the Prometheus instance directly scraping the data.

But fyi the metric has not been reset for whole week. This should sort out problems with the staleness imho.

[qsbao]

@brian-brazil We got the same problem, I can show you the raw data.

query=node_network_transmit_bytes%7Bdevice%3D%22em2%22%2Cinstance%3D%22172.16.61.174%3A9100%22%7D&start=1525213860&end=1525214760&step=3&_=1525227357527

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"node_network_transmit_bytes","device":"em2","instance":"172.16.61.174:9100","job":"node"},"values":[[1525213860,"45429138616057"],[1525213863,"45429138616057"],[1525213866,"45429138616057"],[1525213869,"45429138616057"],[1525213872,"45429138616057"],[1525213875,"45429138616057"],[1525213878,"45429138616057"],[1525213881,"45429581626372"],[1525213884,"45429581626372"],[1525213887,"45429581626372"],[1525213890,"45429581626372"],[1525213893,"45429581626372"],[1525213896,"45429581626372"],[1525213899,"45429581626372"],[1525213902,"45430172730407"],[1525213905,"45430172730407"],[1525213908,"45430172730407"],[1525213911,"45430172730407"],[1525213914,"45430172730407"],[1525213917,"45430172730407"],[1525213920,"45430748999244"],[1525213923,"45430748999244"],[1525213926,"45430748999244"],[1525213929,"45430748999244"],[1525213932,"45430748999244"],[1525213935,"45430748999244"],[1525213938,"45430748999244"],[1525213941,"45431140256463"],[1525213944,"45431140256463"],[1525213947,"45431140256463"],[1525213950,"45431140256463"],[1525213953,"45431140256463"],[1525213956,"45431140256463"],[1525213959,"45431140256463"],[1525213962,"45431570709668"],[1525213965,"45431570709668"],[1525213968,"45431570709668"],[1525213971,"45431570709668"],[1525213974,"45431570709668"],[1525213977,"45431570709668"],[1525213980,"45431848124766"],[1525213983,"45431848124766"],[1525213986,"45431848124766"],[1525213989,"45431848124766"],[1525213992,"45431848124766"],[1525213995,"45431848124766"],[1525213998,"45431848124766"],[1525214001,"45431900491433"],[1525214004,"45431900491433"],[1525214007,"45431900491433"],[1525214010,"45431900491433"],[1525214013,"45431900491433"],[1525214016,"45431900491433"],[1525214019,"45431900491433"],[1525214022,"45431972689047"],[1525214025,"45431972689047"],[1525214028,"45431972689047"],[1525214031,"45431972689047"],[1525214034,"45431972689047"],[1525214037,"45431972689047"],[1525214040,"45432047215483"],[1525214043,"45432047215483"],[1525214046,"45432047215483"],[1525214049,"45432047215483"],[1525214052,"45432047215483"],[1525214055,"45432047215483"],[1525214058,"45432047215483"],[1525214061,"45432220028494"],[1525214064,"45432220028494"],[1525214067,"45432220028494"],[1525214070,"45432220028494"],[1525214073,"45432220028494"],[1525214076,"45432220028494"],[1525214079,"45432220028494"],[1525214082,"45432329896384"],[1525214085,"45432329896384"],[1525214088,"45432329896384"],[1525214091,"45432329896384"],[1525214094,"45432329896384"],[1525214097,"45432329896384"],[1525214100,"45432470250187"],[1525214103,"45432470250187"],[1525214106,"45432470250187"],[1525214109,"45432470250187"],[1525214112,"45432470250187"],[1525214115,"45432470250187"],[1525214118,"45432470250187"],[1525214121,"45432551821965"],[1525214124,"45432551821965"],[1525214127,"45432551821965"],[1525214130,"45432551821965"],[1525214133,"45432551821965"],[1525214136,"45432551821965"],[1525214139,"45432551821965"],[1525214142,"45432605549106"],[1525214145,"45432605549106"],[1525214148,"45432605549106"],[1525214151,"45432605549106"],[1525214154,"45432605549106"],[1525214157,"45432605549106"],[1525214160,"45432740790638"],[1525214163,"45432740790638"],[1525214166,"45432740790638"],[1525214169,"45432740790638"],[1525214172,"45432740790638"],[1525214175,"45432740790638"],[1525214178,"45432740790638"],[1525214181,"45432808388077"],[1525214184,"45432808388077"],[1525214187,"45432808388077"],[1525214190,"45432808388077"],[1525214193,"45432808388077"],[1525214196,"45432808388077"],[1525214199,"45432808388077"],[1525214202,"45432908865709"],[1525214205,"45432908865709"],[1525214208,"45432908865709"],[1525214211,"45432908865709"],[1525214214,"45432908865709"],[1525214217,"45432908865709"],[1525214220,"45433067631321"],[1525214223,"45433067631321"],[1525214226,"45433067631321"],[1525214229,"45433067631321"],[1525214232,"45433067631321"],[1525214235,"45433067631321"],[1525214238,"45433067631321"],[1525214241,"45433124876389"],[1525214244,"45433124876389"],[1525214247,"45433124876389"],[1525214250,"45433124876389"],[1525214253,"45433124876389"],[1525214256,"45433124876389"],[1525214259,"45433124876389"],[1525214262,"45433178817239"],[1525214265,"45433178817239"],[1525214268,"45433178817239"],[1525214271,"45433178817239"],[1525214274,"45433178817239"],[1525214277,"45433178817239"],[1525214280,"45433311594876"],[1525214283,"45433311594876"],[1525214286,"45433311594876"],[1525214289,"45433311594876"],[1525214292,"45433311594876"],[1525214295,"45433311594876"],[1525214298,"45433311594876"],[1525214301,"45433498290870"],[1525214304,"45433498290870"],[1525214307,"45433498290870"],[1525214310,"45433498290870"],[1525214313,"45433498290870"],[1525214316,"45433498290870"],[1525214319,"45433498290870"],[1525214322,"45433981367116"],[1525214325,"45433981367116"],[1525214328,"45433981367116"],[1525214331,"45433981367116"],[1525214334,"45433981367116"],[1525214337,"45433981367116"],[1525214340,"45435138611385"],[1525214343,"45435138611385"],[1525214346,"45435138611385"],[1525214349,"45435138611385"],[1525214352,"45435138611385"],[1525214355,"45435138611385"],[1525214358,"45435138611385"],[1525214361,"45435705083925"],[1525214364,"45435705083925"],[1525214367,"45435705083925"],[1525214370,"45435705083925"],[1525214373,"45435705083925"],[1525214376,"45435705083925"],[1525214379,"45435705083925"],[1525214382,"45436003880795"],[1525214385,"45436003880795"],[1525214388,"45436003880795"],[1525214391,"45436003880795"],[1525214394,"45436003880795"],[1525214397,"45436003880795"],[1525214400,"45436204032914"],[1525214403,"45436204032914"],[1525214406,"45436204032914"],[1525214409,"45436204032914"],[1525214412,"45436204032914"],[1525214415,"45436204032914"],[1525214418,"45436204032914"],[1525214421,"45437048719318"],[1525214424,"45437048719318"],[1525214427,"45437048719318"],[1525214430,"45437048719318"],[1525214433,"45437048719318"],[1525214436,"45437048719318"],[1525214439,"45437048719318"],[1525214442,"45437301471311"],[1525214445,"45437301471311"],[1525214448,"45437301471311"],[1525214451,"45437301471311"],[1525214454,"45437301471311"],[1525214457,"45437301471311"],[1525214460,"45438005607271"],[1525214463,"45438005607271"],[1525214466,"45438005607271"],[1525214469,"45438005607271"],[1525214472,"45438005607271"],[1525214475,"45438005607271"],[1525214478,"45438005607271"],[1525214481,"45438531145856"],[1525214484,"45438531145856"],[1525214487,"45438531145856"],[1525214490,"45438531145856"],[1525214493,"45438531145856"],[1525214496,"45438531145856"],[1525214499,"45438531145856"],[1525214502,"45438903826141"],[1525214505,"45438903826141"],[1525214508,"45438903826141"],[1525214511,"45438903826141"],[1525214514,"45438903826141"],[1525214517,"45438903826141"],[1525214520,"45439440685853"],[1525214523,"45439440685853"],[1525214526,"45439440685853"],[1525214529,"45439440685853"],[1525214532,"45439440685853"],[1525214535,"45439440685853"],[1525214538,"45439440685853"],[1525214541,"45440132999545"],[1525214544,"45440132999545"],[1525214547,"45440132999545"],[1525214550,"45440132999545"],[1525214553,"45440132999545"],[1525214556,"45440132999545"],[1525214559,"45440132999545"],[1525214562,"45440629367870"],[1525214565,"45440629367870"],[1525214568,"45440629367870"],[1525214571,"45440629367870"],[1525214574,"45440629367870"],[1525214577,"45440629367870"],[1525214580,"45440866560289"],[1525214583,"45440866560289"],[1525214586,"45440866560289"],[1525214589,"45440866560289"],[1525214592,"45440866560289"],[1525214595,"45440866560289"],[1525214598,"45440866560289"],[1525214601,"45441478353915"],[1525214604,"45441478353915"],[1525214607,"45441478353915"],[1525214610,"45441478353915"],[1525214613,"45441478353915"],[1525214616,"45441478353915"],[1525214619,"45441478353915"],[1525214622,"45441579022810"],[1525214625,"45441579022810"],[1525214628,"45441579022810"],[1525214631,"45441579022810"],[1525214634,"45441579022810"],[1525214637,"45441579022810"],[1525214640,"45441863628636"],[1525214643,"45441863628636"],[1525214646,"45441863628636"],[1525214649,"45441863628636"],[1525214652,"45441863628636"],[1525214655,"45441863628636"],[1525214658,"45441863628636"],[1525214661,"45442104843845"],[1525214664,"45442104843845"],[1525214667,"45442104843845"],[1525214670,"45442104843845"],[1525214673,"45442104843845"],[1525214676,"45442104843845"],[1525214679,"45442104843845"],[1525214682,"45442417890474"],[1525214685,"45442417890474"],[1525214688,"45442417890474"],[1525214691,"45442417890474"],[1525214694,"45442417890474"],[1525214697,"45442417890474"],[1525214700,"45443096997228"],[1525214703,"45443096997228"],[1525214706,"45443096997228"],[1525214709,"45443096997228"],[1525214712,"45443096997228"],[1525214715,"45443096997228"],[1525214718,"45443096997228"],[1525214721,"45443504374336"],[1525214724,"45443504374336"],[1525214727,"45443504374336"],[1525214730,"45443504374336"],[1525214733,"45443504374336"],[1525214736,"45443504374336"],[1525214739,"45443504374336"],[1525214742,"45444155737145"],[1525214745,"45444155737145"],[1525214748,"45444155737145"],[1525214751,"45444155737145"],[1525214754,"45444155737145"],[1525214757,"45444155737145"],[1525214760,"45444441503220"]]}]}}

query=increase(node_network_transmit_bytes%7Bdevice%3D%22em2%22%2Cinstance%3D%22172.16.61.174%3A9100%22%7D%5B1m%5D)&start=1525213860&end=1525214760&step=3&_=1525227357531

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"device":"em2","instance":"172.16.61.174:9100","job":"node"},"values":[[1525213860,"1376634992"],[1525213863,"1357911004.5"],[1525213866,"1357911004.5"],[1525213869,"1357911004.5"],[1525213872,"1357911004.5"],[1525213875,"1357911004.5"],[1525213878,"1357911004.5"],[1525213881,"1299298399.5"],[1525213884,"1299298399.5"],[1525213887,"1299298399.5"],[1525213890,"1299298399.5"],[1525213893,"1299298399.5"],[1525213896,"1299298399.5"],[1525213899,"1299298399.5"],[1525213902,"1551171525"],[1525213905,"1551171525"],[1525213908,"1551171525"],[1525213911,"1551171525"],[1525213914,"1551171525"],[1525213917,"1551171525"],[1525213920,"1610383187"],[1525213923,"1751059308"],[1525213926,"1751059308"],[1525213929,"1751059308"],[1525213932,"1751059308"],[1525213935,"1751059308"],[1525213938,"1751059308"],[1525213941,"1451289084"],[1525213944,"1451289084"],[1525213947,"1451289084"],[1525213950,"1451289084"],[1525213953,"1451289084"],[1525213956,"1451289084"],[1525213959,"1451289084"],[1525213962,"1232565636"],[1525213965,"1232565636"],[1525213968,"1232565636"],[1525213971,"1232565636"],[1525213974,"1232565636"],[1525213977,"1232565636"],[1525213980,"1099125522"],[1525213983,"1061802454.5"],[1525213986,"1061802454.5"],[1525213989,"1061802454.5"],[1525213992,"1061802454.5"],[1525213995,"1061802454.5"],[1525213998,"1061802454.5"],[1525214001,"494672647.5"],[1525214004,"494672647.5"],[1525214007,"494672647.5"],[1525214010,"494672647.5"],[1525214013,"494672647.5"],[1525214016,"494672647.5"],[1525214019,"494672647.5"],[1525214022,"186846421.5"],[1525214025,"186846421.5"],[1525214028,"186846421.5"],[1525214031,"186846421.5"],[1525214034,"186846421.5"],[1525214037,"186846421.5"],[1525214040,"199090717"],[1525214043,"220086075"],[1525214046,"220086075"],[1525214049,"220086075"],[1525214052,"220086075"],[1525214055,"220086075"],[1525214058,"220086075"],[1525214061,"371009170.5"],[1525214064,"371009170.5"],[1525214067,"371009170.5"],[1525214070,"371009170.5"],[1525214073,"371009170.5"],[1525214076,"371009170.5"],[1525214079,"371009170.5"],[1525214082,"424021351.5"],[1525214085,"424021351.5"],[1525214088,"424021351.5"],[1525214091,"424021351.5"],[1525214094,"424021351.5"],[1525214097,"424021351.5"],[1525214100,"423034704"],[1525214103,"375332539.5"],[1525214106,"375332539.5"],[1525214109,"375332539.5"],[1525214112,"375332539.5"],[1525214115,"375332539.5"],[1525214118,"375332539.5"],[1525214121,"332888371.5"],[1525214124,"332888371.5"],[1525214127,"332888371.5"],[1525214130,"332888371.5"],[1525214133,"332888371.5"],[1525214136,"332888371.5"],[1525214139,"332888371.5"],[1525214142,"202948378.5"],[1525214145,"202948378.5"],[1525214148,"202948378.5"],[1525214151,"202948378.5"],[1525214154,"202948378.5"],[1525214157,"202948378.5"],[1525214160,"270540451"],[1525214163,"283453009.5"],[1525214166,"283453009.5"],[1525214169,"283453009.5"],[1525214172,"283453009.5"],[1525214175,"283453009.5"],[1525214178,"283453009.5"],[1525214181,"304258456.5"],[1525214184,"304258456.5"],[1525214187,"304258456.5"],[1525214190,"304258456.5"],[1525214193,"304258456.5"],[1525214196,"304258456.5"],[1525214199,"304258456.5"],[1525214202,"252112606.5"],[1525214205,"252112606.5"],[1525214208,"252112606.5"],[1525214211,"252112606.5"],[1525214214,"252112606.5"],[1525214217,"252112606.5"],[1525214220,"326840683"],[1525214223,"388864866"],[1525214226,"388864866"],[1525214229,"388864866"],[1525214232,"388864866"],[1525214235,"388864866"],[1525214238,"388864866"],[1525214241,"324016020"],[1525214244,"324016020"],[1525214247,"324016020"],[1525214250,"324016020"],[1525214253,"324016020"],[1525214256,"324016020"],[1525214259,"324016020"],[1525214262,"166778877"],[1525214265,"166778877"],[1525214268,"166778877"],[1525214271,"166778877"],[1525214274,"166778877"],[1525214277,"166778877"],[1525214280,"243963555"],[1525214283,"280077730.5"],[1525214286,"280077730.5"],[1525214289,"280077730.5"],[1525214292,"280077730.5"],[1525214295,"280077730.5"],[1525214298,"280077730.5"],[1525214301,"479210446.5"],[1525214304,"479210446.5"],[1525214307,"479210446.5"],[1525214310,"479210446.5"],[1525214313,"479210446.5"],[1525214316,"479210446.5"],[1525214319,"479210446.5"],[1525214322,"1004658360"],[1525214325,"1004658360"],[1525214328,"1004658360"],[1525214331,"1004658360"],[1525214334,"1004658360"],[1525214337,"1004658360"],[1525214340,"1827016509"],[1525214343,"2460480772.5"],[1525214346,"2460480772.5"],[1525214349,"2460480772.5"],[1525214352,"2460480772.5"],[1525214355,"2460480772.5"],[1525214358,"2460480772.5"],[1525214361,"2585575213.5"],[1525214364,"2585575213.5"],[1525214367,"2585575213.5"],[1525214370,"2585575213.5"],[1525214373,"2585575213.5"],[1525214376,"2585575213.5"],[1525214379,"2585575213.5"],[1525214382,"52252399522735.74"],[1525214385,"55660164709001.13"],[1525214388,"59067929895266.5"],[1525214391,"54524242980246"],[1525214394,"56796086437756.25"],[1525214397,"53388321251490.875"],[1525214400,"45437069302324"],[1525214403,"54523803395740.8"],[1525214406,"57931541107974.59"],[1525214409,"61339278820208.41"],[1525214412,"60203366249463.8"],[1525214415,"56795628537230"],[1525214418,"53387890824996.2"],[1525214421,"1245099239.9083333"],[1525214424,"1323462129.1333332"],[1525214427,"1567257784.5"],[1525214430,"1567257784.5"],[1525214433,"1567257784.5"],[1525214436,"1323462129.1333332"],[1525214439,"1245099239.9083333"],[1525214442,"1646157595.5"],[1525214445,"1646157595.5"],[1525214448,"1646157595.5"],[1525214451,"1646157595.5"],[1525214454,"1646157595.5"],[1525214457,"1646157595.5"],[1525214460,"1801574357"],[1525214463,"1435331929.5"],[1525214466,"1435331929.5"],[1525214469,"1435331929.5"],[1525214472,"1435331929.5"],[1525214475,"1435331929.5"],[1525214478,"1435331929.5"],[1525214481,"1844511817.5"],[1525214484,"1844511817.5"],[1525214487,"1844511817.5"],[1525214490,"1844511817.5"],[1525214493,"1844511817.5"],[1525214496,"1844511817.5"],[1525214499,"1844511817.5"],[1525214502,"1347328305"],[1525214505,"1347328305"],[1525214508,"1347328305"],[1525214511,"1347328305"],[1525214514,"1347328305"],[1525214517,"1347328305"],[1525214520,"1435078582"],[1525214523,"1364309995.5"],[1525214526,"1364309995.5"],[1525214529,"1364309995.5"],[1525214532,"1364309995.5"],[1525214535,"1364309995.5"],[1525214538,"1364309995.5"],[1525214541,"1843760106"],[1525214544,"1843760106"],[1525214547,"1843760106"],[1525214550,"1843760106"],[1525214553,"1843760106"],[1525214556,"1843760106"],[1525214559,"1843760106"],[1525214562,"1783023025.5"],[1525214565,"1783023025.5"],[1525214568,"1783023025.5"],[1525214571,"1783023025.5"],[1525214574,"1783023025.5"],[1525214577,"1783023025.5"],[1525214580,"1425874436"],[1525214583,"1100341116"],[1525214586,"1100341116"],[1525214589,"1100341116"],[1525214592,"1100341116"],[1525214595,"1100341116"],[1525214598,"1100341116"],[1525214601,"1273479067.5"],[1525214604,"1273479067.5"],[1525214607,"1273479067.5"],[1525214610,"1273479067.5"],[1525214613,"1273479067.5"],[1525214616,"1273479067.5"],[1525214619,"1273479067.5"],[1525214622,"1068693781.5"],[1525214625,"1068693781.5"],[1525214628,"1068693781.5"],[1525214631,"1068693781.5"],[1525214634,"1068693781.5"],[1525214637,"1068693781.5"],[1525214640,"997068347"],[1525214643,"577912081.5"],[1525214646,"577912081.5"],[1525214649,"577912081.5"],[1525214652,"577912081.5"],[1525214655,"577912081.5"],[1525214658,"577912081.5"],[1525214661,"788731552.5"],[1525214664,"788731552.5"],[1525214667,"788731552.5"],[1525214670,"788731552.5"],[1525214673,"788731552.5"],[1525214676,"788731552.5"],[1525214679,"788731552.5"],[1525214682,"831392757"],[1525214685,"831392757"],[1525214688,"831392757"],[1525214691,"831392757"],[1525214694,"831392757"],[1525214697,"831392757"],[1525214700,"1233368592"],[1525214703,"1488230074.5"],[1525214706,"1488230074.5"],[1525214709,"1488230074.5"],[1525214712,"1488230074.5"],[1525214715,"1488230074.5"],[1525214718,"1488230074.5"],[1525214721,"1629725793"],[1525214724,"1629725793"],[1525214727,"1629725793"],[1525214730,"1629725793"],[1525214733,"1629725793"],[1525214736,"1629725793"],[1525214739,"1629725793"],[1525214742,"1588109875.5"],[1525214745,"1588109875.5"],[1525214748,"1588109875.5"],[1525214751,"1588109875.5"],[1525214754,"1588109875.5"],[1525214757,"1588109875.5"],[1525214760,"1344505992"]]}]}}

Environment

• System information:

  Linux 2.6.32-642.el6.x86_64 x86_64

• Prometheus version:

  prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c)
  build user: root@149e5b3f0829
  build date: 20180314-14:15:45
  go version: go1.10

Node's scrape interval is 20s.

[brian-brazil]

That's not the raw data, and that looks like a counter going backwards. Check with resets()

[qsbao]

Yes, you are right. There is a counter resets. Thank you!

[qsbao]

Hi Brian, resets() shows there are counter resets happens. But:

1. Why can not increase() hold this situation? Doc of increase() said:

   Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for.
   

2. Why the counter resets occurred periodically (every 2 hours)?
   We have 100+ node_exporter targets, and only this one has the problem. (We noticed that default config storage.tsdb.min-block-duration is 2 hours too.)

query=resets(node_network_transmit_bytes%7Bdevice%3D%22em2%22%2Cinstance%3D%22172.16.61.174%3A9100%22%7D%5B5m%5D)&start=1525171560&end=1525214760&step=172&_=1525227357566

{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"device":"em2","instance":"172.16.61.174:9100","job":"node"},"values":[[1525171560,"0"],[1525171732,"0"],[1525171904,"0"],[1525172076,"0"],[1525172248,"0"],[1525172420,"0"],[1525172592,"0"],[1525172764,"0"],[1525172936,"0"],[1525173108,"0"],[1525173280,"0"],[1525173452,"0"],[1525173624,"0"],[1525173796,"0"],[1525173968,"0"],[1525174140,"0"],[1525174312,"0"],[1525174484,"0"],[1525174656,"0"],[1525174828,"0"],[1525175000,"0"],[1525175172,"0"],[1525175344,"0"],[1525175516,"0"],[1525175688,"0"],[1525175860,"0"],[1525176032,"0"],[1525176204,"0"],[1525176376,"0"],[1525176548,"0"],[1525176720,"0"],[1525176892,"0"],[1525177064,"0"],[1525177236,"0"],[1525177408,"0"],[1525177580,"0"],[1525177752,"0"],[1525177924,"0"],[1525178096,"0"],[1525178268,"0"],[1525178440,"1"],[1525178612,"1"],[1525178784,"0"],[1525178956,"0"],[1525179128,"0"],[1525179300,"0"],[1525179472,"0"],[1525179644,"0"],[1525179816,"0"],[1525179988,"0"],[1525180160,"0"],[1525180332,"0"],[1525180504,"0"],[1525180676,"0"],[1525180848,"0"],[1525181020,"0"],[1525181192,"0"],[1525181364,"0"],[1525181536,"0"],[1525181708,"0"],[1525181880,"0"],[1525182052,"0"],[1525182224,"0"],[1525182396,"0"],[1525182568,"0"],[1525182740,"0"],[1525182912,"0"],[1525183084,"0"],[1525183256,"0"],[1525183428,"0"],[1525183600,"0"],[1525183772,"0"],[1525183944,"0"],[1525184116,"0"],[1525184288,"0"],[1525184460,"0"],[1525184632,"0"],[1525184804,"0"],[1525184976,"0"],[1525185148,"0"],[1525185320,"0"],[1525185492,"0"],[1525185664,"1"],[1525185836,"1"],[1525186008,"0"],[1525186180,"0"],[1525186352,"0"],[1525186524,"0"],[1525186696,"0"],[1525186868,"0"],[1525187040,"0"],[1525187212,"0"],[1525187384,"0"],[1525187556,"0"],[1525187728,"0"],[1525187900,"0"],[1525188072,"0"],[1525188244,"0"],[1525188416,"0"],[1525188588,"0"],[1525188760,"0"],[1525188932,"0"],[1525189104,"0"],[1525189276,"0"],[1525189448,"0"],[1525189620,"0"],[1525189792,"0"],[1525189964,"0"],[1525190136,"0"],[1525190308,"0"],[1525190480,"0"],[1525190652,"0"],[1525190824,"0"],[1525190996,"0"],[1525191168,"0"],[1525191340,"0"],[1525191512,"0"],[1525191684,"0"],[1525191856,"0"],[1525192028,"0"],[1525192200,"0"],[1525192372,"0"],[1525192544,"0"],[1525192716,"0"],[1525192888,"1"],[1525193060,"1"],[1525193232,"0"],[1525193404,"0"],[1525193576,"0"],[1525193748,"0"],[1525193920,"0"],[1525194092,"0"],[1525194264,"0"],[1525194436,"0"],[1525194608,"0"],[1525194780,"0"],[1525194952,"0"],[1525195124,"0"],[1525195296,"0"],[1525195468,"0"],[1525195640,"0"],[1525195812,"0"],[1525195984,"0"],[1525196156,"0"],[1525196328,"0"],[1525196500,"0"],[1525196672,"0"],[1525196844,"0"],[1525197016,"0"],[1525197188,"0"],[1525197360,"0"],[1525197532,"0"],[1525197704,"0"],[1525197876,"0"],[1525198048,"0"],[1525198220,"0"],[1525198392,"0"],[1525198564,"0"],[1525198736,"0"],[1525198908,"0"],[1525199080,"0"],[1525199252,"0"],[1525199424,"0"],[1525199596,"0"],[1525199768,"0"],[1525199940,"0"],[1525200112,"1"],[1525200284,"0"],[1525200456,"0"],[1525200628,"0"],[1525200800,"0"],[1525200972,"0"],[1525201144,"0"],[1525201316,"0"],[1525201488,"0"],[1525201660,"0"],[1525201832,"0"],[1525202004,"0"],[1525202176,"0"],[1525202348,"0"],[1525202520,"0"],[1525202692,"0"],[1525202864,"0"],[1525203036,"0"],[1525203208,"0"],[1525203380,"0"],[1525203552,"0"],[1525203724,"0"],[1525203896,"0"],[1525204068,"0"],[1525204240,"0"],[1525204412,"0"],[1525204584,"0"],[1525204756,"0"],[1525204928,"0"],[1525205100,"0"],[1525205272,"0"],[1525205444,"0"],[1525205616,"0"],[1525205788,"0"],[1525205960,"0"],[1525206132,"0"],[1525206304,"0"],[1525206476,"0"],[1525206648,"0"],[1525206820,"0"],[1525206992,"0"],[1525207164,"0"],[1525207336,"1"],[1525207508,"0"],[1525207680,"0"],[1525207852,"0"],[1525208024,"0"],[1525208196,"0"],[1525208368,"0"],[1525208540,"0"],[1525208712,"0"],[1525208884,"0"],[1525209056,"0"],[1525209228,"0"],[1525209400,"0"],[1525209572,"0"],[1525209744,"0"],[1525209916,"0"],[1525210088,"0"],[1525210260,"0"],[1525210432,"0"],[1525210604,"0"],[1525210776,"0"],[1525210948,"0"],[1525211120,"0"],[1525211292,"0"],[1525211464,"0"],[1525211636,"0"],[1525211808,"0"],[1525211980,"0"],[1525212152,"0"],[1525212324,"0"],[1525212496,"0"],[1525212668,"0"],[1525212840,"0"],[1525213012,"0"],[1525213184,"0"],[1525213356,"0"],[1525213528,"0"],[1525213700,"0"],[1525213872,"0"],[1525214044,"0"],[1525214216,"0"],[1525214388,"1"],[1525214560,"1"],[1525214732,"0"]]}]}}

[brian-brazil]

A break in monotonicity is presumed to be a reset back to zero due to a process restarting, which if that's not the case you get what you're seeing. There's something broken with your counter that you need to fix.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.