Prometheus 2.2.1 - Data Missing

[hmmxp]

What did you do?

Query switch interface utilization (ifHCInOctets/ifHCOutOctets) for past 1 week

What did you expect to see?

Complete data for past 1 week

What did you see instead? Under which circumstances?

Data Loss for certain days as per below image

Environment

• System information:

Centos 7.4 (Linux 3.10.0-514.26.2.el7.x86_64 x86_64)

• Prometheus version:

prometheus, version 2.2.1

• Prometheus configuration file:

insert configuration here

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

level=info ts=2018-03-22T07:00:00.160291411Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521691200000 maxt=1521698400000
level=info ts=2018-03-22T07:00:02.380710983Z caller=head.go:348 component=tsdb msg="head GC completed" duration=102.571012ms
level=info ts=2018-03-22T07:00:03.604042466Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.223173398s
level=info ts=2018-03-22T07:47:43.317488054Z caller=main.go:588 msg="Loading configuration file" filename=/etc/lis/prometheus_v2.yml
level=info ts=2018-03-22T07:53:21.855494907Z caller=main.go:588 msg="Loading configuration file" filename=/etc/lis/prometheus_v2.yml
level=info ts=2018-03-22T07:53:23.846849923Z caller=main.go:588 msg="Loading configuration file" filename=/etc/lis/prometheus_v2.yml
level=info ts=2018-03-22T09:00:00.138738509Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521698400000 maxt=1521705600000
level=info ts=2018-03-22T09:00:02.04154061Z caller=head.go:348 component=tsdb msg="head GC completed" duration=85.785807ms
level=info ts=2018-03-22T09:00:03.181177999Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.139513338s
level=info ts=2018-03-22T09:00:03.40531207Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=1521676800000 maxt=1521698400000
level=info ts=2018-03-22T09:00:05.24590305Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=1521633600000 maxt=1521698400000
level=info ts=2018-03-22T11:00:00.138721024Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521705600000 maxt=1521712800000
level=info ts=2018-03-22T11:00:01.735737965Z caller=head.go:348 component=tsdb msg="head GC completed" duration=68.384189ms
level=info ts=2018-03-22T11:00:02.702610809Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=966.795541ms
level=info ts=2018-03-22T13:00:00.13839843Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521712800000 maxt=1521720000000
level=info ts=2018-03-22T13:00:01.630621522Z caller=head.go:348 component=tsdb msg="head GC completed" duration=65.555315ms
level=info ts=2018-03-22T13:00:02.665056281Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.034333321s
level=info ts=2018-03-22T15:00:00.148312055Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521720000000 maxt=1521727200000
level=info ts=2018-03-22T15:00:01.420679721Z caller=head.go:348 component=tsdb msg="head GC completed" duration=75.100048ms
level=info ts=2018-03-22T15:00:02.356255364Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=935.4871ms
level=info ts=2018-03-22T15:00:02.526419969Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=1521698400000 maxt=1521720000000
level=info ts=2018-03-22T17:00:00.138879637Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521727200000 maxt=1521734400000
level=info ts=2018-03-22T17:00:01.579749825Z caller=head.go:348 component=tsdb msg="head GC completed" duration=54.809389ms
level=info ts=2018-03-22T17:00:01.781441043Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=201.581837ms
level=info ts=2018-03-22T19:00:00.138294083Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521734400000 maxt=1521741600000
level=info ts=2018-03-22T19:00:02.10048277Z caller=head.go:348 component=tsdb msg="head GC completed" duration=62.710639ms
level=info ts=2018-03-22T19:00:03.110967036Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.010385637s
level=info ts=2018-03-22T21:00:00.140686811Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521741600000 maxt=1521748800000
level=info ts=2018-03-22T21:00:01.609793643Z caller=head.go:348 component=tsdb msg="head GC completed" duration=73.567504ms
level=info ts=2018-03-22T21:00:02.591201996Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=981.317075ms
level=info ts=2018-03-22T21:00:02.794024341Z caller=compact.go:393 component=tsdb msg="compact blocks" count=3 mint=1521720000000 maxt=1521741600000
level=info ts=2018-03-22T23:00:00.140292345Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521748800000 maxt=1521756000000
level=info ts=2018-03-22T23:00:01.764628553Z caller=head.go:348 component=tsdb msg="head GC completed" duration=69.26054ms
level=info ts=2018-03-22T23:00:02.742063535Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=977.360047ms
level=info ts=2018-03-23T01:00:00.140658254Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1521756000000 maxt=1521763200000
level=info ts=2018-03-23T01:00:01.534012531Z caller=head.go:348 component=tsdb msg="head GC completed" duration=51.001912ms
level=info ts=2018-03-23T01:00:02.533997765Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=999.893834ms

[brian-brazil]

Was this an upgrade from 2.2.0?

[hmmxp]

Hi Brian,

Yes this was an upgrade, and this issue happened about few days after the upgrade

[brian-brazil]

What day did you perform the upgrade?

[hmmxp]

March 14 2018

[brian-brazil]

Dupe of #3943

[fabxc]

You sure this is a dupe (of the also closed issue)? Apparently this occurred days after the upgrade was done.

[hmmxp]

Dear Fab,

Am currently monitoring the entire Prometheus farm, and should i use the latest git clone of the master branch to ensure no further issue?

[fabxc]

No, please stick with 2.2.1 – master is generally not stable and any issues you'll hit with that will be near impossible to diagnose for us.

[bwplotka]

Wait @fabxc - from the graph I can see the missing gap is 12.03 ~02:00-14:00 so this gap was BEFORE upgrade.

I think this is old issue that was on 2.2.0. @hmmxp do you have any missing gap AFTER upgrade (data that was produced and compacted by 2.2.1)?

so yea.. I think all is good.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.