prometheus drop many metrics sample after restart

[ranbochen]

What did you do?
restart prometheus

What did you expect to see?
reload all metrics correctly。

• Logs:

level=info ts=2017-12-28T08:41:13.6108535Z caller=main.go:220 msg="Starting Prometheus" version="(version=, branch=, revision=)"
level=info ts=2017-12-28T08:41:13.6118315Z caller=main.go:221 build_context="(go=go1.9.2, user=, date=)"
level=info ts=2017-12-28T08:41:13.6118315Z caller=main.go:222 host_details=(windows)
level=info ts=2017-12-28T08:41:13.6118315Z caller=main.go:223 fd_limits=N/A
level=info ts=2017-12-28T08:41:13.6165317Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2017-12-28T08:41:13.6175015Z caller=main.go:400 msg="Starting TSDB ..."
level=info ts=2017-12-28T08:41:13.6175015Z caller=web.go:383 component=web msg="Start listening for connections" address=localhost:9090
level=warn ts=2017-12-28T08:41:34.0433042Z caller=head.go:317 component=tsdb msg="unknown series references in WAL samples" count=35160
level=info ts=2017-12-28T08:41:34.0452695Z caller=main.go:410 msg="TSDB started"
level=info ts=2017-12-28T08:41:34.0452695Z caller=main.go:492 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2017-12-28T08:41:34.0462225Z caller=main.go:387 msg="Server is ready to receive requests."
level=info ts=2017-12-28T08:42:34.0530711Z caller=compact.go:385 component=tsdb msg="compact blocks" count=3 mint=1513620000000 maxt=1513792800000
level=info ts=2017-12-28T08:42:34.6088178Z caller=compact.go:385 component=tsdb msg="compact blocks" count=3 mint=1513792800000 maxt=1513987200000
level=info ts=2017-12-28T08:42:34.9716301Z caller=compact.go:385 component=tsdb msg="compact blocks" count=3 mint=1513987200000 maxt=1514181600000
level=info ts=2017-12-28T08:42:35.4085317Z caller=compact.go:385 component=tsdb msg="compact blocks" count=3 mint=1514181600000 maxt=1514376000000
level=info ts=2017-12-28T08:42:35.7220057Z caller=compact.go:385 component=tsdb msg="compact blocks" count=2 mint=1513620000000 maxt=1513987200000
level=info ts=2017-12-28T09:00:09.0970074Z caller=compact.go:385 component=tsdb msg="compact blocks" count=1 mint=1514440800000 maxt=1514448000000
level=info ts=2017-12-28T09:00:09.2102843Z caller=head.go:345 component=tsdb msg="head GC completed" duration=3.9019ms
level=info ts=2017-12-28T09:00:09.2102843Z caller=head.go:354 component=tsdb msg="WAL truncation completed" duration=0s

seeing of this log msg: "unknown series references in WAL samples" count=35160
prometheus drops a lot of metrics samples!

see:
prometheus/tsdb#21
#3489

[wbh1]

Same issue here. Running in a docker container with a bind mount to a directory on the host. All the files are still there but the data just doesn't get loaded back in upon startup, I guess.

level=info ts=2018-03-09T20:29:35.129570893Z caller=main.go:226 build_context="(go=go1.9.2, user=root@6e784304d3ff, date=20180119-12:01:23)"
level=info ts=2018-03-09T20:29:35.129599858Z caller=main.go:227 host_details="(Linux 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 3fdbc44a7747 (none))"
level=info ts=2018-03-09T20:29:35.129625884Z caller=main.go:228 fd_limits="(soft=65536, hard=65536)"
level=info ts=2018-03-09T20:29:35.191563624Z caller=main.go:499 msg="Starting TSDB ..."
level=info ts=2018-03-09T20:29:35.227876171Z caller=web.go:383 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=warn ts=2018-03-09T20:30:22.052793268Z caller=head.go:320 component=tsdb msg="unknown series references in WAL samples" count=55
level=info ts=2018-03-09T20:30:22.069557789Z caller=main.go:509 msg="TSDB started"
level=info ts=2018-03-09T20:30:22.069610437Z caller=main.go:585 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2018-03-09T20:30:22.073376453Z caller=main.go:486 msg="Server is ready to receive web requests."

[mattouille]

I started seeing this a Chef service restart behavior. I'm still working on troubleshooting it.

[ijonsnow]

Hi,
is this issue resolved?
Am facing data loss in prometheus 2.1 version. Have data loss points in grafana dashboards.
Reload configuration issue, results in data loss points.

https://www.robustperception.io/reloading-prometheus-configuration

[sipian]

Looks similar to #4519

[simonpasquier]

If anything please retest with the latest Prometheus version (v2.3.2) that includes many bug fixes for tsdb. Also it doesn't look exactly like #4519 to me as this one is about restarting Prometheus (not reloading).

[dominikh]

I've hit the same issue with 2.4.2 (version="(version=2.4.2, branch=HEAD, revision=c305ffaa092e94e9d2dbbddf8226c4813b1190a0)").

I wonder if there's any connection to prometheus/tsdb#21

Prometheus log

server1# ./prometheus       
level=info ts=2018-09-25T02:38:29.526367852Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.2, branch=HEAD, revision=c305ffaa092e94e9d2dbbddf8226c4813b1190a0)"
level=info ts=2018-09-25T02:38:29.526448646Z caller=main.go:239 build_context="(go=go1.10.3, user=root@dcde2b74c858, date=20180921-07:36:35)"
level=info ts=2018-09-25T02:38:29.526470926Z caller=main.go:240 host_details=(freebsd)
level=info ts=2018-09-25T02:38:29.526495291Z caller=main.go:241 fd_limits="(soft=3772890, hard=3772890)"
level=info ts=2018-09-25T02:38:29.526514529Z caller=main.go:242 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2018-09-25T02:38:29.527921517Z caller=main.go:554 msg="Starting TSDB ..."
level=info ts=2018-09-25T02:38:29.528080083Z caller=web.go:397 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-09-25T02:38:29.528444404Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537812000000 maxt=1537819200000 ulid=01CR77XARHSKQDZASHZZSX3P2Z
level=info ts=2018-09-25T02:38:29.528540541Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537819200000 maxt=1537826400000 ulid=01CR77XASEG4487PBXDGVTHHRQ
level=info ts=2018-09-25T02:38:29.528612724Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537826400000 maxt=1537833600000 ulid=01CR77XATFHGDJG9P78V99BW7W
level=info ts=2018-09-25T02:38:29.528769769Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537768800000 maxt=1537790400000 ulid=01CR77XAVE4ZWH0AZ917GYRGTM
level=info ts=2018-09-25T02:38:29.528871542Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537790400000 maxt=1537812000000 ulid=01CR77XAWTC46J928BEK6YE2CG
level=warn ts=2018-09-25T02:38:29.615108385Z caller=head.go:371 component=tsdb msg="unknown series references" count=50245
level=info ts=2018-09-25T02:38:29.618446503Z caller=main.go:564 msg="TSDB started"
level=info ts=2018-09-25T02:38:29.618531078Z caller=main.go:624 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-09-25T02:38:29.619960822Z caller=main.go:650 msg="Completed loading of configuration file" filename=prometheus.yml
level=info ts=2018-09-25T02:38:29.619993653Z caller=main.go:523 msg="Server is ready to receive web requests."
^Clevel=warn ts=2018-09-25T02:39:39.851666249Z caller=main.go:398 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2018-09-25T02:39:39.85174263Z caller=main.go:423 msg="Stopping scrape discovery manager..."
level=info ts=2018-09-25T02:39:39.851759987Z caller=main.go:437 msg="Stopping notify discovery manager..."
level=info ts=2018-09-25T02:39:39.85177354Z caller=main.go:459 msg="Stopping scrape manager..."
level=info ts=2018-09-25T02:39:39.851807335Z caller=main.go:433 msg="Notify discovery manager stopped"
level=info ts=2018-09-25T02:39:39.851784873Z caller=main.go:419 msg="Scrape discovery manager stopped"
level=info ts=2018-09-25T02:39:39.851862128Z caller=manager.go:638 component="rule manager" msg="Stopping rule manager..."
level=info ts=2018-09-25T02:39:39.851881953Z caller=manager.go:644 component="rule manager" msg="Rule manager stopped"
level=info ts=2018-09-25T02:39:39.851916549Z caller=main.go:453 msg="Scrape manager stopped"
level=info ts=2018-09-25T02:39:39.855099309Z caller=notifier.go:512 component=notifier msg="Stopping notification manager..."
level=info ts=2018-09-25T02:39:39.855183239Z caller=main.go:608 msg="Notifier manager stopped"
level=info ts=2018-09-25T02:39:39.85535852Z caller=main.go:620 msg="See you next time!"
server1# ./prometheus
level=info ts=2018-09-25T02:39:44.670238657Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.2, branch=HEAD, revision=c305ffaa092e94e9d2dbbddf8226c4813b1190a0)"
level=info ts=2018-09-25T02:39:44.670318165Z caller=main.go:239 build_context="(go=go1.10.3, user=root@dcde2b74c858, date=20180921-07:36:35)"
level=info ts=2018-09-25T02:39:44.670340061Z caller=main.go:240 host_details=(freebsd)
level=info ts=2018-09-25T02:39:44.670361558Z caller=main.go:241 fd_limits="(soft=3772890, hard=3772890)"
level=info ts=2018-09-25T02:39:44.670380632Z caller=main.go:242 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2018-09-25T02:39:44.671788247Z caller=web.go:397 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-09-25T02:39:44.671772107Z caller=main.go:554 msg="Starting TSDB ..."
level=info ts=2018-09-25T02:39:44.672191239Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537812000000 maxt=1537819200000 ulid=01CR77XARHSKQDZASHZZSX3P2Z
level=info ts=2018-09-25T02:39:44.672279677Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537819200000 maxt=1537826400000 ulid=01CR77XASEG4487PBXDGVTHHRQ
level=info ts=2018-09-25T02:39:44.672354588Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537826400000 maxt=1537833600000 ulid=01CR77XATFHGDJG9P78V99BW7W
level=info ts=2018-09-25T02:39:44.672533257Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537768800000 maxt=1537790400000 ulid=01CR77XAVE4ZWH0AZ917GYRGTM
level=info ts=2018-09-25T02:39:44.672621292Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537790400000 maxt=1537812000000 ulid=01CR77XAWTC46J928BEK6YE2CG
level=warn ts=2018-09-25T02:39:44.769766936Z caller=head.go:371 component=tsdb msg="unknown series references" count=50245
level=info ts=2018-09-25T02:39:44.771257145Z caller=main.go:564 msg="TSDB started"
level=info ts=2018-09-25T02:39:44.771336206Z caller=main.go:624 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-09-25T02:39:44.77262786Z caller=main.go:650 msg="Completed loading of configuration file" filename=prometheus.yml
level=info ts=2018-09-25T02:39:44.772651386Z caller=main.go:523 msg="Server is ready to receive web requests."

[brian-brazil]

I've also gotten a report of this with a 2.4.2, which is OOMing.

[hhoffstaette]

Still happens with 2.4.3/2.5.0.

[bentsi]

Using official Prometheus docker v.2.3.2 [prom/prometheus:v2.3.2] with data directory mounted outside the container.
After restarting the container (docker kill and docker start commands) and when GC process
runs the sample data got truncated to the mint time that is close to the start time of the container.
Container log:

level=info ts=2018-11-20T18:22:30.265089618Z caller=main.go:222 msg="Starting Prometheus" version="(version=2.3.2, branch=HEAD, revision=71af5e29e815795e9dd14742ee7725682fa14b7b)"
level=info ts=2018-11-20T18:22:30.265874369Z caller=main.go:223 build_context="(go=go1.10.3, user=root@5258e0bd9cc1, date=20180712-14:02:52)"
level=info ts=2018-11-20T18:22:30.265904784Z caller=main.go:224 host_details="(Linux 4.9.119-44.140.amzn1.x86_64 #1 SMP Fri Aug 10 19:17:29 UTC 2018 x86_64 b9b48dff9c3d (none))"
level=info ts=2018-11-20T18:22:30.265920849Z caller=main.go:225 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2018-11-20T18:22:30.268836379Z caller=main.go:533 msg="Starting TSDB ..."
level=info ts=2018-11-20T18:22:30.312629564Z caller=web.go:415 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-11-20T18:22:47.765035897Z caller=main.go:543 msg="TSDB started"
level=info ts=2018-11-20T18:22:47.765964029Z caller=main.go:603 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2018-11-20T18:22:47.768390071Z caller=main.go:629 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2018-11-20T18:22:47.768824273Z caller=main.go:502 msg="Server is ready to receive web requests."
level=info ts=2018-11-20T21:00:01.904554093Z caller=compact.go:398 component=tsdb msg="write block" mint=1542736800000 maxt=1542744000000 ulid=01CWSERDFS1JPM5PXHMDM4MM42
level=info ts=2018-11-20T21:00:03.711786834Z caller=head.go:348 component=tsdb msg="head GC completed" duration=120.233476ms
level=info ts=2018-11-20T21:00:16.413516117Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=12.701452111s

One can see that data was truncated till 20:00, as the value mint=1542736800000.

[Starefossen]

Just got hit by the same while upgrading form 1.4.2 to 1.6.0:

level=info ts=2019-01-04T15:13:52.689340108Z caller=main.go:243 msg="Starting Prometheus" version="(version=2.6.0, branch=HEAD, revision=dbd1d58c894775c0788470944b818cc724f550fb)"
level=info ts=2019-01-04T15:13:52.689434621Z caller=main.go:244 build_context="(go=go1.11.3, user=root@bf5760470f13, date=20181217-15:14:46)"                                    
level=info ts=2019-01-04T15:13:52.689470651Z caller=main.go:245 host_details="(Linux 4.4.0-141-generic #167-Ubuntu SMP Wed Dec 5 10:40:15 UTC 2018 x86_64 prometheus-v2-64f56fddd9-g2lvn (none))"
level=info ts=2019-01-04T15:13:52.689505414Z caller=main.go:246 fd_limits="(soft=1048576, hard=1048576)"                                                                         
level=info ts=2019-01-04T15:13:52.689537209Z caller=main.go:247 vm_limits="(soft=unlimited, hard=unlimited)"                                                                     
level=info ts=2019-01-04T15:13:52.690642093Z caller=main.go:561 msg="Starting TSDB ..."
level=info ts=2019-01-04T15:13:52.690697541Z caller=web.go:429 component=web msg="Start listening for connections" address=0.0.0.0:9090                                          
level=info ts=2019-01-04T15:13:52.694296026Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546149600000 maxt=1546171200000 ulid=01CZZT25CHACN24ZFQGCEBY459  
level=info ts=2019-01-04T15:13:52.695426952Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546171200000 maxt=1546192800000 ulid=01D00ENBGEPQ2Y32G6DFE4T5XD  
level=info ts=2019-01-04T15:13:52.696701447Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546192800000 maxt=1546214400000 ulid=01D0138HKV0XQA83GJD8158JJ1  
level=info ts=2019-01-04T15:13:52.697804274Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546214400000 maxt=1546236000000 ulid=01D01QVQD5VDWYYDH9BTZT6AHW  
level=info ts=2019-01-04T15:13:52.698853143Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546236000000 maxt=1546257600000 ulid=01D02CEX6C2KGJXNSKZQF6AA3T  
level=info ts=2019-01-04T15:13:52.70001739Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546257600000 maxt=1546279200000 ulid=01D03122FTTW42MHJHFWWHRN5W   
level=info ts=2019-01-04T15:13:52.70119071Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546279200000 maxt=1546300800000 ulid=01D03NN7NCFW4C4RSB1PEPPZB8   
level=info ts=2019-01-04T15:13:52.702280751Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546300800000 maxt=1546322400000 ulid=01D04A8DWFR7AWD8MET4GVDV8R  
level=info ts=2019-01-04T15:13:52.703325905Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546322400000 maxt=1546344000000 ulid=01D04YVKFD82NTMDGJZ3NF1X57  
level=info ts=2019-01-04T15:13:52.704360318Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546344000000 maxt=1546365600000 ulid=01D05KET3DJP6BXRBKMKPAZ2PN  
level=info ts=2019-01-04T15:13:52.705370692Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546365600000 maxt=1546387200000 ulid=01D0681Z9QWQJJJMQC10NC1C2F  
level=info ts=2019-01-04T15:13:52.706579136Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546387200000 maxt=1546408800000 ulid=01D06WN4DNF87VHJ53AXXH1NG4  
level=info ts=2019-01-04T15:13:52.707586026Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546408800000 maxt=1546430400000 ulid=01D07H8A6FD76E77CDXNKW2QZ0  
level=info ts=2019-01-04T15:13:52.708706442Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546430400000 maxt=1546452000000 ulid=01D085VGDM4G3DMNSKWNYAK4BF  
level=info ts=2019-01-04T15:13:52.709940276Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546452000000 maxt=1546473600000 ulid=01D08TEP6E6V5M3KSCB0Y38EY7  
level=info ts=2019-01-04T15:13:52.71100502Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546473600000 maxt=1546495200000 ulid=01D09F1VF05WHKYZ22MJF10BNK   
level=info ts=2019-01-04T15:13:52.712465008Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546495200000 maxt=1546516800000 ulid=01D0A3N1P87CD5KFPVY20TDWE9  
level=info ts=2019-01-04T15:13:52.713585837Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546516800000 maxt=1546538400000 ulid=01D0AR87E54BFVQ0W5XKV675JY  
level=info ts=2019-01-04T15:13:52.714796652Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546538400000 maxt=1546560000000 ulid=01D0BCVAM53JK69AB9VVXXPPPB  
level=info ts=2019-01-04T15:13:52.716396868Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546560000000 maxt=1546581600000 ulid=01D0C1EJWGMRJTZPF2SKDD8351  
level=info ts=2019-01-04T15:13:52.717587591Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546603200000 maxt=1546610400000 ulid=01D0CP34KKS1DCSK4GP02401BS  
level=info ts=2019-01-04T15:13:52.718816213Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1546581600000 maxt=1546603200000 ulid=01D0CP3B12EGRTSA334V9QQT8N  
level=warn ts=2019-01-04T15:14:01.126615603Z caller=head.go:434 component=tsdb msg="unknown series references" count=3129                                                        
level=info ts=2019-01-04T15:14:01.418320775Z caller=main.go:571 msg="TSDB started"
level=info ts=2019-01-04T15:14:01.418379254Z caller=main.go:631 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml                                         
level=info ts=2019-01-04T15:14:01.42145393Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"    
level=info ts=2019-01-04T15:14:01.423045399Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"   
level=info ts=2019-01-04T15:14:01.424546458Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"   
level=info ts=2019-01-04T15:14:01.426056098Z caller=kubernetes.go:201 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"   
level=info ts=2019-01-04T15:14:01.429339883Z caller=main.go:657 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml                            
level=info ts=2019-01-04T15:14:01.429380004Z caller=main.go:530 msg="Server is ready to receive web requests."                                                                   

[gt510]

Experiencing the same right after install.

Images:
prom/prometheus:v2.6.0
prom/alertmanager:v0.15.3
grafana/grafana:5.4.2

level=info ts=2019-01-16T09:09:30.885018243Z caller=main.go:243 msg="Starting Prometheus" version="(version=2.6.0, branch=HEAD, revision=dbd1d58c894775c0788470944b818cc724f550fb)"
level=info ts=2019-01-16T09:09:30.923677395Z caller=main.go:244 build_context="(go=go1.11.3, user=root@bf5760470f13, date=20181217-15:14:46)"
level=info ts=2019-01-16T09:09:30.923728883Z caller=main.go:245 host_details="(Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 7ea427220bad (none))"
level=info ts=2019-01-16T09:09:30.923752207Z caller=main.go:246 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-01-16T09:09:30.923768169Z caller=main.go:247 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-01-16T09:09:30.961633513Z caller=main.go:561 msg="Starting TSDB ..."
level=info ts=2019-01-16T09:09:30.963802438Z caller=web.go:429 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-01-16T09:09:30.980083864Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547046464600 maxt=1547100000000 ulid=01D0VPPF5TN1YBZQ7CHPE4HNGH
level=info ts=2019-01-16T09:09:30.98216642Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547100000000 maxt=1547164800000 ulid=01D0XG8MF2QRQDPE1VQJC3HSCZ
level=info ts=2019-01-16T09:09:30.9841129Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547164800000 maxt=1547229600000 ulid=01D15675Q4SQ0PZFTDNM85SBCD
level=info ts=2019-01-16T09:09:30.985975911Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547229600000 maxt=1547294400000 ulid=01D15676F7PZ2TH7241QHSW1SZ
level=info ts=2019-01-16T09:09:30.987287896Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547294400000 maxt=1547359200000 ulid=01D15676GNRB3NH0AVJ5YRM49Y
level=info ts=2019-01-16T09:09:30.989136229Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547359200000 maxt=1547424000000 ulid=01D15676HQRZ16N6Y81KHVHPJY
level=info ts=2019-01-16T09:09:30.989754831Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547488800000 maxt=1547510400000 ulid=01D17VS0A8T9AR0AP85A9VYXBB
level=info ts=2019-01-16T09:09:30.990324989Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547424000000 maxt=1547488800000 ulid=01D17VS0BJ2212NHCRMYANV9WG
level=info ts=2019-01-16T09:09:30.991107234Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547532000000 maxt=1547539200000 ulid=01D18BTAN8MMDVG8QA1ZMT7WEK
level=info ts=2019-01-16T09:09:30.991672383Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547510400000 maxt=1547532000000 ulid=01D18BTAVPNM80NRGW7EZFAAGF
level=info ts=2019-01-16T09:09:30.99217389Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547539200000 maxt=1547546400000 ulid=01D18JP1WXRK6Q9YHWQ57J4WZ4
level=info ts=2019-01-16T09:09:30.992515031Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1547546400000 maxt=1547553600000 ulid=01D18SHS79QAJK3ETCT53P5GJZ
level=warn ts=2019-01-16T09:09:30.993690365Z caller=wal.go:116 component=tsdb msg="last page of the wal is torn, filling it with zeros" segment=/prometheus/wal/00000012
level=warn ts=2019-01-16T09:09:37.003860267Z caller=head.go:434 component=tsdb msg="unknown series references" count=2853808
level=info ts=2019-01-16T09:09:37.013757078Z caller=main.go:571 msg="TSDB started"
level=info ts=2019-01-16T09:09:37.013836967Z caller=main.go:631 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-01-16T09:09:37.022375458Z caller=main.go:657 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-01-16T09:09:37.022426729Z caller=main.go:530 msg="Server is ready to receive web requests."