prometheus is not stable after restoring a snapshot.

[ryanisroot]

Proposal

Use case. Why is this important?
Snapshots should be restorable.

Bug Report

What did you do?

Trigger a snapshot, tar it up, then try to restore to an identical server.
snapshot is triggered with http://localhost:9090/api/v1/admin/tsdb/snapshot?skip_head=false

What did you expect to see?

1. old data from the snapshot - passed
2. new data being collected after the snapshot is restored - passed
3. prometheus continuing to work stably - fails

What did you see instead? Under which circumstances?

After a few hours of running with the snapshot, and continuing to collect data, prometheus will stop working, and refuse to start again.

I restore the snapshot on the server, then point the data dir to the restored directory and restart. The data from the snapshot appears immediately, as expected.

Prometheus continues to collect data from my servers, again, as expected.

At some point ( it has happened within 10 minutes and also taken several hours ) prometheus will go down and refuse to start again, until the data directory is cleared. It will then start up and collect data as usual.

Environment

• System information:

Linux 3.10.0-862.11.6.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 2.4.1 (branch: HEAD, revision: ce6716f)
build user: root@00c7c051c350
build date: 20180919-13:43:51
go version: go1.10.3

Behaviour was the same on 2.3.1, and was supposed to be fixed in 2.3.2. I have also tested with both of those versions.

• Prometheus configuration file:

global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
monitor: 'codelab-monitor'

rule_files:
scrape_configs:

• job_name: "node"
  scrape_interval: "15s"
  static_configs:
  • targets: ['server-01-staging-02.mydomain.ca:9100']

• Logs:
  Sep 20 19:54:57 ip-10-129-1-159 systemd[1]: Started Prometheus Server.
  Sep 20 19:54:57 ip-10-129-1-159 systemd[1]: Starting Prometheus Server...
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.106842205Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.1, branch=HEAD, revision=ce6716fe90ed67cb91cf8cf38a5de951853dcc2b)"
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.106907045Z caller=main.go:239 build_context="(go=go1.10.3, user=root@00c7c051c350, date=20180919-13:43:51)"
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.106927531Z caller=main.go:240 host_details="(Linux 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 ip-10-129-1-159 (none))"
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.106943953Z caller=main.go:241 fd_limits="(soft=1024, hard=4096)"
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.106959272Z caller=main.go:242 vm_limits="(soft=unlimited, hard=unlimited)"
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.107735716Z caller=main.go:554 msg="Starting TSDB ..."
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108220117Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537387200000 maxt=1537401600000 ulid=01CQTNJXCR80T94ZSFFNJEXZ9P
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108324649Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537401600000 maxt=1537423200000 ulid=01CQV3ACZG0WQ6GW8C2W1J1KR3
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108396602Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537444800000 maxt=1537452000000 ulid=01CQVQXCRXGRKGV4D315BV1VKQ
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108498694Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537423200000 maxt=1537444800000 ulid=01CQVQXGXR82004DT3MQKPAGH2
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108561545Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537452000000 maxt=1537457680887 ulid=01CQVSWWVGGNQFX2YK9GB615EZ
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.10867296Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537387200000 maxt=1537444800000 ulid=01CQW0QNYQEXMESMNJN0TH66MZ
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108739453Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537452000000 maxt=1537459200000 ulid=01CQW46FFVPC1WDDEHPDF91MFW
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108793681Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537452000000 maxt=1537459200000 ulid=01CQW46GKFVJB8AKQDT1B0MZEF
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108845858Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537452000000 maxt=1537459200000 ulid=01CQW46JPG1S32TNR4TW876253
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108899296Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537452000000 maxt=1537459200000 ulid=01CQW46PQY8RP1YMQYJRY7JB37
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.108949738Z caller=repair.go:35 component=tsdb msg="found healthy block" mint=1537452000000 maxt=1537459200000 ulid=01CQW46YPBVM98180J133YC97B
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.111017973Z caller=web.go:397 component=web msg="Start listening for connections" address=0.0.0.0:9090
  Sep 20 19:54:57 ip-10-129-1-159 prometheus[11264]: level=info ts=2018-09-20T19:54:57.12833645Z caller=main.go:423 msg="Stopping scrape discovery manager..."
  Sep 20 19:54:57 ip-10-129-1-159 systemd[1]: prometheus.service: main process exited, code=exited, status=1/FAILURE
  Sep 20 19:54:57 ip-10-129-1-159 systemd[1]: Unit prometheus.service entered failed state.
  Sep 20 19:54:57 ip-10-129-1-159 systemd[1]: prometheus.service failed.

[twforeman]

This is still happening on version 2.4.3. After about four hours the logs start complaining about data compaction errors due to time stamp issues and then a few hours later prometheus locks up.

This really needs to be addressed.

[krasi-georgiev]

could you also add the logs showing the block overlaps errors or if possible the full log

[twforeman]

I paste the logs in the issue I raised in the tsdb sub-project, but I'll paste them here too:

level=info ts=2018-11-06T18:16:23.804510458Z caller=compact.go:398 component=tsdb msg="write block" mint=1541512800000 maxt=1541520000000 ulid=01CVN3R6Y2BVRX3FTM1NSTZ0WG level=error ts=2018-11-06T18:16:35.898970177Z caller=db.go:305 component=tsdb msg="compaction failed" err="reload blocks: invalid block sequence: block time ranges overlap: [mint: 1541512800000, maxt: 1541512897799, range: 1m37s, blocks: 59]: <ulid: 01CVMN89AC8XWPAB8M99RNA9D4, mint: 1541512302989, maxt: 1541512897799, range: 9m54s>, <ulid: 01CVMZPZQ0SRA3PVNGMSMH1NEK, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZJEBQD57VKNQT32X2FDCV, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZK0W09KMF5J0JP1R1YJC9, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZKMQH4DWAFQT9WBXN7WXE, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZMDY47YKY2ZC4E8Z7ZG94, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZNFTWGQ2M8T3FMPHFQJET, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZHW8QARZ4RTWZ8GCX39SN, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZSBMXSPX2Y7JVAYT64HHR, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZVQXB0H7FC7B7MGT7P837, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVMZY5QKHN6BZW4SD5ZW9S7G, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN00KY24RS7W0VB68TDBTDA, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0303A05GAYNKVAG3W7EK2, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN05E4B3K3B866EQW92YX8Q, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN07W92QMDJ6JWYH65D3Q5R, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0A9D5NA0V6ZSK45G1W2SC, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0CPZZP1TMEJX08NRNZ2F3, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0F6PRRMXMYQZD5VEYTEWV, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0HM394AQSN474169KQC2M, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0M2VZS7NMVKBGPBGMSS3C, mint : 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0PHE98BNBKVZN4SCADAC1, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0RYJ5SW7T2ZGSQEZFV8ME, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0VE2W5KB3J01F5BW185ZN, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN0XWTAS9WWW2PGAR4GXY8H, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN10AZZABJ3D7M663MMKTHG, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN12V0773WE4X8NBGZN38FH, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN15B4CX8XVB5CCBFEZ3H57, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN17V7ZN8X2VYEPQBSV6FTD, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1AAGPGS5ACJH8QW2GMSEH, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1CTZZGPHDE9QQ79ZY7D5A, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1FBQAJEZAPKEY0NVP3MCE, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1HW5YHYXMMVWNS42TJQDG, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1MC3A2HAT1DRKXYWEJ5DY, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1PWRQX2JHNMVZTC9WMABB, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1SFW00NW3VJHF2TQKC0X3, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1W22PGXH4S8RFETNP0FQE, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN1YKPC8BW7X899NFD1WSHM, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN216MSGCNA71Z4S1S17JTK, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN23R4YN678J0KZ4YMSBPH4, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN26B545G9HRGNXWZ6W8T4J, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN28XEDKTY9EW5TPB26BWN4, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN2BERFWAXSBVDFM1J9QYCB, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN2DZDPJ6BD7BPZG7714TR3, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN2GHYWJJ5KC53FVSS02H44, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN2K5C5KCD8BS5XVGXZCSDQ, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN2NRMAVH8FBWZ5ESKPPVVZ, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN2RCAXKEV139X3EFV3MM5P, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN2TZPVTREZRWS4CDQ2MZ27, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN2XKW6E5MPAJX45DMVGXTE, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN3079DTN2F8RXHXEHS8TBP, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN32VRJNBQ3NNQ46P15MMX4, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN35G8FEP2JWH47RHFMGJ8F, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN38527P76Y9E112NGZPJQB, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN3ATWT61S2PWY8E9C0WXB6, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN3DGW3RFQ0GM9ERQRZDWDX, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN3G4YV0CKEQHRXS2PS8NQN, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN3JTD9G4MEDEDWPRBCPPRN, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN3NFMSSZ5KGECQET6K46AD, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>, <ulid: 01CVN3R6Y2BVRX3FTM1NSTZ0WG, mint: 1541512800000, maxt: 1541520000000, range: 2h0m0s>"

[danielfm]

Just to let you know I also saw this when trying to restore a snapshot created from a v2.4.3 Prometheus server to a v2.5.0.

The logs looked similar:

level=info ts=2018-12-16T19:37:28.449339228Z caller=compact.go:398 component=tsdb msg="write block" mint=1544961600000 maxt=1544968800000 ulid=01CYW8BXH19X76E9EB8SVHEN2A
level=error ts=2018-12-16T19:37:29.434774751Z caller=db.go:305 component=tsdb msg="compaction failed" err="reload blocks: invalid block sequence: block time ranges overlap: [mint: 1544961600000, maxt: 1544967982811, range: 1h46m22s, blocks: 170]: <ulid: 01CYVYRXADV3DJ78SYW547AT2E, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYJQ5Z72EY815GMX6XK5FX, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYJTQ2A2MEHE88AP8MAMBW, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYJZ0WJ0W0K6A03W9CPR4X, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYK5AHCRB6JR32SRTGR9DR, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYKFH4YBPAA4ME3D38852C, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYM1J62RE7CJ9X6DND2E7V, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYN36B257BAD1BPPHEB74T, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYQ09N1YRXWYKDZRNVS2PS, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVM949XEQRWFMA005FS1HNY, mint: 1544961600000, maxt: 1544967982811, range: 1h46m22s>, <ulid: 01CYVYTT9WJXBPXZ2FQ6PE1386, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYWQARGQXB2RV8KA7VNXKE, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVYYMA44SRR430GQNW734HH, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZ0HFADY0AD0NBXW8NF8JG, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZ2EH58WNED4D80WA8T9MG, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZ4BJCZR14M28BNRK49P4C, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZ68M4BSKMYYHWHXHM8F5E, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZ85QXVSNNJEMQB8BHEWQ0, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZA2SVZTFRQD77NRNYYERJ, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZBZVJ4H1E2TB0NS3Z9TPW, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZDWX5BH1NEXJC0110BGZK, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZFT2XDA5STBFFAD5RAJQS, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZHQ8FHSPX7VPE6BP5BZSJ, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZKMBPECAVVEYXTN2FR93E, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZNHFPWMH2X6JA4CH21B66, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZQEQVWCNHFSYKR09NNJE5, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZSBV4XAMXSQB4S1Q7E12N, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZV90N0HHXCN448HSNVMZV, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZX68KH708TGFJZ3EEC3BY, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYVZZ3D0G0E37Y60NNVGXWJ5, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW010HFH4C50D8N7CK2533P, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW02XMNPZD5EDF98PQFRWSF, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW04TSCHNCTV59AXNWYP7YK, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW06QXX5229K5D8D2M020TX, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW08N2DQNS6E76TZ3169X7N, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0AJ6766V6S6FN6MNX8KPZ, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0CFE5FVR296V1W3GES22X, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0ECM47EC1DR0JTSXQXK94, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0G9SVV58DSE102PMRYC2J, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0J704WKS7PXGHMP4NJP9B, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0M49V5KH1SSY5T89E929P, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0P1FSGNWXCTVJTZMM3R8R, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0QYNR38RKAD4S3TCA53B0, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0SVXGBBQJKMSQBEYHF0ZV, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0VS3XY3FWATKPGMK55AM9, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0XP7XBG0KVN3SX229DWRW, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW0ZKD5C8QK785EE368390S, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW11GHBYKM8Q9VNHGYYVQWM, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW13DQDTRST2P353R3GKR4F, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW15B1DCZG7JJ1H5FM3YNSN, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1788ZG6AWZFGP253W27C8, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW195GJ8EKNSPBXVW4RM1VH, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1B2P0SG505483CE9ZME21, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1CZWKYV4AEQH44N2739FS, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1EX3AYP23QAHVCRZ0CQNG, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1GTAGNST1VZBFQ2P0FAJX, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1JQKZB5X4B2FWB563BCGN, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1MMST7YRGZSRM202Z66R0, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1PHZH7QHZK7JZQGNNBSX1, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1RFDNBMWRVSZCCDPXQ746, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1TCMBES6GM5B958PK08R0, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1W9XB3ATA2AHWRFQ15D5P, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW1Y7BR87D108N1FYS8FP7F, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW204N605GFWQV4TV8AVMJW, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW221WSAJJNX7Z4BE1N0RS1, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW23Z4ZPB4BPPG6VW0GTY6S, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW25WDQP1H88FHNESJR0EWC, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW27SM9V35A11TSGXQZYEBE, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW29PW7M5SE5FSWTC09GHGC, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2BM3Q6G94BY1BF725XQ9C, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2DHD4RY34FMPC90EEDYES, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2FENE8S1H82JVX5ZT2F9W, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2HBWRH82AS10SGX6XRS2B, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2K96M3XWVY5HVGSBD80A8, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2N6FR9FATMCHCRP81MZR6, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2Q3SYRD4R2DSMSAJA4XMQ, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2S11Q8F27CNDSJJQMC04P, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2TYC0XR5H662EX3F5SHSY, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2WVVBRFBJ5EE4D9Q19S65, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW2YS5VFD5ZYDN2ZPMSWVWN, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW30PJPV6SZJWSSSYFRY8Y4, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW32KZAT01Z5AR0YQYW6ZB1, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW34HAE6GHZ3Z5BHRENWTWP, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW36EQJKB98J7NQEGP0GM72, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW38C8W4PDBNN5APPH7NWKT, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3A9SCFTQCRGPWQDEYJQST, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3C79Z85853RN42FP1VE6Z, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3E4QJGBQ7JBR0XTTHPTV1, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3G23YTBYVJJNE36AP3E54, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3HZHTP12TVR5T4JSP5VCR, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3KWY2BG6351SRK9H8JHPP, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3NTBPZZQBH1PGSEFA1FZG, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3QQS62MBQZFYNG28AFNZR, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3SN6KEX3BB3RJ0R53AGX6, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3VJVYGRMWJKJJ0BBNXBW2, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3XG8947VB9YK9A5HYAFV1, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW3ZDNAE60RTTCX7B0TG33Y, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW41B87YG1ZZC9P93XHDGA2, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW438MNKCTTKQ2C4E7MSDHD, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW45635X5CMJ86CZMXTCJQF, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW473E5490EBP8T4EBFF606, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW490WN8DBB1EDV9JZ6VS5N, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4AYBMQKRV5Y9KHNC7SRGM, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4CVSR6QSQWEDFPHKHHMBV, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4ESBF8RYFQK3TDXM6KATT, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4GPTBXFCHN8FBK3M8N6WD, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4JM8K86FEBJ4CF6MT28CH, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4MHTC8VK342TBS4ZBZ325, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4PF9ABD8689GQ1VG3FG5J, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4RCQS5D2FMDRC6T8PABFG, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4TA5W576KEKJP1G14KR00, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4W7Q7Y78N6P3GSZT5Q6JK, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW4Y59PFHXAFMKQ4E69BZWY, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW502WRX6NWD768D4RZKZGX, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW520JHQ58PXRCJAW0GCVHM, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW53Y191Y521GS40FY43S4A, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW55VJHS8GVNTZ75EGXWGX5, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW57S2GFB91Z78DVK1HKEEF, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW59PK17F83MFB5A1PFKPYR, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5BM2RJJY4ZVZX4EC5X5PH, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5DHGX8Q62X1WSNA2D5K7D, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5FF1DZ8RW831QWAWQQT2T, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5HCHCT4TQQ6GH71SP8WN3, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5KA2PH8GZF11WEWJ753JJ, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5N7PZKQPZBR21D6NXX25V, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5Q57NVB3Y88S8D49QA61N, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5S2R3TZWT1355Z4G0G3Z9, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5V0J7BQRYYDS2WB44SR8M, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5WY1VK142PW85TQ9MYEND, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW5YVH9J612G3G2BK3XATQN, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW60S0EPEVVKE0RRF256FF8, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW62PJ3WXXH5SEXS22H9NWX, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW64M174DCJRRAFH3B3FMWT, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW66HQCR0ZBW7BZPG1CNK08, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW68F9NYS1JXEZ22736BKXS, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6AD0MJ1FDD3AKY3CVQDEJ, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6CAKADAHZQ3VKRR10WC2J, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6E85FW5ZD9CVXYM0RK70D, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6G5R13WS4HPJE22ESKCFJ, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6J3BQV201YYJ4ZZX3B915, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6M0ZG3CZZ05SDMTKG3H0Q, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6NYN3ES7MD8K6EJ3F0HSN, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6QW8PKRDNZAWE6WHMWM2F, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6SSWX8P3D8PBTKVPBN8F4, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6VQKA3J7355SW4Y25VYVV, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6XN683VB8B10A5TM9S80Q, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW6ZJR0FDHHNZ567ZBDV2A5, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW71GD3J6HS5MK84XDB4HMH, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW73DZ1BB594DSH3HEQ8WMZ, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW75BQDESE1CMG14CGGT2ZR, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW779DH29FPRRDERDC5Z455, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7970YETS2FC3YTAG1KF0Z, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7B4TH4VD6W2JARK6BM73Q, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7D2JGRA5497Y67P27BKFW, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7F06XWJ7EJN2R8XGQWFBW, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7GXWY5NVYRCV6GS2F853E, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7JVHDJPMSKMV84P43WEER, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7MS64RXAZM03NAV1ZBN7C, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7PPV6QSBATWH9RRPNFH5X, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7RMFTFK1NAKXQ7KWMZ5GS, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7TJ6HNXYJ2EJ7RS6QMEDT, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7WFSCJNVMPYE1WTGQM7TE, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW7YDFMMM9RF6WXRC7PFPJ5, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW80B4T3T0DHWY7M3M1WDY7, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW828T9C5W0A9GSYD6CCS47, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW846EN1WJ9ADH92E5V5YN4, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW864BGEMD1P37TCNTMT8BA, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW88210MVYC1JJTQE8HGBPT, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW89ZQZ95DA8D76RRZ4ZZTD, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>, <ulid: 01CYW8BXH19X76E9EB8SVHEN2A, mint: 1544961600000, maxt: 1544968800000, range: 2h0m0s>"

I'm now using the v2.6.0-rc.1, which included the fix in prometheus/tsdb#461, and saw no issues so far.

[krasi-georgiev]

@danielfm nice , thanks for the update.