sidecar: Upload blocks from oldest

[bwplotka]

Ref: https://cloud-native.slack.com/archives/CK5RSSC10/p1571650833206600

It can happen during long network partition that uploaded blocks in the wrong order can cause compactor to miss one block during compaction. This results in overlapping blocks - compactor halts in that case. Read path works.

AC:

• Sort local blocks based on min/max time
• Optional: Add repair for verifier with vertical compaction - this will allow manual repair for those case (the alternative is to change a bit different meta to the overlapped block OR remove it)

[d-ulyanov]

Just to add some example:
in our case, we have 2 blocks: Block №1 (level=2) and block №2 (level=1) - compactor failed with error that these blocks overlapped.

Time ranges:
Block №1:

17-10-2019 00:00:00
17-10-2019 08:00:00

Block №2

17-10-2019 04:56:24
17-10-2019 06:00:00

Defenitelly, block №2 should be included to block №1, but it's not.

Root cause for compactor
Block №1 was compacted at 11:47:01, block №2 was uploaded at 14:36:12

Probably, root cause could be related with such errors (it was during 10min):

"level=warn ts=2019-10-17T08:50:25.558160695Z caller=shipper.go:333 msg="updating meta file failed" err="write /prometheus/thanos.shipper.json.tmp: no space left on device""

[obiesmans]

I'd like to work on this.