objstore: Be clear of our consistency assumptions or handle eventual consistency gracefully.

[bwplotka]

As per discussion here #282 it is clear that we care about strong consistency for our block storages. However we are not sure if S3 is really strong consistent for our usage, or any other storage that will be implemented, will it be true as well?

Options:

1. State clear that we assume strong consistency for write-read. GCS has strong consistency. S3 has as well, but with some caveats, so we are not 100% sure. Let's invest some time to dig it further. In the worst case, we can add code to make strong consistency happen (poll the block until it is listable? and fully uploaded?)

2. Set up some consistent storage then additionally (etcd?). We wanted to avoid external deps, so this is not really preferable solution.

3. Handle eventual consistency gracefully. This needs to be investigated more deeply, but for example for compactor it can mean optimistic compaction: Refactor compactor to be aware that we are not really sure that these blocks we have visible now are really ALL of them. It kind of works like that when we add support for vertical compactions (?). But it needs to be investigated more for other consequences.

We tried to mitigate lack of certainty of consistency with sync-delay but that introduced other problems, so we needed to remove it. Worth to revisit this issue and be clear about our consistency assumptions in Thanos.

[bwplotka]

https://thanos.io/proposals/201901-read-write-operations-bucket.md/ was created in Dec 31, 2018, we will open separate issue to track work defined there.