store: fails to load compacted block

[FUSAKLA]

Thanos, Prometheus and Golang version used:
Thanos 0.10.0

Object Storage Provider:
S3 (Minio)

What happened:
Store tries to load compacted block right after it was compacted and deleted by the compactor and fails with error:

fetch series for block 01DZ37HG54CRAP6V88CSJGZ4CD: preload chunks: read range for 0: get range reader: The specified key does not exist`

On version 0.7.0 everything was working fine and never seen this issue.
After upgrade to 0.9.0 (skipped 0.8.0) this error started to show up.
Then tried upgrading to 0.10.0 and the issue still persists.

Store has default sync delay of --sync-block-duration=3m and --block-sync-concurrency=20 and compactor has --block-sync-concurrency=20 and --compact.concurrency=1.
None of those changed during upgrade.

Full logs to relevant components:

Logs of compactor, store and query sorted by time.
You can see that issue occurs right after deletion of the impacted block and resolves after first metadata sync of the store.

Logs

thanos-store | level=info ts=2020-01-21T12:53:39.459476675Z caller=fetcher.go:361 component=block.MetaFetcher msg="successfully fetched block metadata" duration=51.950461ms cached=213 returned=213 partial=0
thanos-store | level=info ts=2020-01-21T12:50:39.470942723Z caller=fetcher.go:361 component=block.MetaFetcher msg="successfully fetched block metadata" duration=63.424729ms cached=213 returned=213 partial=0
thanos-query | level=error ts=2020-01-21T12:50:05.452729126Z caller=engine.go:617 msg="error selecting series set" err="proxy Series(): Addr: thanos-store:10901 LabelSets: [...] Mint: 1541030400000 Maxt: 1579600800000: rpc error: code = Aborted desc = fetch series for block 01DZ37HG54CRAP6V88CSJGZ4CD: preload chunks: read range for 0: get range reader: The specified key does not exist."
thanos-query | level=error ts=2020-01-21T12:50:05.452516561Z caller=proxy.go:291 err="Addr: thanos-store:10901 LabelSets: [...] Mint: 1541030400000 Maxt: 1579600800000: rpc error: code = Aborted desc = fetch series for block 01DZ37HG54CRAP6V88CSJGZ4CD: preload chunks: read range for 0: get range reader: The specified key does not exist."
thanos-query | level=error ts=2020-01-21T12:49:14.698811399Z caller=engine.go:617 msg="error selecting series set" err="proxy Series(): Addr: thanos-store:10901 LabelSets: [...] Mint: 1541030400000 Maxt: 1579600800000: rpc error: code = Aborted desc = fetch series for block 01DZ37HG54CRAP6V88CSJGZ4CD: preload chunks: read range for 0: get range reader: The specified key does not exist."
thanos-query | level=error ts=2020-01-21T12:49:14.698693758Z caller=proxy.go:291 err="Addr: thanos-store:10901 LabelSets: [...] Mint: 1541030400000 Maxt: 1579600800000: rpc error: code = Aborted desc = fetch series for block 01DZ37HG54CRAP6V88CSJGZ4CD: preload chunks: read range for 0: get range reader: The specified key does not exist."
thanos-query | level=error ts=2020-01-21T12:48:54.756252245Z caller=proxy.go:291 err="Addr: thanos-store:10901 LabelSets: [...] Mint: 1541030400000 Maxt: 1579600800000: rpc error: code = Aborted desc = fetch series for block 01DZ37HG54CRAP6V88CSJGZ4CD: preload chunks: read range for 0: get range reader: The specified key does not exist."
thanos-query | level=error ts=2020-01-21T12:48:54.756467346Z caller=engine.go:617 msg="error selecting series set" err="proxy Series(): Addr: thanos-store:10901 LabelSets: [...] Mint: 1541030400000 Maxt: 1579600800000: rpc error: code = Aborted desc = fetch series for block 01DZ37HG54CRAP6V88CSJGZ4CD: preload chunks: read range for 0: get range reader: The specified key does not exist."
thanos-compactor | level=info ts=2020-01-21T12:48:43.596508007Z caller=compact.go:834 compactionGroup=0@6519752695378234841 msg="deleting compacted block" old_block=01DZ37HG54CRAP6V88CSJGZ4CD
thanos-compactor | level=info ts=2020-01-21T12:48:32.456010922Z caller=compact.go:441 msg="compact blocks" count=4 mint=1579564800000 maxt=1579593600000 ulid=01DZ429YXJY83A43FCHCH37CVV sources="[01DZ30NRX2AB2Y0BYNSDDT5H36 01DZ37HG54CRAP6V88CSJGZ4CD 01DZ3ED7D4N5132DT56HA2RS4Q 01DZ3N8YN1748FCH839MSGB0Q0]" duration=47.934270524s
thanos-store | level=info ts=2020-01-21T12:47:39.474056526Z caller=fetcher.go:361 component=block.MetaFetcher msg="successfully fetched block metadata" duration=66.506482ms cached=219 returned=219 partial=0
thanos-store | level=info ts=2020-01-21T12:44:39.459117009Z caller=fetcher.go:361 component=block.MetaFetcher msg="successfully fetched block metadata" duration=51.564796ms cached=232 returned=232 partial=1

I went through lot of PRs touching related stuff (#1937, #1934, #1525, #1583, #1505, #1666) but cannot find what change could have caused this.

I suppose I can mitigate it by lowering the --sync-block-duration to something lower than 3m but that won't resolve the issue, just lower the time when the error would happen.

[bwplotka]

Thanks for all of those details. It's duplicate of #564 and our @khyatisoneji is working on this ((: