Thanos compactor should retry failed bucket operation instead retry the whole compaction

[alexqyle]

Is your proposal related to a problem?

Not all bucket clients Thanos using are supporting retry failed operation internally. The current logic Thanos dealing with bucket operation failure is retrying the whole compaction iteration again. This is not ideal for single bucket operation failure during compaction. If it is a relative big compaction, it will waste a lot time to compact it again due to one bucket operation failure.

The following are retriable bucket failure in Thanos compactor:

• download block: https://github.com/thanos-io/thanos/blob/main/pkg/compact/compact.go#L1031
• upload block: https://github.com/thanos-io/thanos/blob/main/pkg/compact/compact.go#L1139
• mark block for deletion: https://github.com/thanos-io/thanos/blob/main/pkg/block/block.go#L200

Describe the solution you'd like

Could have bucket client wrapper on top of each bucket client implementation to have configurable retry logic to retry bucket operation on underlying bucket client.

[fpetkovski]

That's a great suggestion. We can also consider adding retries directly in https://github.com/thanos-io/objstore.