Double compaction after completing partitioning compaction job #7129
Description
Describe the bug
As a follow-up from #7075, double compaction can occur for a completed partitioning compaction group.
Scenario:
T0 Compactor 1: Compaction job for partitioning group version 1 was completed. The cleaner updates the bucket index with the new result blocks and starts deleting parent blocks and partitioned group info file.
T1 Compactor 2: Once bucket index is updated, it groups parent blocks with result blocks of version 1 and created version 2. Starts compacting group 0 of version 2.
T2 Compactor 1: Finishes deleting parent blocks and partitioned group file.
T3 Compactor 2: Finishes compacting group 0 of version 2. Remaining groups of version 2 are invalid because the parent blocks have been deleted. Version 2 is deleted.
The result block from group 0 of version 2 is a double compaction of blocks in version 1.
To Reproduce
It is hard to accurately reproduce these steps as the timing of the compaction and cleaning operations must be very close to one another.
Expected behavior
A completed partition group does not get re-compacted.
Environment:
Infrastructure: A Kubernetes cluster running Cortex as a distributed service, with multiple compactors running concurrently