Improve scalability of compaction coordinator

[keith-turner]

Is your feature request related to a problem? Please describe.

After the changes in #6217 compactors will likely do the following actions to run a compaction

1. Find the coordinator process for their compactor resource group
2. Make an RPC to request a job from that coordinator. On the server side this will:
   1. Pull the next job from an in memory queue
   2. Perform metadata table operations to reserve the files for compaction
3. Run the compactoin
4. Make an RPC to the correct coordinator to complete the compaction. This will initiate a fate operation which involves a write to the fate table.

Describe the solution you'd like

The operation above to write to the metadata table and the fate table all happens on one coordinator for a given compaction resource group. So if a single compaction resource group has a lot of compactor processes then they will funnel all writes through that single process. One way to spread this out is to allow on any coordinator to process these writes. This would primarily be accomplished by breaking the RPC to get a job into two RPCs, one to get a job and another to reserve a job.

1. Find the coordinator process for their resource group
2. Make an RPC to request a job from that coordinator. This is quick RPC that only pulls a job out of an in memory priority queue. The job returned will not be reserved.
3. Make an RPC request to any coordinator to reserve the job. This will do the conditional metadata table writes on the server side and fail if the job can no longer be reserved.
4. Run the compaction job if it was reserved, otherwise look for a new job.
5. Make an RPC request to any coordinator to complete the job which will write to the fate table on the server side.

Additional context

The goal of this change would be to make compactions scale well as the number of compactors increases by taking the following actions as needed.

• Increasing the number of manager processes
• Increasing the number of metadata tservers and splitting the metadata table
• Increasing the number of fate tservers and splitting the fate table.

By adding an RPC this would increase the latency of a single compaction a bit, but hopefully it would allow really high throughput for the entire system.