CompactionManager fails to find configured planner

[milleruntime]

While running CI on apache/accumulo-testing#149 when starting up Accumulo printed the below errors:

2021-08-25T10:53:15,518 [compactions.CompactionManager] ERROR: Tablet 2;466666666666666d;4000000000000006 returned non existent compaction service cs1 for compaction type SYSTEM.  Check the table compaction dispatcher configuration. Attempting to fall back to default service.
2021-08-25T10:53:15,548 [compactions.CompactionManager] ERROR: Tablet 2;466666666666666d;4000000000000006 returned non existent compaction service cs1 for compaction type SELECTOR.  Check the table compaction dispatcher configuration. Attempting to fall back to default service.
2021-08-25T10:53:15,549 [compactions.CompactionManager] ERROR: Tablet 2;466666666666666d;4000000000000006 returned non existent compaction service cs1 for compaction type USER.  Check the table compaction dispatcher configuration. Attempting to fall back to default service.
2021-08-25T10:53:15,556 [compactions.CompactionManager] ERROR: Tablet 2;466666666666666d;4000000000000006 returned non existent compaction service cs1 for compaction type CHOP.  Check the table compaction dispatcher configuration. Attempting to fall back to default service.

Almost immediately after, the CompactionService successfully creates the mentioned "cs1" that could not be found:

2021-08-25T10:53:15,583 [compactions.CompactionService] DEBUG: Created new compaction service id:cs1 rate limit:0 planner:org.apache.accumulo.core.spi.compaction.DefaultCompactionPlanner planner options:{executors=[{"name":"small","type":"internal","maxSize":"16M","numThreads":8},{"name":"medium","type":"internal","maxSize":"128M","numThreads":3},{"name":"large","type":"internal","numThreads":2}]}

[DomGarguilo]

Looks like the ComapactionService might take a while longer to be created. Not sure what the best solution might be. One possibility is adding a retry loop if this case occurs to check again for the service after a short delay.

[Manno15]

The interesting thing about this specific error is I only recall getting this on my initial test runs but I haven't been able to reproduce it since. Even when starting with a fresh cluster.

[dlmarion]

CompactionManager.checkForConfigChanges() creates new CompactionServices in the main loop, but checkForConfigChanges will only check at most every second. It might be useful to refactor checkForConfigChanges to enforce the 1 second rule from the main loop, but not if it's called from somewhere else, like compact

[dlmarion]

I put up a possible solution

[Manno15]

I will run some initial tests on it.