Singularity performance improvements #1702
This is a first pass at updates to the scheduler internals to improve the speed of offer processing, status updates, and other processes that contend for the scheduler lock. The main theme here is parallelizing things that were previously sequential.
The biggest change in this PR that still needs additional testing is the removal of the global scheduler lock in favor of many smaller locks. There are now three classes of locks. One lock for the scheduler state, one for accessing/processing offers, and one ConcurrentHashMap of request locks such that only one process can be updating the state or tasks for a request at a time. This change lets offers, status updates, deploy checks, pending request processes, and more run in parallel.
Other changes include:
The request level locking along with our proxy to leader code opens us up to some better UI experiences in the future as well. Instead of waiting for pollers, we can more easily execute many actions like bounces or task kills at request time and immediately return results to the UI.
Further TODOs include finishing off fixing tests, running additional benchmarks of the new code, and cleaning up some rough edges (like hard coded values that should instead be configurable)
…ests to be synchronous