Singularity performance improvements

[ssalinas]

This is a first pass at updates to the scheduler internals to improve the speed of offer processing, status updates, and other processes that contend for the scheduler lock. The main theme here is parallelizing things that were previously sequential.

The biggest change in this PR that still needs additional testing is the removal of the global scheduler lock in favor of many smaller locks. There are now three classes of locks. One lock for the scheduler state, one for accessing/processing offers, and one ConcurrentHashMap of request locks such that only one process can be updating the state or tasks for a request at a time. This change lets offers, status updates, deploy checks, pending request processes, and more run in parallel.

Other changes include:

• More async handling of status updates. Instead of auto-acking a status update when we receive it, then processing it before grabbing the next, we will kick off an async process for each status update, with concurrency being limited by the AsyncSemaphore class (copied here in the short term until we open source the library)
• Parallelize many processes in offer processing and remove the extra while loop. Base on the first pass over the logic it seemed we were doing many more loops than needed through certain sections of offer processing. This still needs more testing to confirm there is no regression
• parallelize pending request and deploy check processes now that we are locking at a request level
• remove the task credit system. This was originally for debugging. But, given the disabled actions capabilities the extra checks are unneeded overhead

The request level locking along with our proxy to leader code opens us up to some better UI experiences in the future as well. Instead of waiting for pollers, we can more easily execute many actions like bounces or task kills at request time and immediately return results to the UI.

Further TODOs include finishing off fixing tests, running additional benchmarks of the new code, and cleaning up some rough edges (like hard coded values that should instead be configurable)

@darcatron @baconmania @pschoenfelder

[ssalinas]

Additional improvements:

• Calculate more slave usage scores up front instead of recalculating on each pending task loop so that the number of calculations there is liner with number of offers not with numOffers*numPendingTasks
• Also consider cached offers when resourceOffers is called, not only in the scheduler poller

Test updates:

• Most tests rely on there being no offer cache in place. I now bind the SingularityNoOfferCache by default except for a specific test class which leverages the offer cache to test it

[baconmania]

Minor typo

[baconmania]

Does it make sense to use callWithRequestLock() here as well, and then make SingularitySchedulerLock#(lock|unlock) private?

[ssalinas]

yes, I think I had a different exception in here at one point and didn't refactor back to callWithRequestLock, thanks for finding that

[baconmania]

Could it be a good idea to accept the calling class here instead of a name, and then log out the class name in the trace logs below? Just to ensure that the names we pass when locking aren't arbitrary and are useful for tracing the flow of the scheduler.

[ssalinas]

maybe even going as far as class#methodName could be useful for that logging

[baconmania]

Should be safe to make lock() and unlock() private now, and only allow the rest of the scheduler to interact via runWithRequestLock().

[baconmania]

🚢