Rework WorkflowExecutorCache to remove a global lock

[Spikhalskiy]

Why

WorkflowExecutorCache has a global (for all workflow task processors) lock that especially comes into play when forced evictions are needed from the cache.
The only reason for this lock is to update the cache and inProcessing collection in a thread-safe manner.
inProcessing in the cache is not needed, as it duplicates the functionality and the state of WorkflowRunLockManager that maintains all RunIds currently in-flight on the worker and does it in an effective manner for a highly concurrent environment.

What

This PR removes the global lock and makes the cache to use the effective WorkflowRunLockManager as a source of truth of in-flight RunIds

[Spikhalskiy]

Bypassing review requirements to put this PR to test. Code Review will be done retroactively and comments will be addressed in a separate PR before the next release of Java SDK.

[cretz]

LGTM

[cretz]

I know not changed in this PR, but would change to type ConcurrentMap for documentation/expectation purposes. Confused me at first read.

In fact, after review, maybe the documented type for this should be ConcurrentHashMap or at least document the expectations around compute, because Map.compute isn't thread-safe obviously and ConcurrentMap.compute can run the closure multiple times (which is bad for us because we have a side-effecting closure there). But ConcurrentHashMap.compute seems to guarantee a single execution.

[Sushisource]

Makes sense, Core has per-run locks.

It's a much bigger change, but, the bit of code here about multiple WFTs for the same Run ID serializing application by waiting on the lock could likely be simplified by stuffing them into a per-run queue which is kinda how core operates (it's streams/channels, but same idea).

That's a bigger change obviously, food for thought.