Simplify ConcurrentPool

[stIncMale]

This PR mostly replaces ReentrantReadWriteLock in ConcurrentPool.StateAndPermits.lock with ReentrantLock, and simplifies StateAndPermits.throwIfClosedOrPaused.

Originally throwIfClosedOrPaused was called by threads that do not necessary held an exclusive lock, so the method had to make sure it cannot be executed concurrently with the code that modifies the StateAndPermits.paused field (we don't have the same care about the closed field because unlike paused, it is flipped at most once). However, in order to allow throwIfClosedOrPaused to be executed concurrently with itself, it acquired a shared lock.

Then I noticed that each time throwIfClosedOrPaused is called by a thread that does not hold an exclusive lock, the thread then calls another method of StateAndPermits that calls throwIfClosedOrPaused again. This observation allowed to simplify the code, because many original throwIfClosedOrPaused invocations were unnecessary.

[rozza]

It looks neat my simpler, I have a couple of questions:

Then I noticed that each time throwIfClosedOrPaused is called by a thread that does not hold an exclusive lock, the thread then calls another method of StateAndPermits that calls throwIfClosedOrPaused again. This observation allowed to simplify the code, because many original throwIfClosedOrPaused invocations were unnecessary.

I'm not sure I fully understand this scenario - can you elaborate.

Does using a single lock rather than a read / write lock have any performance impact.

[stIncMale]

Does using a single lock rather than a read / write lock have any performance impact.

I guess one may say that it generally does have a negative effect. However, in our case we simply had the method throwIfClosedOrPaused called in multiple places without holding the write lock, and then (in the program order) we called it again while holding the write lock. Clearly, one of such calls that were done one after the other was redundant, and I removed the one that was done without holding the write lock. This reduced the amount of work the pool does, which means that theoretically it improved its performance, though I highly doubt the change is noticeable given the rest of the work. The actual benefit of the change is that the code became simpler by loosing useless pieces.

Ross, does the above clarifies the change for you?

[rozza]

Thanks for the clarifications - LGTM!