consolidate concurrency and rate limiting critical sections #1748
Labels
complex
These issues require codebase familiarity and will need cross-cutting changes
validated
validated as an issue to be worked on
work-in-progress
Work that is in progress and partially complete
Milestone
With virtual threads, it makes more sense to launch a new virtual thread per operation, given the variety of tasks that NB must juggle for an activity. In nearly every case, there is some IO contended resource or activity. Alternately, striking a balance between native threads and virtual threads is not worthwhile at this level since it adds complexity and would be non-trivial to tune across different situations.
Given that we can start tasks very cheaply with virtual threads, the value of keeping native threads around all but disappears. The stateful task (op) tracking and sharing data can be handed to a new virtual thread just as easily as when new threads were constructed before. These structures are fixtures in the execution context already, and add no new burden to virtual thread life cycle.
This change proposes to consolidate the concurrency limit and the rate limit to a single call within SimRate, as there is already a blocking call path there which can do both much more efficiently than before.
Both of these conditions is marshaled by a semaphore. The rate limit uses the semaphore count as a leaky bucket, and the concurrency limit simply uses a semaphore in the classic sense for grants.
The concurrency interface should implement AutoCloseable. In the case that the rate limiter is inactive, as for no rate set, an alternate implementation should be substituted which no-ops the rate portion.
Dynamic changing of the rate as well as the concurrency should be supported. When the concurrency is lowered, pending operations should not be stopped, but new operations should not be started unless/until the concurrency semaphore would otherwise allow.
The amount of code that can be removed is substantial after this is supported. The recommended steps to implement are:
The text was updated successfully, but these errors were encountered: