Integrating Semaphores into Executor #31
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -78,8 +78,9 @@ def cancel(self): | |||
|
|
|||
| class TransferMeta(object): | |||
| """Holds metadata about the TransferFuture""" | |||
| def __init__(self, call_args=None): | |||
| def __init__(self, call_args=None, id=None): | |||
There was a problem hiding this comment.
This shadow's the built-in id.
There was a problem hiding this comment.
It should not be an issue because it is isolated to the class but I can update it to something more explicit.
|
I don't see any major issues, though I think @jamesls should have a look. |
|
Alright I updated the code based on feedback. Let me know what you think. |
| """ | ||
| self._semaphore = threading.Semaphore(count) | ||
|
|
||
| def acquire(self, tag, blocking=True): |
There was a problem hiding this comment.
Probably worth documenting/commenting the reason there's params that are required but not used is because we want to be API compatible with the SlidingWindowSemaphore.
|
|
Looks good to me.This makes it easier to track the submission of tasks as it relates to a specific transfer request and makes it easier for tracking requests from a subscriber.
Refactored interface for BoundedExecutors to be less general.
Adds consistency with Task and avoids confusion with shadowing builtin id.
When result() is called there is no guarantee that the semaphore is released because it is part of a done callback. As a result, submission would sometimes happen before the semaphore being released causing random failures
|
I updated the docstring. @JordonPhillips let me know if you have anymore comments. |
|
⛵ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to support the SlidingWindowSemaphore and fix #20, I decided to rip out the current implementation on how we bound the executors and replace it with a semaphore approach.
The PR builds off of: #30. You can look at that one or look at both of the PR's via this one.
The larger changes in this PR include:
cc @jamesls @JordonPhillips