Split out rate limiter per workqueue #953
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If you share a ratelimiter between workqueues, it breaks. WQ1: Starts processing item (When) WQ1: Fails to process item (When) WQ1: Fails to process item (When) WQ1: Fails to process item (When) --- At this point we've backed off a bit --- WQ2: Starts processing item (with same key, When) WQ2: Succeeds at processing item (Forget) WQ1: Fails to process item (When) ---> THIS RESULTS IN AN ERROR This results in an error because it "forgot" the previous rate limit.
|
Thanks @sargun , looks good |
|
I knew this would give us some trouble. I'd rather just remove it since we are making a change like this anyway. How useful is providing a custom implementation here? |
|
@cpuguy83 How else would you configure it outside of the VK package? |
giorio94
added a commit
to liqotech/liqo
that referenced
this pull request
May 12, 2021
This commit backports the modifications to the queues implemented upstream, in order to simplify the rate-limiting configuration. * virtual-kubelet/virtual-kubelet#937 * virtual-kubelet/virtual-kubelet#953 The custom queue implemented in virtual-kubelet/virtual-kubelet#952 is not backported, sticking to the standard workqueue, as the reason was mostly to get better introspection, which currently would not be properly leveraged in liqo.
giorio94
added a commit
to liqotech/liqo
that referenced
this pull request
May 12, 2021
This commit backports the modifications to the queues implemented upstream, in order to simplify the rate-limiting configuration. * virtual-kubelet/virtual-kubelet#937 * virtual-kubelet/virtual-kubelet#953 The custom queue implemented in virtual-kubelet/virtual-kubelet#952 is not backported, sticking to the standard workqueue, as the reason was mostly to get better introspection, which currently would not be properly leveraged in liqo.
giorio94
added a commit
to liqotech/liqo
that referenced
this pull request
May 13, 2021
This commit backports the modifications to the queues implemented upstream, in order to simplify the rate-limiting configuration. * virtual-kubelet/virtual-kubelet#937 * virtual-kubelet/virtual-kubelet#953 The custom queue implemented in virtual-kubelet/virtual-kubelet#952 is not backported, sticking to the standard workqueue, as the reason was mostly to get better introspection, which currently would not be properly leveraged in liqo.
giorio94
added a commit
to liqotech/liqo
that referenced
this pull request
May 17, 2021
This commit backports the modifications to the queues implemented upstream, in order to simplify the rate-limiting configuration. * virtual-kubelet/virtual-kubelet#937 * virtual-kubelet/virtual-kubelet#953 The custom queue implemented in virtual-kubelet/virtual-kubelet#952 is not backported, sticking to the standard workqueue, as the reason was mostly to get better introspection, which currently would not be properly leveraged in liqo.
giorio94
added a commit
to liqotech/liqo
that referenced
this pull request
May 24, 2021
This commit backports the modifications to the queues implemented upstream, in order to simplify the rate-limiting configuration. * virtual-kubelet/virtual-kubelet#937 * virtual-kubelet/virtual-kubelet#953 The custom queue implemented in virtual-kubelet/virtual-kubelet#952 is not backported, sticking to the standard workqueue, as the reason was mostly to get better introspection, which currently would not be properly leveraged in liqo.
adamjensenbot
pushed a commit
to liqotech/liqo
that referenced
this pull request
May 24, 2021
This commit backports the modifications to the queues implemented upstream, in order to simplify the rate-limiting configuration. * virtual-kubelet/virtual-kubelet#937 * virtual-kubelet/virtual-kubelet#953 The custom queue implemented in virtual-kubelet/virtual-kubelet#952 is not backported, sticking to the standard workqueue, as the reason was mostly to get better introspection, which currently would not be properly leveraged in liqo.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I would say this solves a somewhat worrying / urgent bug.
If you share a ratelimiter between workqueues, it breaks.
WQ1: Starts processing item (When)
WQ1: Fails to process item (When)
WQ1: Fails to process item (When)
WQ1: Fails to process item (When)
--- At this point we've backed off a bit ---
WQ2: Starts processing item (with same key, When)
WQ2: Succeeds at processing item (Forget)
WQ1: Fails to process item (When) ---> THIS RESULTS IN AN ERROR
This results in an error because it "forgot" the previous
rate limit.
CC: @cwdsuzhou