1070 Add discard config for rate-limiter #1096
Conversation
- Meant for Manually Fetching Jobs pattern
| }); | ||
|
|
||
| it('should put a job into the delayed queue when limit is hit', function() { | ||
| queue.on('failed', function(e) { |
There was a problem hiding this comment.
Might be better to create a new queue here as you did in the test below, since this test assumes that the queue max is set to 1.
| }) | ||
| ).then(function() { | ||
| return newQueue.getDelayedCount().then(function(delayedCount) { | ||
| expect(delayedCount).to.eq(0); |
There was a problem hiding this comment.
Should also do an assert on the main queue to make sure 3 objects are there. Might also want to assert the ordering of the jobs to make sure that is untouched?
lib/commands/moveToActive-8.lua
Outdated
| jobCounter = tonumber(rcall("GET", KEYS[6])) | ||
| if jobCounter ~= nil and jobCounter >= maxJobs then | ||
| if discard == 'true' then |
There was a problem hiding this comment.
Are there any tests for these .lua scripts? If so, we should update them for this case.
There was a problem hiding this comment.
there are not test only for the .lua scripts, but integration tests that indirectly call this .lua scripts.
PATTERNS.md
Outdated
|
|
||
| Patterns | ||
| ======== | ||
| # Patterns |
There was a problem hiding this comment.
Do we need all these formatting changes in this PR? :) I would remove if unnecessary, its cluttering things up and distracting from the feature-related changes
There was a problem hiding this comment.
there are great changes though, would be nice to have them in a different PR, but if to cumbersome we could accept it in this one.
There was a problem hiding this comment.
will make another pr
There was a problem hiding this comment.
https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint was used and automatically formatted it for me on save
REFERENCE.md
Outdated
| duration: number, // per duration in milliseconds | ||
| max: number; // Max number of jobs processed | ||
| duration: number; // per duration in milliseconds | ||
| discard: boolean = false; // When jobs get rate limited, nothing happens |
There was a problem hiding this comment.
discard might not be the best name for this.. because we're not entirely discarding the rate limiter, we are just disabling the delayed queue. So maybe disableDelyedQueue or something similar? And then change the comment to say something more explicit like When set to true and jobs get rate limited, they stay in the main queue and are not moved to the delayed queue
There was a problem hiding this comment.
ok, now I full understand the use-case and discard may indeed not be the best name. I thought the idea was to really throw away the jobs that get "throttled". so this name will be misleading. On the other hand, a name like disableDelayedQeuue is a bit techie and exposes implementation details.
There was a problem hiding this comment.
Not sure about this name but what about this alternative bounceBack ? the option is quite special since it only has use in the manual processing pattern...
There was a problem hiding this comment.
I like bounceBack 👍
PATTERNS.md
Outdated
|
|
||
| Patterns | ||
| ======== | ||
| # Patterns |
There was a problem hiding this comment.
there are great changes though, would be nice to have them in a different PR, but if to cumbersome we could accept it in this one.
lib/commands/moveToActive-8.lua
Outdated
| jobCounter = tonumber(rcall("GET", KEYS[6])) | ||
| if jobCounter ~= nil and jobCounter >= maxJobs then | ||
| if discard == 'true' then |
There was a problem hiding this comment.
there are not test only for the .lua scripts, but integration tests that indirectly call this .lua scripts.
test/test_rate_limiter.js
Outdated
| assert.fail(e); | ||
| }); | ||
|
|
||
| return Promise.all( |
There was a problem hiding this comment.
maybe it is just better to write this just as:
Promise.all([queue.add({}), queue.add({}), queue.add({}), queue.add({})])
REFERENCE.md
Outdated
| duration: number, // per duration in milliseconds | ||
| max: number; // Max number of jobs processed | ||
| duration: number; // per duration in milliseconds | ||
| discard: boolean = false; // When jobs get rate limited, nothing happens |
There was a problem hiding this comment.
ok, now I full understand the use-case and discard may indeed not be the best name. I thought the idea was to really throw away the jobs that get "throttled". so this name will be misleading. On the other hand, a name like disableDelayedQeuue is a bit techie and exposes implementation details.
REFERENCE.md
Outdated
| duration: number, // per duration in milliseconds | ||
| max: number; // Max number of jobs processed | ||
| duration: number; // per duration in milliseconds | ||
| discard: boolean = false; // When jobs get rate limited, nothing happens |
There was a problem hiding this comment.
Not sure about this name but what about this alternative bounceBack ? the option is quite special since it only has use in the manual processing pattern...
|
@manast new review is up 👍 |
Meant for Manually Fetching Jobs pattern
closes Config option for rate-limiter to do nothing when enabled and hit #1070