CCM-16073 - Updated rate limiting behaviour#158
Merged
rhyscoxnhs merged 6 commits intofeature/CCM-16073from Apr 29, 2026
Merged
CCM-16073 - Updated rate limiting behaviour#158rhyscoxnhs merged 6 commits intofeature/CCM-16073from
rhyscoxnhs merged 6 commits intofeature/CCM-16073from
Conversation
rhyscoxnhs
force-pushed
the
feature/CCM-16073-rate-limit
branch
from
a136f1f to
38d0fad
Compare
mjewildnhs
reviewed
Apr 23, 2026
Contributor
mjewildnhs
left a comment
mjewildnhs
left a comment
There was a problem hiding this comment.
As far as reviewed the admin script
|
|
||
| return { 1, "allowed", 0, effectiveRate } | ||
| local reason = consumedTokens < 1 and "rate_limited" or "allowed" | ||
| local retryAfter = consumedTokens < 1 and 1000 or 0 |
Contributor
There was a problem hiding this comment.
Do we want to optimise the retry time rather than hardcode to 1s?
We could have it ramp down to 1s based on the period in the recovery.
With the defaults we'll be recovering for 10m with a reduced effectiveRate.
With lower invocationRateLimits (e.g. 10/s) it will take 60s to generate a token.
So there is no point retrying so quickly as there won't be any tokens causing unnecessary spin on the lambda.
Instead we could use the time it takes to generate a token math.ceil(1000 / effectiveRate).
| const [consumedOrFlag, reason, retryAfterMs, effectiveRate] = raw; | ||
|
|
||
| if (allowed === 1) { | ||
| if (reason === "allowed" || reason === "probe") { |
Contributor
There was a problem hiding this comment.
The reason "probe" isn't returned anymore.
Comment on lines
+44
to
61
| local isHalfOpen = isOpen and now > switchedAt + cooldownMs | ||
| local isRecovering = (not isOpen) and now < switchedAt + recoveryPeriodMs | ||
|
|
||
| -- Circuit is open and no probe slot is available — reject | ||
| return { 0, "circuit_open", openedUntil - now, 0 } | ||
| end | ||
|
|
||
| -------------------------------------------------------------------------------- | ||
| -- 2. SLIDING WINDOW | ||
| -- | ||
| -- Two windows (current + previous) together approximate a sliding window over | ||
| -- cbWindowPeriodMs. When the current window expires it is promoted to previous | ||
| -- and a fresh current window starts. record-result.lua blends the two windows | ||
| -- using a time-based weight to smooth the error rate across the boundary rather | ||
| -- than resetting it to zero at expiry. | ||
| -- | ||
| -- record-result.lua is responsible for incrementing the counters; this script | ||
| -- is only responsible for rolling the window boundary forward when it expires. | ||
| -------------------------------------------------------------------------------- | ||
| local effectiveRate | ||
|
|
||
| if cbWindowFrom == 0 then | ||
| -- No window exists yet — start one now | ||
| cbWindowFrom = now | ||
| elseif (now - cbWindowFrom) > cbWindowPeriodMs then | ||
| -- Current window has expired — roll it forward | ||
| if (now - cbWindowFrom) > (2 * cbWindowPeriodMs) then | ||
| -- Both current and previous windows are stale: a long quiet period means | ||
| -- old failure counts are no longer relevant to the health of the endpoint. | ||
| cbPrevFailures = 0 | ||
| cbPrevAttempts = 0 | ||
| if isOpen then | ||
| if isHalfOpen then | ||
| effectiveRate = probeRateLimit | ||
| else | ||
| -- Promote current → previous so it can be blended with the new current window | ||
| cbPrevFailures = cbFailures | ||
| cbPrevAttempts = cbAttempts | ||
| return { 0, "circuit_open", (switchedAt + cooldownMs) - now, 0 } | ||
| end | ||
| else | ||
| if isRecovering then | ||
| effectiveRate = targetRateLimit * (now - switchedAt) / recoveryPeriodMs | ||
| else | ||
| effectiveRate = targetRateLimit | ||
| end | ||
| cbFailures = 0 | ||
| cbAttempts = 0 | ||
| cbWindowFrom = now | ||
| end |
Contributor
There was a problem hiding this comment.
Declaring the isHalfOpen / isRecovering where they are used tidies up and simplifies it.
Further to that I think flipping the isHalfOpen check and labelling it inCooldown makes it simpler, brings the cooldownMs into context and moves away from half open terminology which i think is a bit problematic.
Suggested change
| local isHalfOpen = isOpen and now > switchedAt + cooldownMs | |
| local isRecovering = (not isOpen) and now < switchedAt + recoveryPeriodMs | |
| -- Circuit is open and no probe slot is available — reject | |
| return { 0, "circuit_open", openedUntil - now, 0 } | |
| end | |
| -------------------------------------------------------------------------------- | |
| -- 2. SLIDING WINDOW | |
| -- | |
| -- Two windows (current + previous) together approximate a sliding window over | |
| -- cbWindowPeriodMs. When the current window expires it is promoted to previous | |
| -- and a fresh current window starts. record-result.lua blends the two windows | |
| -- using a time-based weight to smooth the error rate across the boundary rather | |
| -- than resetting it to zero at expiry. | |
| -- | |
| -- record-result.lua is responsible for incrementing the counters; this script | |
| -- is only responsible for rolling the window boundary forward when it expires. | |
| -------------------------------------------------------------------------------- | |
| local effectiveRate | |
| if cbWindowFrom == 0 then | |
| -- No window exists yet — start one now | |
| cbWindowFrom = now | |
| elseif (now - cbWindowFrom) > cbWindowPeriodMs then | |
| -- Current window has expired — roll it forward | |
| if (now - cbWindowFrom) > (2 * cbWindowPeriodMs) then | |
| -- Both current and previous windows are stale: a long quiet period means | |
| -- old failure counts are no longer relevant to the health of the endpoint. | |
| cbPrevFailures = 0 | |
| cbPrevAttempts = 0 | |
| if isOpen then | |
| if isHalfOpen then | |
| effectiveRate = probeRateLimit | |
| else | |
| -- Promote current → previous so it can be blended with the new current window | |
| cbPrevFailures = cbFailures | |
| cbPrevAttempts = cbAttempts | |
| return { 0, "circuit_open", (switchedAt + cooldownMs) - now, 0 } | |
| end | |
| else | |
| if isRecovering then | |
| effectiveRate = targetRateLimit * (now - switchedAt) / recoveryPeriodMs | |
| else | |
| effectiveRate = targetRateLimit | |
| end | |
| cbFailures = 0 | |
| cbAttempts = 0 | |
| cbWindowFrom = now | |
| end | |
| f isOpen then | |
| local inCooldown = now <= switchedAt + cooldownMs | |
| if inCooldown then | |
| return { 0, "circuit_open", (switchedAt + cooldownMs) - now, 0 } | |
| end | |
| effectiveRate = probeRateLimit | |
| else | |
| local isRecovering = now < switchedAt + recoveryPeriodMs | |
| if isRecovering then | |
| effectiveRate = targetRateLimit * (now - switchedAt) / recoveryPeriodMs | |
| else | |
| effectiveRate = targetRateLimit | |
| end | |
| end |
rhyscoxnhs
force-pushed
the
feature/CCM-16073-rate-limit
branch
from
90a5096 to
26d8d59
Compare
rhyscoxnhs
force-pushed
the
feature/CCM-16073-rate-limit
branch
from
6259edd to
fe2e5f8
Compare
rhyscoxnhs
force-pushed
the
feature/CCM-16073-rate-limit
branch
from
fe2e5f8 to
9856bc9
Compare
mjewildnhs
reviewed
Apr 27, 2026
rhyscoxnhs
force-pushed
the
feature/CCM-16073-rate-limit
branch
from
8fb8f8c to
5cb24f9
Compare
mjewildnhs
force-pushed
the
feature/CCM-16073-rate-limit
branch
from
e3fb631 to
5cb24f9
Compare
mjewildnhs
reviewed
Apr 27, 2026
rhyscoxnhs
force-pushed
the
feature/CCM-16073-rate-limit
branch
from
5cb24f9 to
07410f7
Compare
cgitim
reviewed
Apr 28, 2026
cgitim
reviewed
Apr 28, 2026
* Rate limit/circuit breaker fixes and logging improvements Additional handler logging and observability Fix retry time for partial batch rate limiting Add follow option to debug test script Add since var to int debug script High resolution storage metrics * Warm up in the circruit breaker test to ensure circuit is closed * Circuit breaker disabled fixes * Fix perf lambda DLQ purge * Add flush and debug ability to perf runner lambda * Up the burst in rate limit test --------- Co-authored-by: Tim Marston <tim.marston2@nhs.net>
mjewildnhs
added a commit
that referenced
this pull request
Apr 29, 2026
* CCM-16073 - Updated rate limiting behaviour Co-authored-by: Mike Wild <mike.wild5@nhs.net> Co-authored-by: Tim Marston <tim.marston2@nhs.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Context
Type of changes
Checklist
Sensitive Information Declaration
To ensure the utmost confidentiality and protect your and others privacy, we kindly ask you to NOT including PII (Personal Identifiable Information) / PID (Personal Identifiable Data) or any other sensitive data in this PR (Pull Request) and the codebase changes. We will remove any PR that do contain any sensitive information. We really appreciate your cooperation in this matter.