feat: gateway back pressure mechanism implementation

[saurav-malani]

Description of the change

To avoid OOM kill of gateway in scenarios where we have a lot of sudden spike in gateway traffic or when DB is under pressure. We have introduced a back pressure mechanism that restrict on maximum number of active request in memory, by using counting Semaphore.

Notion Link

Notion Link

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added unit tests for the code
• I have made corresponding changes to the documentation

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

Merging #1847 (63b88b4) into master (24252a0) will increase coverage by 0.00%.
The diff coverage is 9.67%.

@@           Coverage Diff           @@
##           master    #1847   +/-   ##
=======================================
  Coverage   25.54%   25.54%           
=======================================
  Files         116      116           
  Lines       28738    28736    -2     
=======================================
  Hits         7340     7340           
+ Misses      21056    21054    -2     
  Partials      342      342           

Impacted Files	Coverage Δ
gateway/gateway.go	50.68% <3.44%> (+0.27%)	⬆️
gateway/configuration.go	96.36% <100.00%> (+0.13%)	⬆️
...ebugger/destination/eventDeliveryStatusUploader.go	90.12% <0.00%> (-3.71%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 24252a0...63b88b4. Read the comment docs.

[lvrach]

removed the Ready for Review, until changes are addressed/pushed

[lvrach]

[minor] and subjective. I would prefer to see expected responses here, instead of computing them in:

require.Equal(t, tt.totalReq-tt.maxConcurrentRequests, resp503, "actual 503 resp different than expected.")

[saurav-malani]

the reason I have written it like this is, to make it more intuitive that anything over allowed concurrentRequests would be 503. tt.totalReq-tt.maxConcurrentRequests mathematically indicates that. @lvrach WDYT?

[atzoum]

LGTM

[fracasula]

Looks fairly good, I've added very few comments, the blocking one is the one about doing atomic.LoadInt32(&concurrentRequests). Let me know if it doesn't make sense of course.

[fracasula]

Up to you guys, I don't think it should be a blocker but definitely a nice to have, especially if easy enough to setup.

…

On Thu, 28 Apr 2022, 10:55 Aris Tzoumas, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In middleware/middleware.go <#1847 (comment)> : > + +func MaxConcurrentRequestsMiddleware(maxConcurrentRequests int) func(http.Handler) http.Handler { + + var requests = make(chan struct{}, maxConcurrentRequests) + return func(next http.Handler) http.Handler { + //this is to make sure that we don't have more than `maxClient` in-memory at any point of time. As, having more http client than `maxClient` + // may lead to gateway OOM kill. + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if maxConcurrentRequests != 0 { + select { + case requests <- struct{}{}: + defer func() { + <-requests + }() + default: + http.Error(w, http.StatusText(http.StatusServiceUnavailable), http.StatusServiceUnavailable) @fracasula <https://github.com/fracasula> we will selectively enable the limit on specific customer(s), by default it is disabled. IMO the existing metric, number of active connections should be enough for monitoring & troubleshooting at this stage. — Reply to this email directly, view it on GitHub <#1847 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2QQABWA3YEEXPAKROADLVHJHBFANCNFSM5TKYXPSQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>