feat(router): limit number of jobs with the same ordering key in worker's queue

[atzoum]

Description

Until now, a single concurrency limiter existed in router's barrier, i.e. our event ordering mechanism, which was there to ensure that when a failed job got drained (aborted), in the next router generator loop only a single job would be picked up by the router for the same ordering key. This happens so as not to flood the worker's queue with jobs which are most probably going to be marked as waiting later on.

We are now introducing an additional concurrency limiter, which is responsible for limiting the overall maximum number of jobs that can enter the barrier for the same ordering key, regardless of failures. This barrier will protect us from a single "chatty" ordering key saturating the worker’s queue, so that jobs for other ordering keys can have a fair chance of being picked up by router.

The new concurrency limiter can be disabled by setting the limit to 0.

Notion Ticket

Link

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

Patch coverage: 93.75% and project coverage change: +0.06 🎉

Comparison is base (be1fef9) 51.85% compared to head (0886083) 51.91%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3243      +/-   ##
==========================================
+ Coverage   51.85%   51.91%   +0.06%     
==========================================
  Files         322      322              
  Lines       53381    53411      +30     
==========================================
+ Hits        27682    27730      +48     
+ Misses      24042    24028      -14     
+ Partials     1657     1653       -4     

Impacted Files	Coverage Δ
router/internal/eventorder/eventorder.go	96.13% <92.68%> (-0.30%)	⬇️
router/router.go	76.12% <100.00%> (+0.08%)	⬆️
...-forwarder/internal/forwarder/abortingforwarder.go	74.50% <100.00%> (ø)
warehouse/internal/repo/staging.go	81.11% <100.00%> (+0.18%)	⬆️

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.