fix: limiter deadlock while trying to notify a dynamic priority item

[atzoum]

Description

In case of high contention, a deadlock was possible if the limiter was trying to notify a dynamic priority item which in turn was trying to increase its priority in the queue.
The limiter now releases the lock earlier, just before notifying the item's channel. The (removed) item will no longer block trying to increase its priority in the queue, it will execute this no-op and then get properly notified.

Notion Ticket

Link

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (release/1.6.x@23f220c). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files

@@               Coverage Diff                @@
##             release/1.6.x    #3056   +/-   ##
================================================
  Coverage                 ?   52.86%           
================================================
  Files                    ?      338           
  Lines                    ?    52135           
  Branches                 ?        0           
================================================
  Hits                     ?    27562           
  Misses                   ?    22967           
  Partials                 ?     1606           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[lvrach]

Can you include parts of the trace that indicate the problem? It would be easier to understand the problem this way

[lvrach]

Can you add a benchmark test that simulates a case of high contention?

[atzoum]

Can you add a benchmark test that simulates a case of high contention?

There is a benchmark test already here, but didn't really managed to manifest the issue :(
The same test, a lighter version of it actually, runs during ci as well

[atzoum]

Can you include parts of the trace that indicate the problem? It would be easier to understand the problem this way

1. Limiter is trying to notify the next goroutine's channel here (has acquired the lock)
2. The goroutine that should be listening to this channel is instead trying to acquire a lock here in order to bump its priority
3. Deadlock!