Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Picker poisoning when try to call Listener() simultaneously with PickAsync() #1

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

kolonist
Copy link
Owner

@kolonist kolonist commented Apr 3, 2024

This test shows situation when Picker become poisoned and cannot return valid pick results.

  1. If picker user often enough there could be case when called ConnectionManager.PickAsync(...), it started to execute and it calls await ConnectionManager.GetPickerAsync(...).
  2. GetPickerAsync(...) finishes execution, returns _nextPickerTcs.Task.WaitAsync(...) which then awaited inside PickAsync(...).
  3. Then before awaited task completed we have new balancer addresses (e.g. from service discovery) and we call PollingResolver.Listener(...) with new addresses. If addresses are really new then new subchannels will be created and at least one of them can get TransientFailure when trying to connect.
  4. And if all of these cases happened (I mean when Listener(...) with new addresses called just after GetPickerAsync(...) returns but before returned task completed and one of sunchannels has TransientFailure state) then Picker become poisoned forever. It will always return ErrorPicker and currentPicker.Pick(...).Type will be PickResultType.Fail.

For this situation to happen there should be client side balancing enabled with custom resolver, ConnectionManager.PickAsync(...) calls should be often, connections should sometimes be broken. Even if it seems to be rare case nevertheless it happened often enough under high load and intensive use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant