Reuse catalyst stream pull node if pull is locked

[leszko]

This is a fix for the following scenario that happens from time to time on prod:

1. We get /pull request which is successful
2. A few seconds after (without getting /terminate) we receive another /pull request for the same stream
3. If it's redirected to a different catalyst node, then the pull fails, because we lock stream pull to one node

The workaround for this is to check if the given stream is currently locked (read: it's being pulled atm) and if yes, then trigger pulling from the same catalyst node. If we redirect to the same node, then Mist handles this.

Sample logs: https://eu-metrics-monitoring.livepeer.live/grafana/goto/wDCuqDYSg?orgId=1

fix https://linear.app/livepeer/issue/PS-518/more-pull-failuress

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
livepeer-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 7, 2024 2:13pm

[victorges]

LGTM

Such race condition. I'd imagine that the second /pull request would simply be redirected to the node currently pulling it. Does this only happen when the second /pull arrives after the stream started but before the load balancer gets its state to redirect to the right node?

[leszko]

Such race condition. I'd imagine that the second /pull request would simply be redirected to the node currently pulling it. Does this only happen when the second /pull arrives after the stream started but before the load balancer gets its state to redirect to the right node?

Yeah, from our perspective it happens, when Load Balancer redirects to a different node AND the stream is not available (so, Mist load balancer does not see it active). I believe that for Trovo it happens when a streamer starts streaming, stops streaming, and then suddenly starts again (maybe some retry from OBS). So, Trovo does not send /terminate properly.

[linear]

PS-518 more pull failuress

[victorges]

LGTM either way

[emranemran]

How do we test this to ensure no regressions?

[leszko]

How do we test this to ensure no regressions?

I think the same way as with other changes => Test pull stream in staging (with the stream stopped and started). And then monitor the prod. Any other thoughts how we could test it?