Fix race condition in Query Scheduler ring with frontend/worker

[trevorwhitney]

What this PR does / why we need it:

When the query frontend and/or query worker targets are enabled on the same instance as the query scheduler (which happens in both the All and Read targets), the fronted and worker require the scheduler's initialization sequence to run before their own so that the scheduler ring will be populated. The Frontend and Worker rely on this ring in the scenario where a fronted or scheduler address has not been specified, which is also usually true in single-binary (All target) or simple scalable deployment (Read target) deployments.

[slim-bean]

I'm a little hesitant on this, we talked about this in slack a little @trevorwhitney but I'd like to avoid this solution if we can better understand what's causing this, it feels like a race with services starting in parallel and I think this may not address the issue if that's the case.

Could you post the error you see when it fails? I'm trying to recreate but not having any luck so far.

[trevorwhitney]

So, the apps did not crash, but queries failed. I have a nginx in front of the read path, and it would return 502s on most queries. When I looked at the nginx logs, they report a downstream code of 499, which is a special code nginx uses for when a client closes a connection.

What I think was happening is that the scheduler ring is nil when the the worker gets initialized (which I confirmed with some printlns), so the worker gets a fronted address of localhost:9095. However, then the query scheduler ring does indeed spin up. As a result, jobs get distributed across all workers, but the workers, if I understand correctly, are only communicating back with the frontend on their own instance. So if a job was handled by a frontend on one instance, then by a worker on the other, that request will never complete and will timeout. That was my guess as to what was happening. After making this change, queries worked normally.

[owen-d]

This looks good to me.