Loki: query scheduler should send shutdown to frontends when ReplicationSet changes

[slim-bean]

What this PR does / why we need it:

This PR attempts to improve a race when adding/removing instances into the scheduler ring where messages could end up stuck in a schedulers queue with no queriers coming back to process them.

Currently when an elected scheduler is no longer "elected" everyone finds out about this through the ring, but the schedulers themselves do nothing and technically stay active, the frontend and queriers poll the ring and when they find out about the scheduler change, disconnect from the old and connect to the new.

The distributed nature of this creates the possibility that all the queriers find out about a scheduler change and disconnect from the scheduler while the frontend still stays connected. The frontend could then be stuck waiting for inflight queries to process (which will never happen because all the queriers have moved on) ultimately leading to a timeout or failure of the query.

To mitigate this we make sure that as soon as the scheduler knows it's not in the ReplicationSet of who should act as a scheduler it should send a shutdown message to connected frontends so they will cancel requests and retry them in other schedulers.

Special notes for your reviewer:

Checklist

• Documentation added
• Tests updated
• Add an entry in the CHANGELOG.md about the changes.

[owen-d]

I've added a suggestion, but I think we can alternatively just reuse the NewRingWatcher in the scheduler rather than implementing similar functionality.