Proposal: add ring-based service discovery for query-scheduler

[pracucci]

Currently, when running Mimir with the query-scheduler, the query-scheduler address needs to be specified in two configs, to let query-frontend and querier discover query-scheduler instances:

• -querier.scheduler-address
• -query-frontend.scheduler-address

DNS-based service discovery is easy if you have an easy way to define a DNS entry (e.g. Kubernetes service), while it could be complicated otherwise. In #2012 I proposed to add support for multiple DNS addresses to overcome some limitations.

In this issue I'm doing a different proposal: add ring-based service discovery for query-scheduler. Following the same pattern used for other Mimir components, the idea is to build a ring for the query-scheduler and use it in query-frontend and querier to discover query-scheduler instances. This would also be preparatory to allow to run query-scheduler on a subset of backend replicas when running the experimental read-write-backend deployment mode (see #2749).

The proposal is to:

1. Add ring-based service discovery for query-scheduler as experimental, along side the existing scheduler-address
2. Gain enough confidence with the ring-based service discovery
3. Provide a migration path to the ring-based service discovery and then deprecate the scheduler-address

Superseeds #2012.

[pracucci]

I'm working on a implementation to show how it would work (PR). Action items:

• Register query-scheduler to its own ring
  • Inject config required for memberlist
  • Add config option to configure the type of service discovery to use and ensure frontend and scheduler addresses are not set when using the ring one
  • Validate the query-frontend and querier config: if query-scheduler ring service discovery mode then there should be no scheduler address configured
  • Improve *.scheduler-address and *.frontend-address CLI flag descriptions
• Auto-forget
• UI
• Add ring-based service discovery support to query-frontend and querier
• Unit tests
  • Ring-based service discovery
• Integration test
• Manual tests
  • Microservices mode
  • Monolithic mode
  • Read-write-backend mode
  • Test rollouts in a dev env
• Documentation
  • List among experimental features
  • HTTP API
  • Document how it works
• CHANGELOG