Add ring-based service discovery to query-scheduler #2957

pracucci · 2022-09-15T09:22:03Z

What this PR does

In this PR I'm proposing to add a ring-based service discovery mechanism to let query-frontends and queriers find query-scheduler addresses. The rationale is explained in #2950 and this work would also be preparatory for scaling the backend component in the read-write deployment mode (see #2749)

How it works:

Query-scheduler addresses can be discovered in two modes: DNS (default), ring (new)
To enable ring-based service discovery just set -query-scheduler.service-discovery-mode=ring and eventually configure the ring backend (nothing to do if you use the default memberlist)
No breaking change expected

Notes to reviewers:

I haven't written many unit tests because they would require to mock almost everything (reason why we didn't have any existing unit test on DNS-based service discovery), but I've covered it through integration tests.
In the config reference doc, where we describe the scheduler address options, I've removed "If neither is set, queries are only received via HTTP endpoint." because running Mimir without query-frontend is a use case we want to discourage (in the doc it's no more mentioned as optional and we assume query-frontend is always running).

Which issue(s) this PR fixes or relates to

Fixes #2950

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

pracucci · 2022-09-16T09:23:12Z

pkg/frontend/config.go

+	if err := cfg.FrontendV2.Validate(log); err != nil {
+		return err
+	}
+	if err := cfg.QueryMiddleware.Validate(); err != nil {


Note to reviewers: previously it was directly called from pkg/mimir/mimir.go.

pracucci · 2022-09-16T09:26:00Z

pkg/mimir/modules.go

+	t.Cfg.Worker.MaxConcurrentRequests = t.Cfg.Querier.EngineConfig.MaxConcurrent
+	t.Cfg.Worker.QuerySchedulerDiscovery = t.Cfg.QueryScheduler.ServiceDiscovery


Note to reviewers: moved above to let the new utility function t.Cfg.Worker.IsFrontendOrSchedulerConfigured() to work correctly (it requires QuerySchedulerDiscovery to be set).

pkg/querier/worker/worker.go

docs/sources/operators-guide/architecture/components/query-scheduler/index.md

Signed-off-by: Marco Pracucci <marco@pracucci.com>

… or query-scheduler are in use in the querier worker initialization Signed-off-by: Marco Pracucci <marco@pracucci.com>

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci · 2022-09-19T08:40:00Z

Thanks @replay. I fixed the typos you reported.

pstibrany

Solid code! My comments are mostly doc related.

We have three interfaces which are exactly the same, and need to be in sync:

worker.serviceDiscoveryNotifications
schedulerdiscovery.Notifications
util.DNSNotifications

This is a design smell. I wonder if ringServiceDiscovery service should live next to DNSWatcher, and we only use single DNSNotifications interface?

(Unrelated question: I am also wondering if more Go-like approach would be to send notifications over channel, instead of using Listener. That way client code doesn't need to adhere to any interface, and just consume updates. But that's a change for different PR.)

development/mimir-microservices-mode/config/mimir.yaml

docs/sources/operators-guide/architecture/components/query-scheduler/index.md

pstibrany · 2022-09-19T08:02:10Z

docs/sources/operators-guide/reference-http-api/index.md

@@ -52,6 +52,7 @@ This document groups API endpoints by service. Note that the API endpoints are e
 | [Label values cardinality](#label-values-cardinality)                                 | Querier, Query-frontend        | `GET, POST <prometheus-http-prefix>/api/v1/cardinality/label_values`      |
 | [Build information](#build-information)                                               | Querier, Query-frontend, Ruler | `GET <prometheus-http-prefix>/api/v1/status/buildinfo`                    |
 | [Get tenant ingestion stats](#get-tenant-ingestion-stats)                             | Querier                        | `GET /api/v1/user_stats`                                                  |
+| [Query-scheduler ring status](#query-scheduler-ring-status)                           | Query-scheduler                | `GET /query-scheduler/ring`                                               |


Why do query-frontends and queriers not expose this ring?

This is a good feedback. As we discussed in a previous call, let me open an issue and then try to address it in a consistent way in Mimir (e.g. store-gateway ring is currently not exposed by queriers, and doc about /ingester/ring is missing to mention it's also exposed by distributors, queriers, rulers).

integration/query_frontend_test.go

pkg/frontend/v2/frontend.go

pkg/querier/worker/worker.go

pkg/scheduler/schedulerdiscovery/config.go

pkg/scheduler/scheduler.go

pkg/scheduler/schedulerdiscovery/discovery.go

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci · 2022-09-19T11:06:43Z

We have three interfaces which are exactly the same, and need to be in sync:

worker.serviceDiscoveryNotifications

schedulerdiscovery.Notifications

util.DNSNotifications

This is a design smell. I wonder if ringServiceDiscovery service should live next to DNSWatcher, and we only use single DNSNotifications interface?

The reason of this design is because of the next follow up PR. In the next PR I will introduce the "shard size" for the query-scheduler and I will need to notify active and inactive query-scheduler instances (because the querier workers will also need to open 1 connection to each inactive one, to ensure queues are drained). This means the service discovery notification interface for the query-scheduler will be a custom one, reason why I've defined schedulerdiscovery.Notifications.

What's about worker.serviceDiscoveryNotifications? The querier worker need to discover either query-frontends or query-schedulers. I could have just used schedulerdiscovery.Notifications interface, but looks queried using it for query-frontends discovery (how is query-scheduler related at all if you're not running it?) so I preferred to duplicate it given wasn't too complex.

I think this is something we can revisit in the next PR once the final logic will be in place.

(Unrelated question: I am also wondering if more Go-like approach would be to send notifications over channel, instead of using Listener. That way client code doesn't need to adhere to any interface, and just consume updates. But that's a change for different PR.)

The main reason why I kept using the previous interface is to reduce the number of changes to the minimum, and hopefully reduce the risk of introducing regressions in this PR (when using the DNS service discovery).

pstibrany

Thanks for addressing my feedback.

colega

Left a couple of nitpicks, although I see it's already merged.

colega · 2022-09-19T14:40:04Z

development/mimir-monolithic-mode/config/mimir.yaml

+query_scheduler:
+  # Comment to not use the query-scheduler and enqueue queries directly in the query-frontend.
+  service_discovery_mode: "ring"
+  ring:
+    kvstore:
+      store: consul
+      consul:
+        host: consul:8500
+


Do we expect someone to run it this way? I mean, users would need to explicitly do -target=all,query-scheduler like you did in docker-compose.yml, right? Sounds like something advanced for someone who is aiming for the simplest deployment.

Do we expect someone to run it this way?

No, we don't. It was introduced to test manually it in monolithic mode too (e.g. the were some bugs in my first implementation). If you prefer, I can revert it.

I'd prefer to remove it in future PRs, as this now doesn't test the usual scenario.

OTOH, I wonder if it would make sense to have query-scheduler enabled by default in monolith in future versions, with memberlist ring?

I'd prefer to remove it in future PRs, as this now doesn't test the usual scenario.

See: #2987

OTOH, I wonder if it would make sense to have query-scheduler enabled by default in monolith in future versions, with memberlist ring?

Yes, but the first pre-requisite would be switch to ring-based service discovery by default. We'll first get some confidence with the query-scheduler ring, then offer a migration path from DNS to ring, finally switch default to ring. Will be a long journey.

colega · 2022-09-19T15:27:21Z

pkg/querier/worker/worker.go

+// serviceDiscoveryNotifications is the interface implemented by the component receiving service discovery notifications.
+type serviceDiscoveryNotifications interface {
+	AddressAdded(address string)
+	AddressRemoved(address string)
+}


We're already importing schedulerdiscovery, so why not using schedulerdiscovery.Notifications interface here?

I see your comment about this:

I could have just used schedulerdiscovery.Notifications interface, but looks queried using it for query-frontends discovery (how is query-scheduler related at all if you're not running it?) so I preferred to duplicate it given wasn't too complex.

I think we can just reuse it.

See: #2957 (comment)

What's about worker.serviceDiscoveryNotifications? The querier worker need to discover either query-frontends or query-schedulers. I could have just used schedulerdiscovery.Notifications interface, but looks queried using it for query-frontends discovery (how is query-scheduler related at all if you're not running it?) so I preferred to duplicate it given wasn't too complex.

I think we can just reuse it.

Ok. Let's see the next PR which will change this interface, and then we can take a better decision on how to proceed on this interface.

pracucci requested review from pstibrany and colega September 15, 2022 09:22

pracucci mentioned this pull request Sep 15, 2022

Add read-write-backend deployment mode #2749

Closed

12 tasks

pracucci added the ease-of-use label Sep 15, 2022

pracucci mentioned this pull request Sep 15, 2022

Proposal: add ring-based service discovery for query-scheduler #2950

Closed

pracucci force-pushed the add-ring-based-service-discovery-to-query-scheduler branch from 5e4f678 to c6611d0 Compare September 15, 2022 15:49

pracucci marked this pull request as ready for review September 16, 2022 09:16

pracucci requested review from osg-grafana and a team as code owners September 16, 2022 09:16

pracucci commented Sep 16, 2022

View reviewed changes

pracucci force-pushed the add-ring-based-service-discovery-to-query-scheduler branch from ad68f32 to 64d4534 Compare September 16, 2022 09:28

replay reviewed Sep 16, 2022

View reviewed changes

pkg/querier/worker/worker.go Outdated Show resolved Hide resolved

replay reviewed Sep 16, 2022

View reviewed changes

docs/sources/operators-guide/architecture/components/query-scheduler/index.md Outdated Show resolved Hide resolved

replay reviewed Sep 16, 2022

View reviewed changes

docs/sources/operators-guide/architecture/components/query-scheduler/index.md Outdated Show resolved Hide resolved

pracucci added 15 commits September 19, 2022 10:30

Register query-scheduler to ring

f020b8e

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Added ring-based discovery support to query-frontend

2574d63

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Added ring-based discovery support to querier

e59c3ba

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Fixed monolithic mode

a813bfd

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Use ring-based service discovery in read-write local dev env

4a2ff94

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Removed debug prints

c9ec0ae

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Fixed unit tests

bfe47e0

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Improve ring client name

f2f5ba3

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Added missing license header

6c0cd82

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Fixed typo 'Decomment'

d292d6e

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Added config validation

502d5bf

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Improved CLI flag descriptions and fixed check whether query-frontend…

5a4126b

… or query-scheduler are in use in the querier worker initialization Signed-off-by: Marco Pracucci <marco@pracucci.com>

Added unit test on ring-based service discovery

5f1f979

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Added integration test

d46ec7e

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Added documentation

93bfd4b

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci added 2 commits September 19, 2022 10:31

Added CHANGELOG entry

ad59142

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Fixed typos in doc

8f1c586

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci force-pushed the add-ring-based-service-discovery-to-query-scheduler branch from 64d4534 to 8f1c586 Compare September 19, 2022 08:39

pstibrany reviewed Sep 19, 2022

View reviewed changes

pracucci added 4 commits September 19, 2022 11:52

Improved CLI flags description based on feedback

875a330

Signed-off-by: Marco Pracucci <marco@pracucci.com>

No need to run the ring client in the query-scheduler

bfcf265

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Removed unused field in scheduler

647d728

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Watch for service discovery errors in parent services

635c14b

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pstibrany approved these changes Sep 19, 2022

View reviewed changes

pracucci mentioned this pull request Sep 19, 2022

Standardise ring HTTP UI between Mimir components using it as client #2977

Open

pracucci merged commit 450f52c into main Sep 19, 2022

pracucci deleted the add-ring-based-service-discovery-to-query-scheduler branch September 19, 2022 12:35

pracucci mentioned this pull request Sep 19, 2022

Fix subservices watcher in query-frontend, query-scheduler and querier #2978

Merged

3 tasks

colega reviewed Sep 19, 2022

View reviewed changes

This was referenced Sep 20, 2022

Chore: do not run query-scheduler in monolithic local dev env #2987

Merged

Chore: refactor service discovery implementation #2991

Merged

Add max query-scheduler instances support #3005

Merged

pracucci mentioned this pull request Oct 5, 2022

Jsonnet: add query-scheduler ring-based service discovery support #3128

Merged

3 tasks

pracucci mentioned this pull request Nov 21, 2022

Docs: document how to do migrate query-scheduler to ring-based service discovery #3473

Closed

pracucci mentioned this pull request Dec 22, 2022

Proposal: add (optional) ring support to overrides-exporter to only export metrics from 1 replica #3806

Closed

jhalterman mentioned this pull request May 4, 2023

Add ring based discovery of alertmanagers #4928

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ring-based service discovery to query-scheduler #2957

Add ring-based service discovery to query-scheduler #2957

pracucci commented Sep 15, 2022 •

edited

Loading

pracucci Sep 16, 2022 •

edited

Loading

pracucci Sep 16, 2022 •

edited

Loading

pracucci commented Sep 19, 2022

pstibrany left a comment

pstibrany Sep 19, 2022

pracucci Sep 19, 2022

pracucci Sep 19, 2022

pracucci commented Sep 19, 2022

pstibrany left a comment

colega left a comment

colega Sep 19, 2022

pracucci Sep 19, 2022

colega Sep 19, 2022

pracucci Sep 20, 2022

colega Sep 19, 2022

colega Sep 19, 2022

pracucci Sep 19, 2022

pracucci Sep 19, 2022

		t.Cfg.Worker.MaxConcurrentRequests = t.Cfg.Querier.EngineConfig.MaxConcurrent
		t.Cfg.Worker.QuerySchedulerDiscovery = t.Cfg.QueryScheduler.ServiceDiscovery

Add ring-based service discovery to query-scheduler #2957

Add ring-based service discovery to query-scheduler #2957

Conversation

pracucci commented Sep 15, 2022 • edited Loading

What this PR does

Which issue(s) this PR fixes or relates to

Checklist

pracucci Sep 16, 2022 • edited Loading

Choose a reason for hiding this comment

pracucci Sep 16, 2022 • edited Loading

Choose a reason for hiding this comment

pracucci commented Sep 19, 2022

pstibrany left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci commented Sep 19, 2022

pstibrany left a comment

Choose a reason for hiding this comment

colega left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci commented Sep 15, 2022 •

edited

Loading

pracucci Sep 16, 2022 •

edited

Loading

pracucci Sep 16, 2022 •

edited

Loading