Configurable brokerService for console queries

[aho135]

Description

When using the manual routing strategy in combination with druid.broker.segment.watchedTiers we are losing console visibility for Historicals. This can happen with the combination of configurations

Router

druid.router.sql.enable: true
druid.router.strategies: [{"type":"manual","defaultManualBrokerService":"druid/broker1"}]
druid.router.tierToBrokerMap: {"_default_tier": "druid/broker1", "broker2": "druid/broker2"}

broker1

druid.broker.segment.watchedTiers=["t1","t2"]

broker2

druid.broker.segment.watchedTiers=["t1","t2","t3","t4"]

Since the Router query goes to druid/broker1 which has limited segment visibility, the console will incorrectly report Historicals in t3/t4 as having no data. This patch allows us to configure Router console queries to go to druid/broker2 instead so that the console can have full visibility over Historicals.

Release note

Allows operators to configure brokerService for Router console queries. This can be accomplished by setting something like

consoleBrokerService: 'druid/broker2',

in web-console/console-config.js

---

Key changed/added classes in this PR

• druid-query.ts
• entry.tsx

---

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• been tested in a test Druid cluster.

[abhishek-chouhan]

There is also the case where broker1 watches ["t1","t2"] and broker2 watches ["t3","t4"]. In this case, to get a complete view both need to be queried. Would this case work as well?

[aho135]

There is also the case where broker1 watches ["t1","t2"] and broker2 watches ["t3","t4"]. In this case, to get a complete view both need to be queried. Would this case work as well?

Thanks for the review @abhishek-chouhan! This case is not covered. This would be more invasive as it requires merging BrokerServerView from broker1 and broker2. In this scenario I think it would be simpler to spin up an additional broker (e.g. druid/broker-admin) which watches all tiers and configure consoleBrokerService to point there.

[abhishekrb19]

What do you think about exposing this config via web-console-config.tsx as well? It has this dialog on the console, which seems cool:

I think web-console-config.tsx is session-based that stores config overrides in the browser, while the ConsoleConfig from console-config.js is persistent via the build. Not super familiar with the web-console code here, so correct me if I'm missing something here.

Can console-config.js be overridden per environment, given that this file is baked into the build as part of the source code?

[abhishekrb19]

Is this used anywhere?

[abhishekrb19]

I see that https://github.com/apache/druid/blob/master/web-console/src/views/explore-view/explore-view.tsx#L64
directly calls the /druid/v2/sql endpoint without invoking queryDruidSql. Should we update it to use this function instead, so the context is appended for the Explore view as well?

[abhishek-chouhan]

Thanks for the review @abhishek-chouhan! This case is not covered. This would be more invasive as it requires merging BrokerServerView from broker1 and broker2. In this scenario I think it would be simpler to spin up an additional broker (e.g. druid/broker-admin) which watches all tiers and configure consoleBrokerService to point there.

I do think we need the merging functionality to support the watchedTiers functionality in its entirety. While the workaround with a separate broker works, its too heavy of a workaround just to make the console work with watchedTiers.
This change is still valuable, we can take up the merging brokerServerView change as a followup enhancement.

[abhishekrb19]

Since brokerService is part of the query context, would setting baseQueryContext or defaultQueryContext in console-config.js work out of the box?