Skip to content

Conversation

@shauryachats
Copy link
Collaborator

@shauryachats shauryachats commented Dec 29, 2025

Summary

This PR adds multi-cluster routing support for SSE queries by introducing a new query option enableMultiClusterRouting in the SQL query.
The PR preserves existing behavior when multi-cluster routing is disabled.

Changes

  • Plumbs MultiClusterRoutingContext through all broker request handlers to support federated routing.
  • Adds a query option, enableMultiClusterRouting, to select the federated RoutingManager at query time.
  • Updates single-stage broker routing to dynamically choose local vs. multi-cluster routing based on query options and logical table usage.
  • Enhances logical table routing to resolve table metadata from local and federated table caches.
  • Relaxes logical table validation for cross-cluster physical tables using a new PhysicalTableConfig.multiCluster flag.

Testing

Added a multi-cluster integration test that validates logical table federation across two clusters and verifies aggregated results with multi-cluster routing enabled.

: _routingManager;
// Get the tables hit by the request
TableRouteInfo routeInfo = routeProvider.getTableRouteInfo(tableName, _tableCache, _routingManager);
TableRouteInfo routeInfo = routeProvider.getTableRouteInfo(tableName, _tableCache, routingManager);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems it's pretty easy to create a bug in a future code change where instead of passing routingManager someone uses _routingManager.

Maybe rename _routingManager to _defaultRoutingManager and add a comment on the var declaration?

Copy link
Collaborator Author

@shauryachats shauryachats Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. The intent was to keep _routingManager semantics unchanged and limit the change to the per-query path. Renaming routingManager to selectedRoutingManager makes the selection explicit and also tells us this path uses a conditional routing manager.
Renaming _routingManager would be much broader given its usage.

@codecov-commenter
Copy link

codecov-commenter commented Dec 29, 2025

Codecov Report

❌ Patch coverage is 66.00000% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.26%. Comparing base (6182bed) to head (c05b95f).

Files with missing lines Patch % Lines
.../pinot/core/routing/LogicalTableRouteProvider.java 72.72% 5 Missing and 1 partial ⚠️
...pinot/core/routing/MultiClusterRoutingContext.java 0.00% 6 Missing ⚠️
...sthandler/BaseSingleStageBrokerRequestHandler.java 66.66% 1 Missing and 1 partial ⚠️
...e/pinot/common/utils/config/QueryOptionsUtils.java 0.00% 2 Missing ⚠️
...he/pinot/common/utils/LogicalTableConfigUtils.java 83.33% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             master   #17439   +/-   ##
=========================================
  Coverage     63.26%   63.26%           
  Complexity     1474     1474           
=========================================
  Files          3161     3161           
  Lines        188390   188427   +37     
  Branches      28822    28831    +9     
=========================================
+ Hits         119185   119216   +31     
- Misses        59977    59985    +8     
+ Partials       9228     9226    -2     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-11 63.21% <66.00%> (-0.02%) ⬇️
java-21 63.23% <66.00%> (+<0.01%) ⬆️
temurin 63.26% <66.00%> (+<0.01%) ⬆️
unittests 63.26% <66.00%> (+<0.01%) ⬆️
unittests1 55.66% <50.00%> (+<0.01%) ⬆️
unittests2 33.96% <40.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ankitsultana ankitsultana merged commit 5b2b90d into apache:master Dec 30, 2025
18 checks passed
@shauryachats shauryachats deleted the federated_sse branch January 5, 2026 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants