Lead controller resource is design to logically separate the per-table functionality in pinot cluster.
As lead controller resource feature has been implemented and running for a while, we'd like to enable this feature to keep the behavior consistent in all Pinot clusters.
Before enabling the lead controller resource, there is only 1 pinot controller (which is also the helix lead controller) that does all the pinot controller periodic tasks and realtime segment completions for all the tables. The lead helix/pinot controller can be found at the CONTROLLER/LEADER ZNode.
After enabling the lead controller resource, helix controllers and pinot controllers don't have to run on the same hardware. Znode CONTROLLER/LEADER only denotes Helix lead controller. All the controller periodic tasks and realtime segment completions are distributed to all the available pinot controllers. The lead Pinot controller for each of the tables can be found by the following APIs:
-
Get the leaders for all the tables in the cluster
GET /leader/tables
-
Given a table name, return the partition id and lead controller instance id
GET /leader/tables/{tableName}
Offline segment upload won't bind to any of the pinot controllers; all the available pinot controllers accepts segment upload since it's neither controller periodic tasks nor realtime segment completions.
Details can be found in doc: https://cwiki.apache.org/confluence/display/PINOT/Controller+Separation+between+Helix+and+Pinot
There will be two PRs for this issue:
The first PR will enable lead controller in all Pinot clusters. New onboard clusters or existing running clusters that start to run with this code will have lead controller resource enabled.
The second PR will clean up the code that handles the old protocol in the repo, which will be merged in Apache Pinot Release 0.8.0 or higher. The cleanup should be done in the following classes:
- Controller: LeadControllerManager.java
- Server: ControllerLeaderLocator.java
Lead controller resource is design to logically separate the per-table functionality in pinot cluster.
As lead controller resource feature has been implemented and running for a while, we'd like to enable this feature to keep the behavior consistent in all Pinot clusters.
Before enabling the lead controller resource, there is only 1 pinot controller (which is also the helix lead controller) that does all the pinot controller periodic tasks and realtime segment completions for all the tables. The lead helix/pinot controller can be found at the
CONTROLLER/LEADERZNode.After enabling the lead controller resource, helix controllers and pinot controllers don't have to run on the same hardware. Znode
CONTROLLER/LEADERonly denotes Helix lead controller. All the controller periodic tasks and realtime segment completions are distributed to all the available pinot controllers. The lead Pinot controller for each of the tables can be found by the following APIs:Get the leaders for all the tables in the cluster
GET /leader/tables
Given a table name, return the partition id and lead controller instance id
GET /leader/tables/{tableName}
Offline segment upload won't bind to any of the pinot controllers; all the available pinot controllers accepts segment upload since it's neither controller periodic tasks nor realtime segment completions.
Details can be found in doc: https://cwiki.apache.org/confluence/display/PINOT/Controller+Separation+between+Helix+and+Pinot
There will be two PRs for this issue:
The first PR will enable lead controller in all Pinot clusters. New onboard clusters or existing running clusters that start to run with this code will have lead controller resource enabled.
The second PR will clean up the code that handles the old protocol in the repo, which will be merged in Apache Pinot Release 0.8.0 or higher. The cleanup should be done in the following classes: