-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HIVE-26412: Create interface to fetch available slots and add the default implementation #3456
Conversation
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezAvailableSlotsCalculator.java
Outdated
Show resolved
Hide resolved
|
||
void initialize(Configuration conf); | ||
|
||
int getAvailaleSlots(InputInitializerContext inputInitializerContext); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@adesh-rao : could you please describe this scenario and a possible implementation class further?
jira description implies that you want to get rid of tightly coupled tez input context, but the interface has tez InputInitializerContext in the signature, so I don't think we addressed the original description
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed the dependency of tez input context on the interface.
@abstractdog Please can you review this request? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM +1
…ault implementation (Adesh Rao, reviewed by Laszlo Bodor, Nikhil Gupta, Sankar Hariappan)) Signed-off-by: Sankar Hariappan <sankarh@apache.org> Closes (apache#3456)
…ault implementation (Adesh Rao, reviewed by Laszlo Bodor, Nikhil Gupta, Sankar Hariappan)) Signed-off-by: Sankar Hariappan <sankarh@apache.org> Closes (apache#3456)
HiveSplitGenerator is tightly coupled with Tez's InputContext to fetch the available slots during split calculation. Creating a interface to fetch the available slots will allow having other implementation too which are more suitable for different cloud vendors.
The change contains a default implementation using Tez's InputContext and a new HiveConfiguration pointing to the class implementing the interface. Different cloud vendors can plugin there strategy by implementing the interface.