[PUBDEV-4331][SW-311] Override default MapReduce type to H2O type #1065
Conversation
The h2odriver can specify H2O specific application master. The application master is still based on standard MRAppMaster, however it overrides default application type to 'H2O'. The is an optional feature! To enable it, a launcher needs to override default MRAppMaster by specifying the following parameter: ``` -Dmapreduce.framework.name=h2o-yarn ``` For example: ``` hadoop jar h2odriver.jar -Dmapreduce.framework.name=h2o-yarn -nodes 1 -mapperXmx 4G -output -ichal_1 ```
| /** | ||
| * H2O Specific MRAppMaster | ||
| */ | ||
| public class H2OMRAppMaster extends MRAppMaster { |
There was a problem hiding this comment.
Why do we need this class? It doesn't seem to do anything H2O specific at this point.
There was a problem hiding this comment.
This is just preparation for new changes - registration of Flow UI port in Mapred UI. Not necessary here, i can remove and include in next one.
| // Modify MRAppMaster commands to point to our master | ||
| LOG.info("Setting MRAppMaster to " + H2OMRAppMaster.class.toString()); | ||
| ContainerLaunchContext origClc = appContext.getAMContainerSpec(); | ||
| ContainerLaunchContext newClc = ContainerLaunchContext.newInstance( |
There was a problem hiding this comment.
If we only need to set application type to H2O then we don't need a custom application master yet.
Is this meant as a groundwork for the future improvements?
|
|
||
| @Override | ||
| public ClientProtocol create(Configuration conf) throws IOException { | ||
| if ("h2o-yarn".equals(conf.get(MRConfig.FRAMEWORK_NAME))) { |
There was a problem hiding this comment.
What are the naming conventions here? I know that Apache Tez uses "yarn-h2o". Not sure if there are other examples like that. Perhaps we should use "-h2o" as a suffix instead.
There was a problem hiding this comment.
Default is yarn. I did not find anything like that, but suffix makes sense, will change it.
|
@michalkurka @tomkraljevic we were discussing this PR yesterday with @jakubhava in the context of Kluster and related customer requests (include Flow UI, include download logs directly in Yarn RM). We were thinking if we can step-by-step introduce our AM by refining default MapReduce AM? |
The h2odriver can specify H2O specific application master.
The application master is still based on standard MRAppMaster, however
it overrides default application type to 'H2O'.
The is an optional feature! To enable it, a launcher needs to override
default MRAppMaster by specifying the following parameter:
For example: