Please report Application Type to Yarn Resource Manager

[exalate-issue-sync]

SW starts an external cluster in Yarn (when called with the appropriate incantation) with a MAPREDUCE application type.

This should be changed to reflect the kind of application running (either H2O or SPARKLINGWATER), this would enable cluster administrators to manage workloads more effectively.

[exalate-issue-sync]

Michal Malohlava commented: The plan is to make it working in the 2.0.5 release.

[exalate-issue-sync]

Vlad Patryshev commented: This is impossible with the current version of spark/yarn.

Spark, when it talks to yarn, sends a hard-coded application type: see
{{
org.apache.spark.deploy.yarn.Client:
...
appContext.setApplicationType("SPARK")
}}

It does take application name from parameters (e.g. if you launch it from spark shell, the name is "Spark Shell"; this is possible to change in the script. But app type - no.

[exalate-issue-sync]

Michal Malohlava commented: We need to look at h2odriver and figure out if we can change ApplicationType there.

[exalate-issue-sync]

Jakub Hava commented: This depends on the PR in H2O introducing custom application master, once it's there, we can proceed with this one

[DinukaH2O]

JIRA Issue Migration Info

Jira Issue: SW-311
Assignee: Jakub Hava
Reporter: Rares Mirica
State: Resolved
Fix Version: 2.0.10
Attachments: N/A
Development PRs: Available

Linked PRs from JIRA

h2oai/h2o-3#1065
h2oai/h2o-3#1151
h2oai/h2o-3#1212
#278

[hasithjp]

JIRA Issue Migration Info Cont'd

Jira Issue Created Date: 2017-01-18T03:21:56.471-0800