Using YARN

[sonsoleslp]

Hi, I used your repo for a class project and succeeded at submitting a spark job in Spark Standalone client mode. How would one go around submitting it in a YARN cluster? I ran "yarn resourcemanager" and "yarn nodemanager" in the spark master and then tried to submit the app with yarn as master. The application was submitted but stuck at this:

18/01/06 21:37:56 INFO yarn.Client: Application report for application_1515274638576_0001 (state: ACCEPTED)
18/01/06 21:37:56 INFO yarn.Client:
         client token: N/A
         diagnostics: [Sat Jan 06 21:37:55 +0000 2018] Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty.  Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:1024, vCores:1>; Queue Resource Limit for AM = <memory:0, vCores:0>; User AM Resource Limit of the queue = <memory:0, vCores:0>; Queue AM Resource Usage = <memory:0, vCores:0>;
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1515274674664
         final status: UNDEFINED
         tracking URL: http://9ffd3c47538e:8088/proxy/application_1515274638576_0001/
         user: root
18/01/06 21:37:57 INFO yarn.Client: Application report for application_1515274638576_0001 (state: ACCEPTED)
18/01/06 21:37:58 INFO yarn.Client: Application report for application_1515274638576_0001 (state: ACCEPTED)
18/01/06 21:37:59 INFO yarn.Client: Application report for application_1515274638576_0001 (state: ACCEPTED)

I tried adding additional env vars from this other project of yours: https://github.com/big-data-europe/docker-hbase/blob/master/distributed/hadoop.env with no luck.

Could you give me any pointers on how to proceed?
Thanks in advance

[earthquakesan]

Hi @sonsoleslp!

Check out 2.0.0-hadoop2.7.4-java8. That branch got example application, see Makefile. To run it (cluster should be running):

make wordcount

[sonsoleslp]

Thanks, I 'll check it out!