running an s3 jar file with dependencies

[Fatima-yo]

Hi and thanks for this examples, they are great.
Am trying to run an scala jar file from s3 with some dependencies, like this:

aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Spark Name=Pig --ec2-attributes '{"KeyName":"devtmp","InstanceProfile":"EMR_EC2_DefaultRole","SubnetId":"subnet-3ce5d87a","EmrManagedSlaveSecurityGroup":"sg-5b245e3e","EmrManagedMasterSecurityGroup":"sg-58245e3d"}' --enable-debugging --instance-groups '[{"InstanceCount":1,"BidPrice":"1.5","InstanceGroupType":"MASTER","InstanceType":"r3.xlarge","Name":"Master instance group - 1"},{"InstanceCount":10,"BidPrice":"1.5","InstanceGroupType":"CORE","InstanceType":"r3.xlarge","Name":"Core instance group - 2"}]' --log-uri 's3n://aws-logs-032299448736-us-east-1/elasticmapreduce/' --name 'Test Fatima Job01' --region us-east-1 --release-label emr-4.7.2 --scale-down-behavior TERMINATE_AT_TASK_COMPLETION --service-role EMR_DefaultRole
{
"ClusterId": "j-2LN2PH6VJZIDB"
}
[ec2-user@ip-172-31-36-110 ~]$ aws emr add-steps --cluster-id j-2LN2PH6VJZIDB --steps Name=Job01,Jar=s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,Args=[/home/hadoop/spark/bin/spark-submit,--deploy-mode,cluster,--master,yarn,--class,com.jumbo.mop.jobs.Job01,s3://dev.exalitica.com/cencosud/mop-jumbo-2_2.11-0.1.0-TEST.jar,s3://dev.exalitica.com/cencosud/mysql-connector-java-5.0.8.jar,10],ActionOnFailure=TERMINATE_CLUSTER
{
"StepIds": [
"s-2NRYWKTG4CZBY"
]
}

All I get is this error :-( in stderr for the second step.

Exception in thread "main" java.lang.RuntimeException: Local file does not exist.
at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.fetchFile(ScriptRunner.java:30)
at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.main(ScriptRunner.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

It would great if you could add an example for running a step using a jar with jar dependencies.

Best,
Fatima

[dacort]

Hi there - thanks for your contribution. We're updating this repository to include more relevant and recent information.

As such, we're cleaning up and closing old issues and PRs.

Feel free to open an issue if you still use EMR and would like to see an example of something!