is there any change for us to run emr-etl-runner on hadoop 2 AMI versions?

[ngocthanhit]

Hello All,
Can we run the emr-etl-runner in hadoop 2 AMI Versions like AMI 3.4.0 ?
I tried with my existing cluster but saw this error :

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.twitter.scalding.Job$.apply(Job.scala:47)
at com.twitter.scalding.Tool.getJob(Tool.scala:48)
at com.twitter.scalding.Tool.run(Tool.scala:68)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.snowplowanalytics.snowplow.enrich.hadoop.JobRunner$.main(JobRunner.scala:33)
at com.snowplowanalytics.snowplow.enrich.hadoop.JobRunner.main(JobRunner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
com/snowplowanalytics/snowplow/enrich/common/utils/ConversionUtils$.decodeBase64Url(Ljava/lang/String;Ljava/lang/String;)Lscalaz/Validation; @13: invokevirtual
Reason:
Type 'org/apache/commons/codec/binary/Base64' (current frame, stack[0]) is not assignable to 'org/apache/commons/codec/binary/BaseNCodec'

Thanks for the support.

[alexanderdean]

Hi @ngocthanhit - afraid not. Our Hadoop jobs are all built on Scalding, which is still Hadoop 1.x only.