NoSuchMethodException when using AvroScheme sink

[poneill00]

Hey,

I get the following
2014-12-18 00:31:19,273 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
at org.apache.avro.mapred.AvroOutputFormat.getRecordWriter(AvroOutputFormat.java:156)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.(ReduceTask.java:484)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

when I try to write an avro file as output, note I can read without issues.

Seems to be a version mismatch, I've tried the following:
hadoop 2.6
cascading 2.6.1 (and 2.5.5)

hadoop 2.3
cascading 2.6.1 (and 2.5.5)

any ideas what Im doing wrong?

[poneill00]

note that the same flow works when using Local Tap
//works
return new FileTap(new AvroScheme(projectedFields, projectedTypes), path, SinkMode.REPLACE);

// fails
return new Hfs(new cascading.avro.AvroScheme(projectedFields, projectedTypes), path, SinkMode.REPLACE);

[kkrugler]

This seems like a version miss-match between the avro being used by cascading.avro and the avro that's part of Hadoop.

What version of cascading.avro are you using? And what's the version of the Avro jar in your job jar, versus what's provided with your Hadoop installation?

If it's a version miss-match, setting mapreduce.job.user.classpath.first = true in your job conf can solve these types of issues.

-- Ken

From: PaulOn
Sent: December 17, 2014 4:42:45pm PST
To: ScaleUnlimited/cascading.avro
Subject: [cascading.avro] NoSuchMethodException when using AvroScheme sink (#33)

Hey,

I get the following
2014-12-18 00:31:19,273 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
at org.apache.avro.mapred.AvroOutputFormat.getRecordWriter(AvroOutputFormat.java:156)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.(ReduceTask.java:484)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

when I try to write an avro file as output, note I can read without issues.

Seems to be a version mismatch, I've tried the following:
hadoop 2.6
cascading 2.6.1 (and 2.5.5)

hadoop 2.3
cascading 2.6.1 (and 2.5.5)

any ideas what Im doing wrong?

—
Reply to this email directly or view it on GitHub.

---

Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

---

Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

[kkrugler]

Yes, exactly. If you're using a local tap, then it's not running with the Hadoop jars that are part of the Hadoop installation.

An answer to my previous question would help figure out what's going on…

What version of cascading.avro are you using? And what's the version of the Avro jar in your job jar, versus what's provided with your Hadoop installation?

Regards,

-- Ken

From: PaulOn
Sent: December 17, 2014 5:00:35pm PST
To: ScaleUnlimited/cascading.avro
Subject: Re: [cascading.avro] NoSuchMethodException when using AvroScheme sink (#33)

note that the same flow works when using Local Tap
//works

return new FileTap(new AvroScheme(projectedFields, projectedTypes), path, SinkMode.REPLACE);

// fails
return new Hfs(new cascading.avro.AvroScheme(projectedFields, projectedTypes), path, SinkMode.REPLACE);

—
Reply to this email directly or view it on GitHub.

---

Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

---

Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

[poneill00]

Hey,

I'm using cascading-avro 2.5.0 in my pom and apache-avro 1.7.6 is being pulled into my fatjar.

The stack trace doesnst match the 1.7.6 code (line numbers etc) but I cant figure where its pulling the wrong one from.

I'll try the user.classpath thing, but it seems a bit hacky?

Looking at the cascading.avro pom it looks like its expecting hadoop 2.2...does anyone have this working on a later version of hadoop?

[poneill00]

ok looks like hadoop 2.6 bundles with apache avro 1.7.4....

[poneill00]

ok I've set that in my job conf and same issue,
so I tried actually removing/renaming the 1.7.4 jars bundled in hadoop but now I get a classNotFound error on some Avro files, even though they are part of my fat jar....?

Thanks for your help!

[poneill00]

Looks like user error Im afraid, I was depending on cascading-hadoop rather than hadoop2-mr1

apologies!