Getting Avro error while trying to load data from EMR spark dataframe to redshift table.

[manisha803]

16/06/27 17:00:14 ERROR InsertIntoHadoopFsRelation: Aborting task.
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
at org.apache.avro.mapreduce.AvroKeyOutputFormat.getRecordWriter(AvroKeyOutputFormat.java:85)
at com.databricks.spark.avro.AvroOutputWriter.(AvroOutputWriter.scala:82)
at com.databricks.spark.avro.AvroOutputWriterFactory.newInstance(AvroOutputWriterFactory.scala:31)
at org.apache.spark.sql.sources.DefaultWriterContainer.initWriters(commands.scala:470)
at org.apache.spark.sql.sources.BaseWriterContainer.executorSideSetup(commands.scala:360)
at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation.org$apache$spark$sql$sources$InsertIntoHadoopFsRelation$$writeRows$1(commands.scala:172)
at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation$$anonfun$insert$1.apply(commands.scala:160)
at org.apache.spark.sql.sources.InsertIntoHadoopFsRelation$$anonfun$insert$1.apply(commands.scala:160)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)

[JoshRosen]

This looks like a duplicate of #180 and is likely a problem with dependency conflicts in your environment.

[manisha803]

I am using EMR 4.6.0 and spark 1.6.1. #180 is for the older version of the spark.

From: Josh Rosen notifications@github.com
Reply-To: databricks/spark-redshift reply@reply.github.com
Date: Monday, June 27, 2016 at 6:03 PM
To: databricks/spark-redshift spark-redshift@noreply.github.com
Cc: Manisha Sojitra Manisha.Sojitra@sheknows.com, State change state_change@noreply.github.com
Subject: Re: [databricks/spark-redshift] Getting Avro error while trying to load data from EMR spark dataframe to redshift table. (#222)

This looks like a duplicate of #180#180 and is likely a problem with dependency conflicts in your environment.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHubhttps://github.com//issues/222#issuecomment-228889923, or mute the threadhttps://github.com/notifications/unsubscribe/ATPW4toRLMk8FYHJ_lBSVsivdD7_DXIJks5qQEjDgaJpZM4I_gqy.

[JoshRosen]

Which versions of spark-redshift, spark-avro, and avro-mapred are you using?

[manisha803]

Hello,

I am using following dependency.

com.databricks
spark-avro_2.10
2.0.1


org.apache.avro
avro-mapred
hadoop2
1.7.7

  <dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-redshift_2.10</artifactId>
<version>0.6.0</version>

Thanks
Manisha,
From: Josh Rosen notifications@github.com
Reply-To: databricks/spark-redshift reply@reply.github.com
Date: Monday, June 27, 2016 at 6:40 PM
To: databricks/spark-redshift spark-redshift@noreply.github.com
Cc: Manisha Sojitra Manisha.Sojitra@sheknows.com, State change state_change@noreply.github.com
Subject: Re: [databricks/spark-redshift] Getting Avro error while trying to load data from EMR spark dataframe to redshift table. (#222)

Which versions of spark-redshift, spark-avro, and avro-mapred are you using?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHubhttps://github.com//issues/222#issuecomment-228897498, or mute the threadhttps://github.com/notifications/unsubscribe/ATPW4pSq8uHU_2UnahEhbEsVzJ1lU58uks5qQFFkgaJpZM4I_gqy.

[JoshRosen]

Please give this a try using the 1.0.1 release of this library and re-open if this still an issue.