NoSuchMethodError: org.elasticsearch.spark.sql.package$.sparkSchemaRDDFunctions

[mborkar]

I am using ES with spark 1.4.1 and hope to write data to ES using spark.
This is my code - standard example given on ES website

package scalaProject

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext   
import org.apache.spark.sql.SQLContext._
import org.apache.spark.sql.functions._
import org.elasticsearch.spark.sql._

object DFload {

// case class used to define the DataFrame
case class Person(name: String, age: Int)

  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("LD").setMaster("spark://localhost:7077")
    conf.set("es.index.auto.create", "true")
    val sc = new SparkContext(conf)
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    // required for 1.3+ data frames
    import sqlContext.implicits._


//  create DataFrame
val people = sc.textFile("/opt/spark-1.2.1-bin-hadoop2.4/examples/src/main/resources/people.txt")   
        .map(_.split(","))
        .map(p => Person(p(0), p(1).trim.toInt))
        .toDF()

people.printSchema()
people.select("name").show()
people.saveToEs("spark/people")
}
}

I create a jar with sbt and use elasticsearch-spark-1.2_2.10-2.1.0.Beta4.jar in my classpath.

But I get this error

java.lang.NoSuchMethodError: org.elasticsearch.spark.sql.package$.sparkSchemaRDDFunctions(Lorg/apache/spark/sql/DataFrame;)Lorg/elasticsearch/spark/sql/package$SparkSchemaRDDFunctions;

I have tried to compile after cleaning cache and using the above jar while submitting spark job.

spark-submit --class scalaProject.DFload --master spark --jars /opt/elasticsearch-hadoop-2.1.0.Beta4/dist/elasticsearch-spark-1.2_2.10-2.1.0.Beta4.jar,test.jar /home/admin/sbtscalaproj/target/scala-2.10/hello_2.10-1.0.jar

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

It fails at the last line of code saveToES.

[costin]

Updated your post with minimal formatting to make it readable

It seems you have a configuration problem:

• you mention Spark 1.4.1 but you're using the es-spark jar for Spark 1.1-1.2. See this section for more information. Basically DataFrames were added in Spark 1.3 while previously there were SchemaRDDs - using the proper version to switch between the two.
• your code refers to /opt/spark-1.2.1/
• you are using es-hadoop 2.1.0.Beta4 - use the final 2.1.0 or better yet 2.1.1.

[mborkar]

worked for me. Thanks Costin.