"java.lang.VerifyError: Bad return type" runnig spark-solr in Scala on Cloudera-5.5 #36

dsimmie · 2016-02-02T18:08:58Z

Apologies if this is an error in my setup rather than an issue with the code, but I am not sure myself.

I am trying to ingest documents into Solr from a Spark Streaming job in Scala.

This is the Solr related code block:

    val docs = kafkaStream.flatMap(msg => {
      Json.parse(msg._2).as[JsArray].value.flatMap(x => x.as[JsArray].value).map(x => {
        val doc:SolrInputDocument = new common.SolrInputDocument()
        doc.setField("id", (x \ "id").as[String])
        doc.setField("description", (x \ "description").as[String])
        doc
      })
    })
    SolrSupport.indexDStreamOfDocs("hdp-vm-02:2181,hdp-vm-02:2181/solr", "descriptions_all", 10, docs)

When I run this via spark-submit the job fails straight away because of a java.lang.VerifyError. I presume I am mismatching versions of code somewhere here to get the following error:

java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;)Lorg/apache/http/impl/client/CloseableHttpClient; @57: areturn
  Reason:
    Type 'org/apache/http/impl/client/SystemDefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
  Current Frame:
    bci: @57
    flags: { }
    locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/SystemDefaultHttpClient' }
    stack: { 'org/apache/http/impl/client/SystemDefaultHttpClient' }
  Bytecode:
    0000000: bb00 0359 2ab7 0004 4cb2 0005 b900 0601
    0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
    0000020: b600 0a2b b600 0bb6 000c b900 0d02 00b8
    0000030: 000e 4d2c 2bb8 000f 2cb0               
  Stackmap Table:
    append_frame(@47,Object[#141])

    at org.apache.solr.client.solrj.impl.CloudSolrClient.<init>(CloudSolrClient.java:195)
    at com.lucidworks.spark.SolrSupport.getSolrServer(SolrSupport.java:81)
    at com.lucidworks.spark.SolrSupport$5.call(SolrSupport.java:212)
    at com.lucidworks.spark.SolrSupport$5.call(SolrSupport.java:210)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:222)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:222)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:898)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:898)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1850)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

From what I can tell the SystemDefaultHttpClient extends from CloseableHttpClient so I don't see why this fails. Does anyone have an idea what is wrong?

I am using:

Cloudera 5.5
Solr 4.10.3-CDH-5.5
I can connect to zookeeper on those nodes and see data in the /solr root
The collection I have given exists, in zookeeper and the Solr UI

Dependencies from build.sbt:

libraryDependencies ++= Seq(
  "com.typesafe" % "config" % "1.3.0",
  "com.datastax.spark" %% "spark-cassandra-connector" % "1.5.0-M1",
  "org.apache.spark" % "spark-streaming-kafka_2.10"  % "1.5.1" % "provided",
  "org.apache.spark" %% "spark-core" % "1.5.1" % "provided",
  "org.apache.spark" %% "spark-streaming" % "1.5.1" % "provided",
  "org.apache.spark" %% "spark-graphx" % "1.5.1" % "provided",
  "org.apache.spark" %% "spark-sql" % "1.5.1" % "provided",
  "org.json4s" % "json4s-jackson_2.10" % "3.2.11",
  "com.typesafe.play" %% "play-json" % "2.3.4",
  "com.fasterxml.uuid" % "java-uuid-generator" % "3.1.4",
  "joda-time" % "joda-time" % "2.9.1",
  "com.databricks" %% "spark-avro" % "2.0.1",
  "com.lucidworks.spark" % "spark-solr" % "1.1.2",
  "org.apache.solr" % "solr-core" % "4.10.3",
  "org.apache.solr" % "solr-common" % "1.3.0"
)

Thanks,
Donal

The text was updated successfully, but these errors were encountered:

dsimmie · 2016-02-16T17:00:14Z

I was importing solr-core 4.10.3 but spark-solr has a dependency on Solr 5.3.1 that I was not aware of. This means it is not compatible with the current cloudera version (4.10.3)

kiranchitturi · 2016-02-16T19:19:56Z

4.10.3 is an old version of Solr. The spark-solr project will always try to be on the latest version of Solr

hakanilter · 2016-05-20T14:15:59Z

Hello,

I'm having the same error with Apache Solr 6.0.0. I tried it with Spark-Solr 2.0.0 and also with master branch with Solr version set to the latest version too.

I'm using CDH 5.7 and Spark 1.6.0. Here is the stacktrace:

com.google.common.util.concurrent.ExecutionError: java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;)Lorg/apache/http/impl/client/CloseableHttpClient; @57: areturn
  Reason:
    Type 'org/apache/http/impl/client/SystemDefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
  Current Frame:
    bci: @57
    flags: { }
    locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/SystemDefaultHttpClient' }
    stack: { 'org/apache/http/impl/client/SystemDefaultHttpClient' }
  Bytecode:
    0x0000000: bb00 0359 2ab7 0004 4cb2 0005 b900 0601
    0x0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
    0x0000020: b600 0a2b b600 0bb6 000c b900 0d02 00b8
    0x0000030: 000e 4d2c 2bb8 000f 2cb0
  Stackmap Table:
    append_frame(@47,Object[#143])

    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2232)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
    at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:93)
    at com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:97)
    at com.lucidworks.spark.util.SolrQuerySupport$.getUniqueKey(SolrQuerySupport.scala:82)
    at com.lucidworks.spark.rdd.SolrRDD.<init>(SolrRDD.scala:32)
    at com.lucidworks.spark.SolrRelation.<init>(SolrRelation.scala:53)
    at solr.DefaultSource.createRelation(DefaultSource.scala:26)
    at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
    at veloxity.streaming.job.SparkStreamingSolrWriterJob$$anonfun$main$1$$anonfun$apply$1$$anonfun$apply$2.apply(SparkStreamingSolrWriterJob.scala:77)
    at veloxity.streaming.job.SparkStreamingSolrWriterJob$$anonfun$main$1$$anonfun$apply$1$$anonfun$apply$2.apply(SparkStreamingSolrWriterJob.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at veloxity.streaming.job.SparkStreamingSolrWriterJob$$anonfun$main$1$$anonfun$apply$1.apply(SparkStreamingSolrWriterJob.scala:61)
    at veloxity.streaming.job.SparkStreamingSolrWriterJob$$anonfun$main$1$$anonfun$apply$1.apply(SparkStreamingSolrWriterJob.scala:52)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:223)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;)Lorg/apache/http/impl/client/CloseableHttpClient; @57: areturn
  Reason:
    Type 'org/apache/http/impl/client/SystemDefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
  Current Frame:
    bci: @57
    flags: { }
    locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/SystemDefaultHttpClient' }
    stack: { 'org/apache/http/impl/client/SystemDefaultHttpClient' }
  Bytecode:
    0x0000000: bb00 0359 2ab7 0004 4cb2 0005 b900 0601
    0x0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
    0x0000020: b600 0a2b b600 0bb6 000c b900 0d02 00b8
    0x0000030: 000e 4d2c 2bb8 000f 2cb0
  Stackmap Table:
    append_frame(@47,Object[#143])

    at com.lucidworks.spark.util.SolrSupport$.setupKerberosIfNeeded(SolrSupport.scala:64)
    at com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:81)
    at com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:89)
    at com.lucidworks.spark.util.CacheSolrClient$$anon$1.load(SolrSupport.scala:38)
    at com.lucidworks.spark.util.CacheSolrClient$$anon$1.load(SolrSupport.scala:36)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
    ... 35 more
Exception in thread "main" com.google.common.util.concurrent.ExecutionError: java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;)Lorg/apache/http/impl/client/CloseableHttpClient; @57: areturn
  Reason:
    Type 'org/apache/http/impl/client/SystemDefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
  Current Frame:
    bci: @57
    flags: { }
    locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/SystemDefaultHttpClient' }
    stack: { 'org/apache/http/impl/client/SystemDefaultHttpClient' }
  Bytecode:
    0x0000000: bb00 0359 2ab7 0004 4cb2 0005 b900 0601
    0x0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
    0x0000020: b600 0a2b b600 0bb6 000c b900 0d02 00b8
    0x0000030: 000e 4d2c 2bb8 000f 2cb0
  Stackmap Table:
    append_frame(@47,Object[#143])

    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2232)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
    at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:93)
    at com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:97)
    at com.lucidworks.spark.util.SolrQuerySupport$.getUniqueKey(SolrQuerySupport.scala:82)
    at com.lucidworks.spark.rdd.SolrRDD.<init>(SolrRDD.scala:32)
    at com.lucidworks.spark.SolrRelation.<init>(SolrRelation.scala:53)
    at solr.DefaultSource.createRelation(DefaultSource.scala:26)
    at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
    at veloxity.streaming.job.SparkStreamingSolrWriterJob$$anonfun$main$1$$anonfun$apply$1$$anonfun$apply$2.apply(SparkStreamingSolrWriterJob.scala:77)
    at veloxity.streaming.job.SparkStreamingSolrWriterJob$$anonfun$main$1$$anonfun$apply$1$$anonfun$apply$2.apply(SparkStreamingSolrWriterJob.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at veloxity.streaming.job.SparkStreamingSolrWriterJob$$anonfun$main$1$$anonfun$apply$1.apply(SparkStreamingSolrWriterJob.scala:61)
    at veloxity.streaming.job.SparkStreamingSolrWriterJob$$anonfun$main$1$$anonfun$apply$1.apply(SparkStreamingSolrWriterJob.scala:52)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
    at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:661)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
    at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
    at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
    at scala.util.Try$.apply(Try.scala:161)
    at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:223)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;)Lorg/apache/http/impl/client/CloseableHttpClient; @57: areturn
  Reason:
    Type 'org/apache/http/impl/client/SystemDefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
  Current Frame:
    bci: @57
    flags: { }
    locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/SystemDefaultHttpClient' }
    stack: { 'org/apache/http/impl/client/SystemDefaultHttpClient' }
  Bytecode:
    0x0000000: bb00 0359 2ab7 0004 4cb2 0005 b900 0601
    0x0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
    0x0000020: b600 0a2b b600 0bb6 000c b900 0d02 00b8
    0x0000030: 000e 4d2c 2bb8 000f 2cb0
  Stackmap Table:
    append_frame(@47,Object[#143])

    at com.lucidworks.spark.util.SolrSupport$.setupKerberosIfNeeded(SolrSupport.scala:64)
    at com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:81)
    at com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:89)
    at com.lucidworks.spark.util.CacheSolrClient$$anon$1.load(SolrSupport.scala:38)
    at com.lucidworks.spark.util.CacheSolrClient$$anon$1.load(SolrSupport.scala:36)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
    ... 35 more

Thanks in advance.

kiranchitturi · 2016-05-20T16:28:12Z

@hakanilter The master branch and 2.0.x are tested only with 5.5.1. Once the bug release (6.0.1) is out, I will update the master and re-compile, fix any bugs.

Does CDH 5.7 ship with stock 6.0.0 Solr ?

hakanilter · 2016-05-26T06:44:08Z

@kiranchitturi CDH 5.7 comes with Solr 4.3.10, I run Solr 6.0.0 with docker.

Thanks.

iRajashekharC · 2016-08-12T06:01:42Z

Download the following jars (just google for it) and copy them to your %hadoop_home%\share\hadoop\common\lib folder.

httpclient-4.4.1.jar
httpcore-4.4.1.jar

Refer to this link: http://prasi82.blogspot.in/2015/11/spark-solr-integration-on-windows.html

dsimmie closed this as completed Feb 16, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"java.lang.VerifyError: Bad return type" runnig spark-solr in Scala on Cloudera-5.5 #36

"java.lang.VerifyError: Bad return type" runnig spark-solr in Scala on Cloudera-5.5 #36

dsimmie commented Feb 2, 2016

dsimmie commented Feb 16, 2016

kiranchitturi commented Feb 16, 2016

hakanilter commented May 20, 2016

kiranchitturi commented May 20, 2016

hakanilter commented May 26, 2016

iRajashekharC commented Aug 12, 2016

"java.lang.VerifyError: Bad return type" runnig spark-solr in Scala on Cloudera-5.5 #36

"java.lang.VerifyError: Bad return type" runnig spark-solr in Scala on Cloudera-5.5 #36

Comments

dsimmie commented Feb 2, 2016

dsimmie commented Feb 16, 2016

kiranchitturi commented Feb 16, 2016

hakanilter commented May 20, 2016

kiranchitturi commented May 20, 2016

hakanilter commented May 26, 2016

iRajashekharC commented Aug 12, 2016