Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-derived decoder fails with NoClassDefFoundError #574

Open
jonas opened this issue Feb 25, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@jonas
Copy link
Contributor

commented Feb 25, 2017

Not sure if this is a bug in circe or with how circe is used, or a problem related with spark and circe classpaths or something entirely different. Would appreciate any pointers how to debug this.

build.sbt

scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
libraryDependencies ++= Seq("core", "generic").map(module => "io.circe" %% s"circe-$module" % "0.7.0")

Main.scala

import org.apache.spark.sql._
import io.circe.generic.auto._
import io.circe.Json

case class Repo[T: Encoder](path: String) {
  def read(spark: SparkSession): Seq[T] =
    spark.read.option("header", true).option("inferSchema", true).csv(path).as[T].collect().toSeq
}

case class Record(a: Int, b: Int, c: Int)

object Main {
  val json = Json.obj("path" -> Json.fromString("data.csv"))

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[1]").getOrCreate()

    import spark.implicits._
    val repo = json.as[Repo[Record]].right.get

    println(repo.read(spark))
  }
}

data.csv

a,b,c
1,2,3

console

> sbt clean run
[info] Set current project to circe-error (in build file:/opt/circe-error/)
[success] Total time: 1 s, completed 25-Feb-2017 11:23:00 AM
[info] Updating {file:/opt/circe-error/}circe-error...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 1 Scala source to /opt/circe-error/target/scala-2.11/classes...
[info] Running Main
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/02/25 11:23:28 INFO SparkContext: Running Spark version 2.1.0
[...]
17/02/25 11:23:30 INFO SharedState: Warehouse path is 'file:/opt/circe-error/spark-warehouse/'.
[error] (run-main-0) java.lang.NoClassDefFoundError: Main/repo$anon$macro$18
java.lang.NoClassDefFoundError: Main/repo$anon$macro$18
        at Main$anon$exportDecoder$macro$21$1$anon$macro$18$1.from(Main.scala:19)
        at Main$anon$exportDecoder$macro$21$1$anon$macro$18$1.from(Main.scala:19)
        at shapeless.LabelledGeneric$$anon$1.from(generic.scala:229)
        at shapeless.LabelledGeneric$$anon$1.from(generic.scala:226)
        at io.circe.generic.decoding.DerivedDecoder$$anon$1.apply(DerivedDecoder.scala:14)
        at io.circe.Json.as(Json.scala:97)
        at Main$.main(Main.scala:19)
        at Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.lang.ClassNotFoundException: Main.repo$anon$macro$18
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at Main$anon$exportDecoder$macro$21$1$anon$macro$18$1.from(Main.scala:19)
        at Main$anon$exportDecoder$macro$21$1$anon$macro$18$1.from(Main.scala:19)
        at shapeless.LabelledGeneric$$anon$1.from(generic.scala:229)
        at shapeless.LabelledGeneric$$anon$1.from(generic.scala:226)
        at io.circe.generic.decoding.DerivedDecoder$$anon$1.apply(DerivedDecoder.scala:14)
        at io.circe.Json.as(Json.scala:97)
        at Main$.main(Main.scala:19)
        at Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

@jonas jonas changed the title Auto-derived encoder fails with NoClassDefFoundError Auto-derived decoder fails with NoClassDefFoundError Feb 25, 2017

@stevenchen3

This comment has been minimized.

Copy link

commented Jul 10, 2017

Encountered the same issue with circe-0.8.0

@stevenchen3

This comment has been minimized.

Copy link

commented Jul 10, 2017

Some update:

For my case, it is the dependency conflict between circe and spark. circe-generic % 0.8.0 and circe-generic-extras % 0.8.0 use shapeless % 2.3.2, while spark use shapeless % 2.0.0. And shapeless is no backward compatible in this case.

The workaround is to shade shapeless % 2.3.2 to another name during sbt assembly.

@jonas

This comment has been minimized.

Copy link
Contributor Author

commented Sep 18, 2017

Current master of Spark depends on breeze version 0.13.2 which uses shapeless version 2.3.2. So there's hope that this will resolve itself.

@jonas

This comment has been minimized.

Copy link
Contributor Author

commented Sep 18, 2017

I was able to reproduce this with Spark 2.2.0 which uses breeze 0.13.1 and shapeless 2.3.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.