An error occurred when using HarvesterJsonSerDe.impl.JsonToEntity

[thbzh]

I have a code like the following:

override def send(plan: ExecutionPlan): Unit = {
    import za.co.absa.spline.harvester.json.HarvesterJsonSerDe.impl._
    logInfo("================ fromJson :" + plan.toJson.fromJson[ExecutionPlan].name)
  }

this function fromJson throwed a exception like this:

Exception in thread "main" org.json4s.package$MappingException: No usable value for attributes
No usable value for childRefs
No constructor for type AttrOrExprRef, JObject(List((id,JString(expr-9))))
	at org.json4s.reflect.package$.fail(package.scala:95)
	at org.json4s.Extraction$ClassInstanceBuilder.buildCtorArg(Extraction.scala:526)
	at org.json4s.Extraction$ClassInstanceBuilder.$anonfun$instantiate$4(Extraction.scala:546)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at org.json4s.Extraction$ClassInstanceBuilder.instantiate(Extraction.scala:546)
	at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:597)
	at org.json4s.Extraction$.$anonfun$extract$9(Extraction.scala:400)
	at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
	at org.json4s.Extraction$.extract(Extraction.scala:392)
	at org.json4s.Extraction$.extract(Extraction.scala:39)
	at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21)
	at za.co.absa.commons.json.AbstractJsonSerDe$JsonToEntity.fromJson(AbstractJsonSerDe.scala:57)
	at com.thb.spark.spline.udf.Lineage$.main(Lineage.scala:11)
	at com.thb.spark.spline.udf.Lineage.main(Lineage.scala)
Caused by: org.json4s.package$MappingException: No usable value for childRefs
No constructor for type AttrOrExprRef, JObject(List((id,JString(expr-9))))
	at org.json4s.reflect.package$.fail(package.scala:95)
	at org.json4s.Extraction$ClassInstanceBuilder.buildCtorArg(Extraction.scala:526)
	at org.json4s.Extraction$ClassInstanceBuilder.$anonfun$instantiate$4(Extraction.scala:546)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at org.json4s.Extraction$ClassInstanceBuilder.instantiate(Extraction.scala:546)
	at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:597)
	at org.json4s.Extraction$.$anonfun$extract$9(Extraction.scala:400)
	at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
	at org.json4s.Extraction$.extract(Extraction.scala:392)
	at org.json4s.Extraction$CollectionBuilder.$anonfun$mkCollection$1(Extraction.scala:410)
	at scala.collection.immutable.List.map(List.scala:297)
	at org.json4s.Extraction$CollectionBuilder.mkCollection(Extraction.scala:410)
	at org.json4s.Extraction$CollectionBuilder.result(Extraction.scala:443)
	at org.json4s.Extraction$.$anonfun$extract$8(Extraction.scala:382)
	at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
	at org.json4s.Extraction$.extract(Extraction.scala:382)
	at org.json4s.Extraction$ClassInstanceBuilder.buildCtorArg(Extraction.scala:514)
	... 18 more
Caused by: org.json4s.package$MappingException: No constructor for type AttrOrExprRef, JObject(List((id,JString(expr-9))))
	at org.json4s.reflect.package$.fail(package.scala:95)
	at org.json4s.Extraction$ClassInstanceBuilder.$anonfun$constructor$2(Extraction.scala:477)
	at scala.Option.getOrElse(Option.scala:189)
	at org.json4s.Extraction$ClassInstanceBuilder.constructor(Extraction.scala:477)
	at org.json4s.Extraction$ClassInstanceBuilder.instantiate(Extraction.scala:532)
	at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:597)
	at org.json4s.Extraction$.$anonfun$extract$9(Extraction.scala:400)
	at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
	at org.json4s.Extraction$.extract(Extraction.scala:392)
	at org.json4s.Extraction$CollectionBuilder.$anonfun$mkCollection$1(Extraction.scala:410)
	at scala.collection.immutable.List.map(List.scala:293)
	at org.json4s.Extraction$CollectionBuilder.mkCollection(Extraction.scala:410)
	at org.json4s.Extraction$CollectionBuilder.result(Extraction.scala:443)
	at org.json4s.Extraction$.$anonfun$extract$8(Extraction.scala:382)
	at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
	at org.json4s.Extraction$.extract(Extraction.scala:382)
	at org.json4s.Extraction$ClassInstanceBuilder.buildCtorArg(Extraction.scala:514)
	... 39 more

How can I call serialization/deserialization?

[thbzh]

my maven dependencies:

<dependency>
            <groupId>za.co.absa.spline.agent.spark</groupId>
            <artifactId>spark-3.0-spline-agent-bundle_2.12</artifactId>
            <version>1.0.5</version>
            <exclusions>
                <exclusion>
                    <artifactId>scala-library</artifactId>
                    <groupId>org.scala-lang</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-xml_2.12</artifactId>
                    <groupId>org.scala-lang.modules</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-reflect</artifactId>
                    <groupId>org.scala-lang</groupId>
                </exclusion>
            </exclusions>
        </dependency>

my scala version is 2.12.17

[cerveada]

The bundle is not supposed to be used as a maven dependency. You should use agent-core instead.

See https://github.com/AbsaOSS/spline-spark-agent#selecting-artifact

[cerveada]

Why are you converting toJson.fromJson ???

[wajda]

looks like it's a classpath conflict between different json4s versions. Try to analyse your Maven dependency graph and see if there is a conflict.

[cerveada]

@thbzh We removed the dependencies on json libraries, now they are all shaded. So your issue should be solved by using the latest agent version.

Feel free to reopen or create a new issue if you still have a problem.