IncompatibleClassChangeError: class z.c.a.s.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead #579

tiguanin · 2023-01-25T08:44:51Z

Hi! Got exception in Spline initialization.

Version: za.co.absa.spline.agent.spark:spark-3.2-spline-agent-bundle_2.12:1.0.1

java.lang.RuntimeException: Plugin instantiation failure: class za.co.absa.spline.harvester.plugin.embedded.DeltaPlugin
	at za.co.absa.spline.harvester.plugin.registry.AutoDiscoveryPluginRegistry$$anonfun$$nestedInanonfun$allPlugins$1$1.applyOrElse(AutoDiscoveryPluginRegistry.scala:54)
	at za.co.absa.spline.harvester.plugin.registry.AutoDiscoveryPluginRegistry$$anonfun$$nestedInanonfun$allPlugins$1$1.applyOrElse(AutoDiscoveryPluginRegistry.scala:54)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
	at scala.util.Failure.recover(Try.scala:234)
	at za.co.absa.spline.harvester.plugin.registry.AutoDiscoveryPluginRegistry.$anonfun$allPlugins$1(AutoDiscoveryPluginRegistry.scala:54)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at za.co.absa.spline.harvester.plugin.registry.AutoDiscoveryPluginRegistry.<init>(AutoDiscoveryPluginRegistry.scala:51)
	at za.co.absa.spline.agent.SplineAgent$.create(SplineAgent.scala:66)
	at za.co.absa.spline.harvester.SparkLineageInitializer.createListener(SparkLineageInitializer.scala:162)
	at za.co.absa.spline.harvester.SparkLineageInitializer.$anonfun$createListener$6(SparkLineageInitializer.scala:139)
	at za.co.absa.spline.harvester.SparkLineageInitializer.withErrorHandling(SparkLineageInitializer.scala:176)
	at za.co.absa.spline.harvester.SparkLineageInitializer.createListener(SparkLineageInitializer.scala:138)
	at za.co.absa.spline.harvester.listener.SplineQueryExecutionListener.<init>(SplineQueryExecutionListener.scala:37)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.spark.util.Utils$.$anonfun$loadExtensions$1(Utils.scala:2879)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2868)
	at org.apache.spark.sql.util.ExecutionListenerManager.$anonfun$new$1(QueryExecutionListener.scala:85)
	at org.apache.spark.sql.util.ExecutionListenerManager.$anonfun$new$1$adapted(QueryExecutionListener.scala:84)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.sql.util.ExecutionListenerManager.<init>(QueryExecutionListener.scala:84)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$listenerManager$2(BaseSessionStateBuilder.scala:330)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder.listenerManager(BaseSessionStateBuilder.scala:330)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:358)
	at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1148)
	at org.apache.spark.sql.SparkSession.$anonfun$sessionState$2(SparkSession.scala:158)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:156)
	at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:153)
	at org.apache.spark.sql.DataFrameReader.<init>(DataFrameReader.scala:732)
	at org.apache.spark.sql.SparkSession.read(SparkSession.scala:658)
	at $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:24)
	at $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:28)
	at $line16.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:30)
	at $line16.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:32)
	at $line16.$read$$iw$$iw$$iw$$iw.<init>(<console>:34)
	at $line16.$read$$iw$$iw$$iw.<init>(<console>:36)
	at $line16.$read$$iw$$iw.<init>(<console>:38)
	at $line16.$read$$iw.<init>(<console>:40)
	at $line16.$read.<init>(<console>:42)
	at $line16.$read$.<init>(<console>:46)
	at $line16.$read$.<clinit>(<console>)
	at $line16.$eval$.$print$lzycompute(<console>:7)
	at $line16.$eval$.$print(<console>:6)
	at $line16.$eval.$print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
	at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
	at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
	at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
	at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
	at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
	at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:865)
	at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:733)
	at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:435)
	at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:456)
	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:239)
	at org.apache.spark.repl.Main$.doMain(Main.scala:78)
	at org.apache.spark.repl.Main$.main(Main.scala:58)
	at org.apache.spark.repl.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at za.co.absa.spline.harvester.plugin.registry.AutoDiscoveryPluginRegistry.$anonfun$instantiatePlugin$1(AutoDiscoveryPluginRegistry.scala:73)
	at scala.util.Try$.apply(Try.scala:213)
	at za.co.absa.spline.harvester.plugin.registry.AutoDiscoveryPluginRegistry.instantiatePlugin(AutoDiscoveryPluginRegistry.scala:63)
	at za.co.absa.spline.harvester.plugin.registry.AutoDiscoveryPluginRegistry.$anonfun$allPlugins$1(AutoDiscoveryPluginRegistry.scala:53)
	... 89 more
Caused by: java.lang.IncompatibleClassChangeError: class za.co.absa.spline.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead has interface org.apache.spark.sql.catalyst.plans.logical.LeafNode as super class
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at za.co.absa.spline.harvester.plugin.embedded.DeltaPlugin.<init>(DeltaPlugin.scala:43)
	... 97 more

The text was updated successfully, but these errors were encountered:

cerveada · 2023-01-25T08:50:33Z

Are you on Databricks? If so which version?

tiguanin · 2023-01-25T10:45:45Z

Are you on Databricks? If so which version?

@cerveada No, without using DataBricks. Clean server with Spark. With Spline agent bundle only.

cerveada · 2023-01-25T11:57:58Z

Can you check which Spark and Scala version you are using?

tiguanin · 2023-01-25T12:00:05Z

Scala: 2.12.17
Spark: 3.2.1

tiguanin · 2023-01-25T12:19:01Z

And same problem with 1.0.0 version

wajda · 2023-01-25T17:49:08Z

In the Spark 3.2 the LeafNode is a trait while in older versions in was a class. The change is source compatible, but not binary compatible. Spline bundles for all Spark versions are compiled against Spark 2.4. It's not a problem when Spline only calls Spark API, but when Spline components extend Spark API it becomes a problem.
The fix we have agreed on is to redesign the DeltaPlugin to avoid extending Spark abstractions.

wajda · 2023-01-25T18:02:54Z

The error was not caught by the integration tests because of #581

tiguanin · 2023-01-25T18:14:32Z

Thank you!

…until the issue #579 is fixed

cerveada · 2023-01-26T10:35:32Z

This should be fixed by #575

…until the issue #579 is fixed

wajda · 2023-01-26T13:37:00Z

@tiguanin please try the snapshot build, it should fix the error - https://teamcity.jetbrains.com/buildConfiguration/OpenSourceProjects_AbsaOSS_SplineAgentSpark_AutoBuildArtifactsSparkScala212/4063111?buildTab=artifacts

wajda · 2023-01-26T16:59:01Z

Integration tests now cover this case and are passing on CI. So I consider this ticket solved.

wajda added the bug Something isn't working label Jan 25, 2023

wajda assigned cerveada Jan 25, 2023

wajda added this to the 1.0.2 milestone Jan 25, 2023

wajda changed the title ~~Spline ver 1.0.1 troubles with DeltaPlugin instantiation~~ java.lang.IncompatibleClassChangeError: class za.co.absa.spline.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead Jan 25, 2023

wajda changed the title ~~java.lang.IncompatibleClassChangeError: class za.co.absa.spline.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead~~ IncompatibleClassChangeError: class z.c.a.spline.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead Jan 25, 2023

wajda changed the title ~~IncompatibleClassChangeError: class z.c.a.spline.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead~~ IncompatibleClassChangeError: class z.c.a.s.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead Jan 25, 2023

wajda added a commit that referenced this issue Jan 25, 2023

spline #581 Temporarily disable DeltaPlugin and MergeIntoNodeBuilder …

03cb8d8

…until the issue #579 is fixed

wajda mentioned this issue Jan 25, 2023

Bugfix/spline spark agent 581 integration tests #582

Merged

wajda added a commit that referenced this issue Jan 26, 2023

spline #581 Temporarily disable DeltaPlugin and MergeIntoNodeBuilder …

ec296b8

…until the issue #579 is fixed

wajda linked a pull request Jan 26, 2023 that will close this issue

Bugfix/agent 574 merge into using fix #575

Merged

wajda closed this as completed Jan 26, 2023

wajda mentioned this issue Feb 6, 2023

Spark not able to query S3 after adding the jar spark-3.3-spline-agent-bundle_2.12-1.0.1.jar #592

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IncompatibleClassChangeError: class z.c.a.s.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead #579

IncompatibleClassChangeError: class z.c.a.s.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead #579

tiguanin commented Jan 25, 2023 •

edited by cerveada

Loading

cerveada commented Jan 25, 2023

tiguanin commented Jan 25, 2023 •

edited

Loading

cerveada commented Jan 25, 2023

tiguanin commented Jan 25, 2023

tiguanin commented Jan 25, 2023

wajda commented Jan 25, 2023

wajda commented Jan 25, 2023

tiguanin commented Jan 25, 2023

cerveada commented Jan 26, 2023

wajda commented Jan 26, 2023

wajda commented Jan 26, 2023

IncompatibleClassChangeError: class z.c.a.s.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead #579

IncompatibleClassChangeError: class z.c.a.s.harvester.plugin.embedded.DeltaPlugin$SyntheticDeltaRead #579

Comments

tiguanin commented Jan 25, 2023 • edited by cerveada Loading

cerveada commented Jan 25, 2023

tiguanin commented Jan 25, 2023 • edited Loading

cerveada commented Jan 25, 2023

tiguanin commented Jan 25, 2023

tiguanin commented Jan 25, 2023

wajda commented Jan 25, 2023

wajda commented Jan 25, 2023

tiguanin commented Jan 25, 2023

cerveada commented Jan 26, 2023

wajda commented Jan 26, 2023

wajda commented Jan 26, 2023

tiguanin commented Jan 25, 2023 •

edited by cerveada

Loading

tiguanin commented Jan 25, 2023 •

edited

Loading