NoSuchMethodError (ScalaObjectMapper)

[dvanbrug]

Issue

When trying to use the Flight Recorder functionality, I get the following error:

java.lang.NoSuchMethodError: com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper.$init$(Lcom/fasterxml/jackson/module/scala/experimental/ScalaObjectMapper;)V
        at ch.cern.sparkmeasure.IOUtils$$anon$1.<init>(ioutils.scala:22)
        at ch.cern.sparkmeasure.IOUtils$.<init>(ioutils.scala:22)
        at ch.cern.sparkmeasure.IOUtils$.<clinit>(ioutils.scala)
        at ch.cern.sparkmeasure.FlightRecorderStageMetrics.onApplicationEnd(flightrecorder.scala:56)
        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:57)
        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
        at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
        at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
        at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
        at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1404)
        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)

Reproduce

To reproduce, checkout the latest version of the code and run:

spark-submit --master local[*] --jars ./target/scala-2.12/spark-measure_2.12-0.18-SNAPSHOT.jar --conf spark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics ~/pyspark_helloworld.py

where pyspark_helloworld.py is just:

import pyspark
sc = pyspark.SparkContext('local[*]')

txt = sc.textFile('file:////usr/share/doc/python-2.7.5/README')
print(txt.count())

python_lines = txt.filter(lambda line: 'python' in line.lower())
print(python_lines.count())

I am using the following versions:

• Scala 2.12
• PySpark 3.2.0
• Python 3.7.11

Fix

Bumping the version of jackson-module-scala to 2.13.0 and rebuilding seems to fix the problem.

[LucaCanali]

Thanks for reporting this. I have applied the proposed fix.

[santvasu]

HI Luca Canali

i am trying spark-measure_2.12-0.17.jar with the example program mentioned in the post. and as suggested i am using jackson-module-scala_2.13-2.13.1.jar, and i am still getting the same exception.

Note: jackson* jars and dependencies were manually downloaded from jar-download.com

• Spark:3.1.2, Scala:2.12, Python:3.8.8
• jars jackson-annotations-2.13.1.jar,jackson-core-2.13.1.jar,jackson-databind-2.13.1.jar,jackson-module-scala_2.13-2.13.1.jar,paranamer-2.8.jar,scala-library-2.13.6.jar,spark-measure_2.12-0.17.jar

Command used:
spark-submit --master local[*] --jars ./spark-measure_2.12-0.17. jar,./jackson-annotations-2.13.1.jar,./jackson-core-2.13.1.jar,./jackson-databind-2.13.1.jar,./jackson-module-scala_2.13-2.13.1.jar,./paranamer-2.8.jar,./scala-library-2.13.6.jar --conf sp ark.extraListeners=ch.cern.sparkmeasure.FlightRecorderStageMetrics ./hello_world.py

pl do let me know if i am missing someting.

thanks & regards
santhosh

Correction...

I am sorry.. i built spark-measure_2.12-0.18-SNAPSHOT.jar from the github and its working fine. i.e its printing the statistics as expected.

i will try building spark-measure_2.12-0.17.jar from source and update the status.

thanks and regards
santhosh

[LucaCanali]

Hi @santvasu thanks for testing and reporting this.
This reminds me that I should release version 0.18 with the latest fixes.