New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caused by: java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved #1953

Closed
maqianghua opened this Issue Mar 14, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@maqianghua
Copy link

maqianghua commented Mar 14, 2018

Logged in as: dr.who
Application

About
Jobs 

Tools

Log Type: stderr

Log Upload Time: Wed Mar 14 14:54:06 +0800 2018

Log Length: 16143

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop-mapreduce/alluxio-1.6.1-default-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/03/14 14:54:01 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
18/03/14 14:54:01 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1517218274663_1124_000002
18/03/14 14:54:02 INFO spark.SecurityManager: Changing view acls to: maqianghua
18/03/14 14:54:02 INFO spark.SecurityManager: Changing modify acls to: maqianghua
18/03/14 14:54:02 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(maqianghua); users with modify permissions: Set(maqianghua)
18/03/14 14:54:02 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
18/03/14 14:54:02 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
18/03/14 14:54:02 INFO cli.ADAMMain: ADAM invoked with args: "transformAlignments" "/user/maqianghua/GATK/GATKinput/bam_8/NA12878.sam" "/user/maqianghua/adam_1/bam_output/alignment.adam_1"
18/03/14 14:54:02 INFO spark.SparkContext: Running Spark version 1.6.0
18/03/14 14:54:03 INFO spark.SecurityManager: Changing view acls to: maqianghua
18/03/14 14:54:03 INFO spark.SecurityManager: Changing modify acls to: maqianghua
18/03/14 14:54:03 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(maqianghua); users with modify permissions: Set(maqianghua)
18/03/14 14:54:03 INFO util.Utils: Successfully started service 'sparkDriver' on port 18518.
18/03/14 14:54:03 INFO slf4j.Slf4jLogger: Slf4jLogger started
18/03/14 14:54:03 INFO Remoting: Starting remoting
18/03/14 14:54:03 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.1.0.64:6422]
18/03/14 14:54:03 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.1.0.64:6422]
18/03/14 14:54:03 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 6422.
18/03/14 14:54:03 INFO spark.SparkEnv: Registering MapOutputTracker
18/03/14 14:54:03 INFO spark.SparkEnv: Registering BlockManagerMaster
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sda/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-33b6cb65-b1f7-4cf2-8b27-1aea844f1c35
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdb/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-c593657f-0a4e-4046-9ca6-23c5346b4055
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdc/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-b2ea137a-77de-4422-9788-505d4b44cb88
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdd/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-bfa7a260-d511-4d0d-a66e-ba6b5439ec75
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sde/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-fb3103e4-c8a6-44f5-b83e-944e0b51bb34
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdg/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-cb0b8fe4-a5e9-4f42-8da7-a0ada4e64513
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdh/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-7e2324d3-981a-4f45-b937-757afa9ad57a
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdi/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-fd401537-9550-4a33-90e2-f8f48abc7f32
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdj/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-d6f4e2b6-bea9-4b77-970e-38d26ab88a1a
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdk/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-ede5eb4c-eb52-42b7-b10a-4a9bca3cc819
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdl/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-1da90118-841b-4723-a769-902f3b0e848b
18/03/14 14:54:03 INFO storage.DiskBlockManager: Created local directory at /export/sdf/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/blockmgr-45991bff-75a4-4a8d-b5a4-dffa262bdd91
18/03/14 14:54:03 INFO storage.MemoryStore: MemoryStore started with capacity 25.4 GB
18/03/14 14:54:03 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/03/14 14:54:04 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
18/03/14 14:54:04 INFO util.Utils: Successfully started service 'SparkUI' on port 10051.
18/03/14 14:54:04 INFO ui.SparkUI: Started SparkUI at http://10.1.0.64:10051
18/03/14 14:54:04 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
18/03/14 14:54:04 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 31598.
18/03/14 14:54:04 INFO netty.NettyBlockTransferService: Server created on 31598
18/03/14 14:54:04 INFO storage.BlockManager: external shuffle service port = 7337
18/03/14 14:54:04 INFO storage.BlockManagerMaster: Trying to register BlockManager
18/03/14 14:54:04 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.1.0.64:31598 with 25.4 GB RAM, BlockManagerId(driver, 10.1.0.64, 31598)
18/03/14 14:54:04 INFO storage.BlockManagerMaster: Registered BlockManager
18/03/14 14:54:04 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.fasterxml.jackson.module.scala.ser.IteratorSerializerModule$class.$init$(IteratorSerializerModule.scala:70)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala:34)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala)
at org.apache.spark.util.JsonProtocol$.(JsonProtocol.scala:59)
at org.apache.spark.util.JsonProtocol$.(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:297)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:131)
at org.apache.spark.SparkContext.(SparkContext.scala:557)
at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:52)
at org.bdgenomics.adam.cli.TransformAlignments.run(TransformAlignments.scala:138)
at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:126)
at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:65)
at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
18/03/14 14:54:04 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;)
18/03/14 14:54:04 ERROR yarn.ApplicationMaster: Uncaught exception:
java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:570)
Caused by: java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.fasterxml.jackson.module.scala.ser.IteratorSerializerModule$class.$init$(IteratorSerializerModule.scala:70)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala:34)
at com.fasterxml.jackson.module.scala.DefaultScalaModule$.(DefaultScalaModule.scala)
at org.apache.spark.util.JsonProtocol$.(JsonProtocol.scala:59)
at org.apache.spark.util.JsonProtocol$.(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:297)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:131)
at org.apache.spark.SparkContext.(SparkContext.scala:557)
at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:52)
at org.bdgenomics.adam.cli.TransformAlignments.run(TransformAlignments.scala:138)
at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:126)
at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:65)
at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)
18/03/14 14:54:04 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase;)
18/03/14 14:54:04 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1517218274663_1124
18/03/14 14:54:04 INFO storage.DiskBlockManager: Shutdown hook called
18/03/14 14:54:04 INFO util.ShutdownHookManager: Shutdown hook called
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdh/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-5dc19f66-4d38-4551-93f9-c165de22a649
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdj/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-11699572-1fcb-45f6-bac5-fc406cc563e3
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdi/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-fca56a9d-5139-474a-b7f3-9f28f37dce37
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdk/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-41c66c42-cbc8-4d6f-8211-ec3e44eaf009
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdd/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-48b9009d-a0f0-4deb-bd0a-5cf6ac129051
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sde/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-11d3335d-8c4c-40fe-ba14-f26e8bc9497e
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sda/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-1a1b671a-ab16-41d0-bfa9-f0f0247e353e/userFiles-bc3f9fcd-cbaf-4d4d-bee9-bae4d924b81a
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdc/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-279a04cc-da81-4ac8-b437-2542d502e02b
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sda/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-1a1b671a-ab16-41d0-bfa9-f0f0247e353e
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdb/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-d35d9bdc-3a7d-4a2c-8926-52d4dc01119d
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdg/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-0b0d0273-87b8-4343-905c-e8a9f46dc634
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdl/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-8da77191-bfac-4a53-b9e2-2351b6fbfadc
18/03/14 14:54:04 INFO util.ShutdownHookManager: Deleting directory /export/sdf/yarn/nm/usercache/maqianghua/appcache/application_1517218274663_1124/spark-e4711a9b-9b90-4a9e-8a38-47cc079954aa

@maqianghua

This comment has been minimized.

Copy link

maqianghua commented Mar 14, 2018

Did anyone meet this question?when I run ADAM transformAlignments ,and I always meet this error .my ADAM version is 0.23.0 and Built for: Apache Spark 1.6.0, Scala 2.10.5, and Hadoop 2.6.0. I whill appreciate your reply to this quetion.

@fnothaft

This comment has been minimized.

Copy link
Member

fnothaft commented Mar 21, 2018

Hi @maqianghua! We've dropped support for Spark 1.6.0 and Scala 2.10 since ADAM 0.23.0. Does this error reproduce with Spark 2.x and Scala 2.11?

@fnothaft

This comment has been minimized.

Copy link
Member

fnothaft commented Jul 6, 2018

Closing this issue as we haven't had an update. Please feel free to reopen @maqianghua if you are able to reproduce on Spark 2.x/Scala 2.11.

@fnothaft fnothaft closed this Jul 6, 2018

@heuermh heuermh added this to the 0.24.1 milestone Aug 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment