GLIBCXX_3.4.9' not found

[naven084k]

Hi,
i am running hadoop 2.4 cluster. when trying to execute a job it is giving following error

Caused by: java.lang.UnsatisfiedLinkError: /tmp/hadoop-npokala/nm-local-dir/usercache/npokala/appcache/application_1415889688781_0001/container_1415889688781_0001_01_000001/tmp/snappy-1.0.5.3-cd779f87-74a6-4403-a2d3-999009f3b57b-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /tmp/hadoop-npokala/nm-local-dir/usercache/npokala/appcache/application_1415889688781_0001/container_1415889688781_0001_01_000001/tmp/snappy-1.0.5.3-cd779f87-74a6-4403-a2d3-999009f3b57b-libsnappyjava.so)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1929)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1814)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1083)
at org.xerial.snappy.SnappyNativeLoader.load(SnappyNativeLoader.java:39).

I am breaking my head with the issue. Please help me..
i don't have permissions to upgrade GLIBCXX_3.4.9 version. i tried to change snappy jar versions like 1.0.3 and 1.1.1.6 in the following paths

/home/npokala/hadoop/share/hadoop/common/lib/snappy-java-1.1.0.jar
/home/npokala/hadoop/share/hadoop/tools/lib/snappy-java-1.1.0.jar
/home/npokala/hadoop/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/snappy-java-1.1.0.jar
/home/npokala/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.1.0.jar

and restarted the hadoop cluster i am receiving the same error (it mentioning same version 1.0.5.3 snappy).

[xerial]

Are you using Linux?

If so, please use the latest version, which doesn't require glibc+

[naven084k]

Do you mean linux latest version?

Server which i am using is a busy server, i cannot upgrade it.

Any other solution?

[xerial]

No. Using snappy-java-1.0.5.4 or 1.1.1.6, which embeds libstdc++ (GLIBCXX), is the only solution.

[naven084k]

if we downgrade snappy-java version is that works?

[xerial]

not sure. is there any problem in using 1.0.5.4 or 1.1.1.6?

[naven084k]

I have tried all the versions you mentioned. all of them giving the same error.

[xerial]

Please paste the log (e.g., when using 1.1.1.6). It should not be the same since the version of native library is different.

And also let me know your os.name and os.arch JVM properties.

[naven084k]

I am pasting the entire log..................................sorry for that...

14/11/14 05:07:27 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
14/11/14 05:07:28 INFO spark.SparkContext: Starting job: saveAsTextFile at SparkJob.java:27
14/11/14 05:07:28 INFO scheduler.DAGScheduler: Got job 0 (saveAsTextFile at SparkJob.java:27) with 48 output partitions (allowLocal=false)
14/11/14 05:07:28 INFO scheduler.DAGScheduler: Final stage: Stage 0(saveAsTextFile at SparkJob.java:27)
14/11/14 05:07:28 INFO scheduler.DAGScheduler: Parents of final stage: List()
14/11/14 05:07:28 INFO scheduler.DAGScheduler: Missing parents: List()
14/11/14 05:07:28 INFO scheduler.DAGScheduler: Submitting Stage 0 (MappedRDD[1] at saveAsTextFile at SparkJob.java:27), which has no missing parents
14/11/14 05:07:28 INFO storage.MemoryStore: ensureFreeSpace(57136) called with curMem=0, maxMem=2061647216
14/11/14 05:07:28 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 55.8 KB, free 1966.1 MB)
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:319)
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:226)
at org.xerial.snappy.Snappy.(Snappy.java:48)
at org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:79)
at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125)
at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:207)
at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:83)
at org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:68)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:36)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:809)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:829)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:769)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:753)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1360)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.UnsatisfiedLinkError: /tmp/hadoop-npokala/nm-local-dir/usercache/npokala/appcache/application_1415889688781_0008/container_1415889688781_0008_01_000001/tmp/snappy-1.0.5.3-4de28674-1225-4ffe-847b-872e1f400ac1-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /tmp/hadoop-npokala/nm-local-dir/usercache/npokala/appcache/application_1415889688781_0008/container_1415889688781_0008_01_000001/tmp/snappy-1.0.5.3-4de28674-1225-4ffe-847b-872e1f400ac1-libsnappyjava.so)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1929)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1814)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1083)
at org.xerial.snappy.SnappyNativeLoader.load(SnappyNativeLoader.java:39)
... 29 more
14/11/14 05:07:28 INFO cluster.YarnClusterScheduler: Cancelling stage 0
14/11/14 05:07:28 INFO scheduler.DAGScheduler: Failed to run saveAsTextFile at SparkJob.java:27
14/11/14 05:07:28 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED
14/11/14 05:07:28 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
14/11/14 05:07:28 INFO yarn.ApplicationMaster: All executors have launched.
14/11/14 05:07:28 INFO yarn.ApplicationMaster: Started progress reporter thread - heartbeat interval : 5000
Exception in thread "Driver" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:162)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:236)
org.xerial.snappy.Snappy.(Snappy.java:48)
org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:79)
org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125)
org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:207)
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:83)
org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:68)
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:36)
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
org.apache.spark.SparkContext.broadcast(SparkContext.scala:809)
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:829)
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:769)
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:753)
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1360)
akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
akka.actor.ActorCell.invoke(ActorCell.scala:456)
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
akka.dispatch.Mailbox.run(Mailbox.scala:219)
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:837)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:769)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:753)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1360)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

14/11/14 05:07:28 INFO yarn.ApplicationMaster$$anon$1: Invoking sc stop from shutdown hook
14/11/14 05:07:28 INFO yarn.ApplicationMaster: AppMaster received a signal.
14/11/14 05:07:28 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1415889688781_0008
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/metrics/json,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/kill,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/static,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors/json,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/environment/json,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/environment,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/rdd/json,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/rdd,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage/json,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/storage,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/pool/json,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/pool,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/json,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/json,null}
14/11/14 05:07:28 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages,null}
14/11/14 05:07:28 INFO ui.SparkUI: Stopped Spark web UI at http://slave.mhf.mhc:50878
14/11/14 05:07:28 INFO scheduler.DAGScheduler: Stopping DAGScheduler
14/11/14 05:07:28 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
14/11/14 05:07:28 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
14/11/14 05:07:29 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
14/11/14 05:07:29 INFO network.ConnectionManager: Selector thread was interrupted!
14/11/14 05:07:29 INFO network.ConnectionManager: ConnectionManager stopped
14/11/14 05:07:29 INFO storage.MemoryStore: MemoryStore cleared
14/11/14 05:07:29 INFO storage.BlockManager: BlockManager stopped
14/11/14 05:07:29 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
14/11/14 05:07:29 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
14/11/14 05:07:29 INFO spark.SparkContext: Successfully stopped SparkContext

os name : Linux
os version : 2.6.18-308.
Architecture : x86_64

[naven084k]

OS details:

os name : Linux
os version : 2.6.18-308.
Architecture : x86_64

Thanks in advance...

[xerial]

the log shows it uses snappy-java-1.0.5.3, which doesn't have embedded glibc++. I guess you need to upgrade the version of spark since it also uses snappy-java.

[naven084k]

Ohhh..It is working..what you suggested is exactly correct.Thanks

[xerial]

I'm glad to know that :)