Connection refused errors when transforming BAM file with BQSR #516

Closed
ansalaza opened this Issue Dec 1, 2014 · 7 comments

Comments

Projects
None yet
5 participants
@ansalaza

ansalaza commented Dec 1, 2014

I am trying to transform a NA12878 BAM file (~300gb) with base quality score recalibration. I am running this on standalone mode (through a cluster with 1 master and 4 workers). After ~30min into this process, I get "java.net.ConnectException: Connection refused" errors.

It is important to note that I can successfully transform the BAM file to ADAM format as long as I don't turn on the recalibration parameter. I am also able to transform a shortened version of a NA12878 SAM (~250kb) file with base quality score recalibration.

Any ideas on why this error persists?

I've provided my spark cluster configurations and most of the stack trace message below.

------- cluster specs -------
1 master (174 g of memory)
4 workers (each with 240g of memory, 31 cores)

------- ~/spark-1.1.0-bin-hadoop2.3/conf/spark-env.sh -------
export SPARK_DAEMON_MEMORY=100g
export SPARK_WORKER_CORES=30
export SPARK_WORKER_MEMORY=220g

------ stack trace -----
-bash-4.1$ adam transform /pod/pstore/projects/BigDataGenomics/NA12878.hiseq.wgs.bwa.raw.bam /pod/pstore/projects/BigDataGenomics/NA12878.recal.adam -known_snps /pod/pstore/projects/BigDataGenomics/All.suspectedremoved.vcf -recalibrate_base_qualities
Spark assembly has been built with Hive, including Datanucleus jars on classpath
2014-12-01 11:30:44 WARN SparkConf:71 - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
2014-12-01 11:30:46 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-12-01 11:46:45 WARN BlockManagerMasterActor:71 - Removing BlockManager BlockManagerId(2, podk-1-2.local, 41993, 0) with no recent heart beats: 60946ms exceeds 45000ms
2014-12-01 11:46:47 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-1.local,48288) not found
2014-12-01 11:46:47 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-2.local,41993) not found
2014-12-01 11:46:47 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-3.local,50288) not found
2014-12-01 11:46:48 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-4.local,49200) not found
2014-12-01 11:46:48 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-4.local,49200) not found
2014-12-01 11:46:49 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:46:57 WARN TaskSetManager:71 - Lost task 13.0 in stage 2.0 (TID 15, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:04 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 108.0 in stage 2.0 (TID 110, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 118.0 in stage 2.0 (TID 120, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 78.0 in stage 2.0 (TID 80, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 90.0 in stage 2.0 (TID 92, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 71.0 in stage 2.0 (TID 73, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 60.0 in stage 2.0 (TID 62, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 27.0 in stage 2.0 (TID 29, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 41.0 in stage 2.0 (TID 43, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 96.0 in stage 2.0 (TID 98, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 107.0 in stage 2.0 (TID 109, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 36.0 in stage 2.0 (TID 38, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 45.0 in stage 2.0 (TID 47, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 61.0 in stage 2.0 (TID 63, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 65.0 in stage 2.0 (TID 67, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 103.0 in stage 2.0 (TID 105, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 67.0 in stage 2.0 (TID 69, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 88.0 in stage 2.0 (TID 90, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 77.0 in stage 2.0 (TID 79, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 119.0 in stage 2.0 (TID 121, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 99.0 in stage 2.0 (TID 101, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 39.0 in stage 2.0 (TID 41, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 79.0 in stage 2.0 (TID 81, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 80.0 in stage 2.0 (TID 82, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 42.0 in stage 2.0 (TID 44, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 58.0 in stage 2.0 (TID 60, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 115.0 in stage 2.0 (TID 117, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 56.0 in stage 2.0 (TID 58, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 112.0 in stage 2.0 (TID 114, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 72.0 in stage 2.0 (TID 74, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 46.0 in stage 2.0 (TID 48, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 43.0 in stage 2.0 (TID 45, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 84.0 in stage 2.0 (TID 86, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 68.0 in stage 2.0 (TID 70, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 28.0 in stage 2.0 (TID 30, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 55.0 in stage 2.0 (TID 57, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 75.0 in stage 2.0 (TID 77, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 111.0 in stage 2.0 (TID 113, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 116.0 in stage 2.0 (TID 118, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 105.0 in stage 2.0 (TID 107, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 86.0 in stage 2.0 (TID 88, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 93.0 in stage 2.0 (TID 95, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 26.0 in stage 2.0 (TID 28, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 104.0 in stage 2.0 (TID 106, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 89.0 in stage 2.0 (TID 91, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 100.0 in stage 2.0 (TID 102, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 66.0 in stage 2.0 (TID 68, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 57.0 in stage 2.0 (TID 59, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 82.0 in stage 2.0 (TID 84, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 30.0 in stage 2.0 (TID 32, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 70.0 in stage 2.0 (TID 72, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 69.0 in stage 2.0 (TID 71, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 38.0 in stage 2.0 (TID 40, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 74.0 in stage 2.0 (TID 76, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 110.0 in stage 2.0 (TID 112, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 114.0 in stage 2.0 (TID 116, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 54.0 in stage 2.0 (TID 56, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 50.0 in stage 2.0 (TID 52, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 102.0 in stage 2.0 (TID 104, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 98.0 in stage 2.0 (TID 100, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 34.0 in stage 2.0 (TID 36, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 106.0 in stage 2.0 (TID 108, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 33.0 in stage 2.0 (TID 35, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 94.0 in stage 2.0 (TID 96, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 37.0 in stage 2.0 (TID 39, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 109.0 in stage 2.0 (TID 111, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 9.0 in stage 2.0 (TID 11, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 101.0 in stage 2.0 (TID 103, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 85.0 in stage 2.0 (TID 87, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 73.0 in stage 2.0 (TID 75, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 81.0 in stage 2.0 (TID 83, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 97.0 in stage 2.0 (TID 99, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 117.0 in stage 2.0 (TID 119, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 113.0 in stage 2.0 (TID 115, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 29.0 in stage 2.0 (TID 31, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 1.0 in stage 2.0 (TID 3, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 49.0 in stage 2.0 (TID 51, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 53.0 in stage 2.0 (TID 55, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 ERROR TaskSchedulerImpl:75 - Lost executor 1 on podk-1-1.local: remote Akka client disassociated
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 135.0 in stage 2.0 (TID 137, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 39.1 in stage 2.0 (TID 200, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 153.0 in stage 2.0 (TID 155, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 126.0 in stage 2.0 (TID 128, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 56.1 in stage 2.0 (TID 194, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 77.1 in stage 2.0 (TID 185, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 138.0 in stage 2.0 (TID 140, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 147.0 in stage 2.0 (TID 149, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 132.0 in stage 2.0 (TID 134, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 119.1 in stage 2.0 (TID 184, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 149.0 in stage 2.0 (TID 151, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 158.0 in stage 2.0 (TID 160, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 161.0 in stage 2.0 (TID 163, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 125.0 in stage 2.0 (TID 127, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 84.1 in stage 2.0 (TID 199, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 134.0 in stage 2.0 (TID 136, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 137.0 in stage 2.0 (TID 139, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 42.1 in stage 2.0 (TID 189, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 103.1 in stage 2.0 (TID 180, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 96.1 in stage 2.0 (TID 174, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 88.1 in stage 2.0 (TID 183, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 75.1 in stage 2.0 (TID 201, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 99.1 in stage 2.0 (TID 186, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 166.0 in stage 2.0 (TID 168, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 148.0 in stage 2.0 (TID 150, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 121.0 in stage 2.0 (TID 123, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 90.1 in stage 2.0 (TID 171, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 151.0 in stage 2.0 (TID 153, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 160.0 in stage 2.0 (TID 162, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 65.1 in stage 2.0 (TID 179, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 ERROR TaskSchedulerImpl:75 - Lost executor 2 on podk-1-2.local: remote Akka client disassociated
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 74.1 in stage 2.0 (TID 218, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 129.0 in stage 2.0 (TID 131, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 34.1 in stage 2.0 (TID 226, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 100.1 in stage 2.0 (TID 211, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 164.0 in stage 2.0 (TID 166, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 114.1 in stage 2.0 (TID 220, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 140.0 in stage 2.0 (TID 142, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 102.1 in stage 2.0 (TID 223, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 105.1 in stage 2.0 (TID 205, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 30.1 in stage 2.0 (TID 214, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 38.1 in stage 2.0 (TID 217, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 79.1 in stage 2.0 (TID 190, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 69.1 in stage 2.0 (TID 225, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 86.1 in stage 2.0 (TID 207, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 66.1 in stage 2.0 (TID 216, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 145.0 in stage 2.0 (TID 147, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 106.1 in stage 2.0 (TID 228, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 127.0 in stage 2.0 (TID 129, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 110.1 in stage 2.0 (TID 219, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 163.0 in stage 2.0 (TID 165, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 82.1 in stage 2.0 (TID 213, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 50.1 in stage 2.0 (TID 222, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 72.1 in stage 2.0 (TID 195, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 142.0 in stage 2.0 (TID 144, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 124.0 in stage 2.0 (TID 126, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 54.1 in stage 2.0 (TID 221, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 57.1 in stage 2.0 (TID 212, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 80.1 in stage 2.0 (TID 188, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 141.0 in stage 2.0 (TID 143, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 168.0 in stage 2.0 (TID 170, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 ERROR TaskSchedulerImpl:75 - Lost executor 3 on podk-1-3.local: remote Akka client disassociated

@massie

This comment has been minimized.

Show comment
Hide comment
@massie

massie Dec 1, 2014

Member

Are you running ADAM 0.15.0?

Member

massie commented Dec 1, 2014

Are you running ADAM 0.15.0?

@ansalaza

This comment has been minimized.

Show comment
Hide comment
@ansalaza

ansalaza Dec 1, 2014

No, I am running ADAM 0.14.0. I wasn't aware of the 0.15.0 release. I can try using this version, perhaps this issue may have been resolved in the latest release.

ansalaza commented Dec 1, 2014

No, I am running ADAM 0.14.0. I wasn't aware of the 0.15.0 release. I can try using this version, perhaps this issue may have been resolved in the latest release.

@massie

This comment has been minimized.

Show comment
Hide comment
@massie

massie Dec 1, 2014

Member

Let us know what you find either way. If I were to guess, the issues you're seeing with disconnecting services has to do with them being overwhelmed by GC. You should find that 0.15.0 uses much less memory.

https://github.com/bigdatagenomics/adam/releases/tag/adam-parent-0.15.0

Member

massie commented Dec 1, 2014

Let us know what you find either way. If I were to guess, the issues you're seeing with disconnecting services has to do with them being overwhelmed by GC. You should find that 0.15.0 uses much less memory.

https://github.com/bigdatagenomics/adam/releases/tag/adam-parent-0.15.0

@ansalaza

This comment has been minimized.

Show comment
Hide comment
@ansalaza

ansalaza Dec 5, 2014

Following up on this issue, I still get the same connection refused errors even with the 0.15.0 release.

I've attached a screen shot of the Spark Web UI right before it crashes.

adam-transform-screenshot-20141205

ansalaza commented Dec 5, 2014

Following up on this issue, I still get the same connection refused errors even with the 0.15.0 release.

I've attached a screen shot of the Spark Web UI right before it crashes.

adam-transform-screenshot-20141205

@alartin

This comment has been minimized.

Show comment
Hide comment
@alartin

alartin Dec 19, 2014

From the errors shown above, it should be a problem of your spark cluster setting. Have you tried a wordcount test on a large file on your spark cluster? If it failed, it may not be related to adam.

alartin commented Dec 19, 2014

From the errors shown above, it should be a problem of your spark cluster setting. Have you tried a wordcount test on a large file on your spark cluster? If it failed, it may not be related to adam.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Mar 24, 2016

Member

Would it be ok to close this issue as unable to reproduce?

Member

heuermh commented Mar 24, 2016

Would it be ok to close this issue as unable to reproduce?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 24, 2016

Member

Yes.

Member

fnothaft commented Mar 24, 2016

Yes.

@fnothaft fnothaft closed this Mar 24, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment