Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Connection refused errors when transforming BAM file with BQSR #516
I am trying to transform a NA12878 BAM file (~300gb) with base quality score recalibration. I am running this on standalone mode (through a cluster with 1 master and 4 workers). After ~30min into this process, I get "java.net.ConnectException: Connection refused" errors.
It is important to note that I can successfully transform the BAM file to ADAM format as long as I don't turn on the recalibration parameter. I am also able to transform a shortened version of a NA12878 SAM (~250kb) file with base quality score recalibration.
Any ideas on why this error persists?
I've provided my spark cluster configurations and most of the stack trace message below.
------- cluster specs -------
------- ~/spark-1.1.0-bin-hadoop2.3/conf/spark-env.sh -------
------ stack trace -----
Let us know what you find either way. If I were to guess, the issues you're seeing with disconnecting services has to do with them being overwhelmed by GC. You should find that 0.15.0 uses much less memory.