No description, website, or topics provided.
Scala C Java Perl C++ Shell Other
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src
README.md
compile.pl
license.md
pom.xml
run_test.sh

README.md

Cloud-Scale BWAMEM

Introduction

Cloud-scale BWAMEM (CS-BWAMEM) is an ultrafast and highly scalable aligner built on top of cloud infrastructures, including Spark and Hadoop distributed file system (HDFS). It leverages the abundant computing resources in a public or private cloud to fully exploit the parallelism obtained from the enormous number of reads. With CSBWAMEM, the pair-end whole-genome reads (30x) can be aligned within 80 minutes in a 25-node cluster with 300 cores.

Build and Install

  1. git clone git@github.com:ytchen0323/cloud-scale-bwamem.git

  2. cd cloud-scale-bwamem

  3. updated absolute path in two pom.xml: src/pom.xml and src/main/jni_fpga/pom.xml
    update:

     <systemPath>/curr/pengwei/github/cloud-scale-bwamem/target/cloud-scale-bwamem-0.2.2.jar</systemPath>
    

    to your path:

     <systemPath>/youpath/cloud-scale-bwamem/target/cloud-scale-bwamem-0.2.2.jar</systemPath>
    
  4. ./compile.pl

Upload FASTQ file(s) to HDFS

  • commands: upload-fastq [-bn INT] isPairEnd filePartitionNum inputFASTQFilePath1 [inputFASTQFilePath2] outFileHDFSPath

  • Required arguments (in the following order):

    (1) isPairEnd:

    1: pair-end
    
    0: single-end (not fully verified yet)
    

    (2) inputFASTQFilePath1: the first input path of the FASTQ file in the local file system (for both single-end and pair-end)

    (3) inputFASTQFilePath2: (optional) the second input path of the FASTQ file in the local file system (for pair-end)

    (4) outFileHDFSPath: the root path of the output FASTQ files in HDFS

  • Optional arguments: (1) -bn (optional): the number of lines to be read in one batch, which depends on the DRAM you have on the driver node.

Use CS-BWAMEM aligner

  • commands: cs-bwamem [-bfn INT] [-bPSW (0/1)] [-sbatch INT] [-bPSWJNI (0/1)] [-jniPath STRING] [-oType (0/1/2)] [-oPath STRING] isPairEnd fastaInputPath fastqHDFSInputPath fastqInputFolderNum

  • Required arguments (in the following order):

    (1) isPairEnd:

    1: pair-end
    
    0: single-end (not fully verified yet)
    

    (2) fastaInputPath: the path of BWA index files (bns, pac, and so on). This path is locate at local machine instead of HDFS.

    (3) fastqHDFSInputPath: the path of the raw read files stored in HDFS

    (4) fastqInputFolderNum: the number of folders generated in the HDFS for the raw reads (output from Usage1). (NOTE: this parameter can be automatically fetched in the next version)

  • Optional arguments:

    (1) -bfn (optional): the number of folders of raw reads to be processed in a batch

    (2) -bPSW (optional): whether the pair-end Smith Waterman is performed in a batched way

    (3) -sbatch (optional): the number of reads to be processed in a subbatch using JNI library

    (4) -bPSWJNI (optional): whether the native JNI library is called for better performance

    (5) -jniPath (optional): the JNI library path in the local machine

    (6) -oChoice (optional): the output format choice

    0: no output (pure computation)
    
    1: SAM file output in the local file system (default)
    
    2: ADAM format output in the distributed file system
    

    (7) -oPath (optional): the output path; users need to provide correct path in the local or distributed file system

Merge the output ADAM folders

  • This command merges the output ADAM folders after alignment and then save the output as a new ADAM file in HDFS
  • commands: merge adamHDFSRootInputPath adamHDFSOutputPath

Sort the output ADAM folders

  • This commands sorts the output ADAM folders after alignment and then save the output as a new ADAM file in HDFS
  • commands: sort adamHDFSRootInputPath adamHDFSOutputPath