Migrating wiki contents from Google Code

Compiling.md

@@ -0,0 +1,36 @@
+# Introduction #
+
+Compiling & installing.
+
+
+# Details #
+
+You should be able to run
+
+```
+> make
+```
+and
+```
+> make test
+```
+
+... to install.
+
+Some caveats are:
+
+The sparse hash library may need updating.
+
+The new version of varcall requires the GNU scientific library to be installed in order to compile.
+
+On UBUNTU :
+
+apt-get install libgsl0-dev
+
+On CENTOS/REDHAT :
+
+rpm -i gsl-devel
+
+On WINDOWS:
+
+Use MinGW, and use the [Windows port of GSL](http://gnuwin32.sourceforge.net/packages/gsl.htm)

FastqJoin.md

@@ -0,0 +1,58 @@
+# Usage #
+
+```
+Usage: fastq-join [options] <read1.fq> <read2.fq> [mate.fq] -o <read.%.fq>
+
+Joins two paired-end reads on the overlapping ends.
+
+Options:
+
+-o FIL          See 'Output' below
+-v C            Verifies that the 2 files probe id's match up to char C
+                  use '/' for Illumina reads
+-p N            N-percent maximum difference (8)
+-m N            N-minimum overlap (6)
+-r FIL          Verbose stitch length report
+-R              No reverse complement
+-V              Show version
+
+Output:
+
+  You can supply 3 -o arguments, for un1, un2, join files, or one
+argument as a file name template.  The suffix 'un1, un2, or join' is
+appended to the file, or they replace a %-character if present.
+
+  If a 'mate' input file is present (barcode read), then the files
+'un3' and 'join2' are also created.
+
+  Files named ".gz" are assumed to be compressed, and can be 
+read/written as long as "gzip" is in the path.
+```
+
+## Etc ##
+
+This uses our sqr(distance)/len for anchored alignment quality algorithm.  It's a good measure of anchored alignment quality, akin (in my mind) to squared-deviation for means.
+
+## Overlapping Bases ##
+
+### When the bases match ###
+
+The higher quality base is used, and it is increased by up to 3
+
+### When the bases don't match ###
+
+If one quality is greater than "3" (50%), then the the resulting quality is the difference between the two qualities (reduced quality due to mismatch), or "3" )(50%), whichever is greater.
+
+#### Examples: ####
+
+```
+40 vs 3 = 37 : second base has low quality... doesn't change top by much
+40 vs 40 = 3 : two equal quality bases that don't match = qual of 3
+2 vs 2 = 2 : neither base has a high quality
+```
+
+#### Some caveats: ####
+
+Illumina's quality scores are not accurate and estimates vary by chemistry and sequencer.   I would recommend using a profiling tool,  on PhiX, and adjusting your qualities using the results of the tool.
+
+For example the quality score "2" has a true quality, typically of "11" ... this is Illumina's code for "quality estimation failure".   The quality scores at the high end ("34-40") are often overestimates.

FastqMcf.md

@@ -0,0 +1,124 @@
+# Introduction #
+
+fastq-mcf attempts to:
+
+  * Detect & remove sequencing adapters and primers
+  * Detect limited skewing at the ends of reads and clip
+  * Detect poor quality at the ends of reads and clip
+  * Detect Ns, and remove from ends
+  * Remove reads with CASAVA 'Y' flag (purity filtering)
+  * Discard sequences that are too short after all of the above
+  * Keep multiple mate-reads in sync while doing all of the above
+
+# Usage #
+
+```
+
+Usage: fastq-mcf [options] <adapters.fa> <reads.fq> [mates1.fq ...]
+Version: 1.04.636
+
+Detects levels of adapter presence, computes likelihoods and
+locations (start, end) of the adapters.   Removes the adapter
+sequences from the fastq file(s).
+
+Stats go to stderr, unless -o is specified.
+
+Specify -0 to turn off all default settings
+
+If you specify multiple 'paired-end' inputs, then a -o option is
+required for each.  IE: -o read1.clip.q -o read2.clip.fq
+
+Options:
+    -h       This help
+    -o FIL   Output file (stats to stdout)
+    -s N.N   Log scale for adapter minimum-length-match (2.2)
+    -t N     % occurance threshold before adapter clipping (0.25)
+    -m N     Minimum clip length, overrides scaled auto (1)
+    -p N     Maximum adapter difference percentage (10)
+    -l N     Minimum remaining sequence length (19)
+    -L N     Maximum remaining sequence length (none)
+    -D N     Remove duplicate reads : Read_1 has an identical N bases (0)
+    -k N     sKew percentage-less-than causing cycle removal (2)
+    -x N     'N' (Bad read) percentage causing cycle removal (20)
+    -q N     quality threshold causing base removal (10)
+    -w N     window-size for quality trimming (1)
+    -H       remove >95% homopolymer reads (no)
+    -0       Set all default parameters to zero/do nothing
+    -U|u     Force disable/enable Illumina PF filtering (auto)
+    -P N     Phred-scale (auto)
+    -R       Dont remove Ns from the fronts/ends of reads
+    -n       Dont clip, just output what would be done
+    -C N     Number of reads to use for subsampling (300k)
+    -S       Save all discarded reads to '.skip' files
+    -d       Output lots of random debugging stuff
+
+Quality adjustment options:
+    --cycle-adjust    CYC,AMT     Adjust cycle CYC (negative = offset from end) by amount AMT
+    --phred-adjust    SCORE,AMT   Adjust score SCORE by amount AMT
+
+Filtering options*:
+    --[mate-]qual-mean  NUM       Minimum mean quality score
+    --[mate-]qual-gt    NUM,THR   At least NUM quals > THR
+    --[mate-]max-ns     NUM       Maxmium N-calls in a read (can be a %)
+    --[mate-]min-len    NUM       Minimum remaining length (same as -l)
+    --hompolymer-pct    PCT       Homopolymer filter percent (95)
+
+If mate- prefix is used, then applies to second non-barcode read only
+
+Adapter files are 'fasta' formatted:
+
+Specify n/a to turn off adapter clipping, and just use filters
+
+Increasing the scale makes recognition-lengths longer, a scale
+of 100 will force full-length recognition of adapters.
+
+Adapter sequences with _5p in their label will match 'end's,
+and sequences with _3p in their label will match 'start's,
+otherwise the 'end' is auto-determined.
+
+Skew is when one cycle is poor, 'skewed' toward a particular base.
+If any nucleotide is less than the skew percentage, then the
+whole cycle is removed.  Disable for methyl-seq, etc.
+
+Set the skew (-k) or N-pct (-x) to 0 to turn it off (should be done
+for miRNA, amplicon and other low-complexity situations!)
+
+Duplicate read filtering is appropriate for assembly tasks, and
+never when read length < expected coverage.  -D 50 will use
+4.5GB RAM on 100m DNA reads - be careful. Great for RNA assembly.
+
+*Quality filters are evaluated after clipping/trimming
+```
+
+## Notes ##
+
+Adapter file format is fasta.  You can set it to /dev/null, and pass "-f" to do skew detection only.
+
+## Todo ##
+
+  * When discarding one read for being "too short", it has to discard both pairs.   For a sequencing run of normal quality this is not an issue.   It should, though, write "un-mated" reads (whose mate was skipped) to a separate file.  Typically, since these read mates were poor quality, it's not really useful... but it can be for diagnostics.   I've seen runs where these provide valuable data.
+
+  * Like any tool that does many things, fastq-mcf can be limited in it's ability to be flexible.   The biggest missing feature is for it to be able to read files that are formatted like it's stderr output, and use them to guide the process.   Given that feature, fastq-mcf would be complete.
+
+## Notes ##
+
+  * Default settings are probably too conservative when it comes to trimming poor quality/detecting base-skew.
+
+  * It won't trim the "insides" of a paired-end read.   It also will no longer attempt to quality filter a barcode read.   No override for these, but I can't think of a reason to.
+
+  * The -x percentage can be a confusing parameter.   it causes the **entire** cycle to be removed from **all** reads.... even ones without N's... if more than, say, 20% of that cycle is N's.
+
+## Cleaning multiple files ##
+
+Using process substitution, or named pipes, you can clean multiple fastq's in one pass.  This is useful for combining multiple MiSeq runs, or multiple lanes for example:
+
+```
+fastq-mcf \
+  -o cleaned.R1.fq.gz \
+  -o cleaned.R2.fq.gz \
+  adapters.fa \
+  <(gunzip -c uncleaned.lane1.R1.fq.gz uncleaned.lane2.R1.fq.gz;) \
+  <(gunzip -c uncleaned.lane1.R2.fq.gz uncleaned.lane2.R2.fq.gz;)
+```
+
+(Many bioinformatic tools are not "stream friendly",  and some may require the "buffer" command to work.   But fastq-mcf does its own buffering internally.)

FastqMultx.md

@@ -0,0 +1,112 @@
+# Introduction #
+
+The idea behind this is to reduce the amount of "piping" going on in a pipeline.   A lot of time, disk space and nail-chewing is spent keeping files in sync, figuring out what barcodes are on what samples, etc.  The goal of this program is to make it easier to demultiplex possibly paired-end sequences, and also to allow the "guessing" of barcode sets based on master lists of barcoding protocols (fluidigm, truseq, etc.)
+
+# Usage #
+
+```
+Usage: fastq-multx [-g|-l] <barcodes.fil> <read1.fq> -o r1.%.fq [mate.fq -o r2.%.fq] ...
+
+Output files must contain a '%' sign which is replaced with the barcode id in the barcodes file.
+
+Barcodes file looks like this:
+
+<id1> <sequence1>
+<id2> <sequence2> ...
+
+Default is to guess the -bol or -eol based on clear stats.
+
+If -g is used, then it's parameter is an index lane, and frequently occuring sequences are used.
+
+If -l is used then all barcodes in the file are tried, and the *group* with the *most* matches is chosen.
+
+Grouped barcodes file looks like this:
+
+<id1> <sequence1> <group1>
+<id1> <sequence1> <group1>
+<id2> <sequence2> <group2>...
+
+Mated reads, if supplied, are kept in-sync
+
+Options:
+
+-o FIL1 [FIL2]  Output files (one per input, required)
+-g FIL          Determine barcodes from indexed read FIL
+-l FIL          Determine barcodes from any read, using FIL as a master list
+-b              Force beginning of line
+-e              Force end of line
+-x              Don't trim barcodes before writing
+-n              Don't execute, just print likely barcode list
+-v C            Verify that mated id's match up to character C ('/' for illumina)
+-m N            Allow up to N mismatches, as long as they are unique
+```
+
+
+Files named ".gz" are assumed to be compressed, and can be
+read/written as long as "gzip" is in the path.
+
+# Example 1 #
+
+# this example will read/output files that are gzipped, since -B is used and only 1 sequence files is present, it will look for barcodes on the "ends" of the sequence and will tell you which end it found them on
+
+`fastq-multx -B barcodes.fil seq.fastq.gz -o %.fq.gz`
+
+Contents of barcodes.fil:
+
+```
+mock_a ACCC
+salt_a CGTA
+mock_b  GAGT
+salty_b  CGGT
+```
+
+# Example 2 #
+
+# this example will first determine which "barcode group" to use, will select the most likely set of barcodes from that file, and will then proceed as if only that set was specified.   this allows for a single pipeline that works with multiple technologies
+
+`fastq-multx -l barcodes.grp seq2.fastq.gz seq1.fastq.gz -o n/a -o out%.fq`
+
+Contents of barcodes.grp:
+
+```
+id      seq     style
+LB1     ATCACG  TruSeq
+LB2     CGATGT  TruSeq
+LB3     TTAGGC  TruSeq
+LB4     TGACCA  TruSeq
+LB5     ACAGTG  TruSeq
+A01_01  TAGCTTGT        Fluidigm
+B01_02  CGATGTTT        Fluidigm
+C01_03  GCCAATGT        Fluidigm
+D01_04  ACAGTGGT        Fluidigm
+E01_05  ATCACGTT        Fluidigm
+```
+
+Standard error will output:
+
+`Using Barcode Group: TruSeq on File: seq2.fastq.gz (start), Threshold 0.59%`
+
+This indicated that The LB1-LB5 barcodes will be used, and that the filess will be named LB1-LB5, and that the barcode was at the "start" of the reads in the seq2 file.
+
+
+Example of Nextera/Dual-Indexed input:
+
+```
+id      seq     style
+D708_508        TAATGCGC-GTACTGAC       TruSeq RNA
+D709_501        CGGCTATG-TATAGCCT       TruSeq RNA
+D709_502        CGGCTATG-ATAGAGGC       TruSeq RNA
+D709_503        CGGCTATG-CCTATCCT       TruSeq RNA
+D709_504        CGGCTATG-GGCTCTGA       TruSeq RNA
+D709_505        CGGCTATG-AGGCGAAG       TruSeq RNA
+D709_506        CGGCTATG-TAATCTTA       TruSeq RNA
+D709_507        CGGCTATG-CAGGACGT       TruSeq RNA
+D709_508        CGGCTATG-GTACTGAC       TruSeq RNA
+D710_501        TCCGCGAA-TATAGCCT       TruSeq RNA
+D710_502        TCCGCGAA-ATAGAGGC       TruSeq RNA
+D710_503        TCCGCGAA-CCTATCCT       TruSeq RNA
+D710_504        TCCGCGAA-GGCTCTGA       TruSeq RNA
+D710_505        TCCGCGAA-AGGCGAAG       TruSeq RNA
+D710_506        TCCGCGAA-TAATCTTA       TruSeq RNA
+D710_507        TCCGCGAA-CAGGACGT       TruSeq RNA
+```

ProjectHome.md

@@ -0,0 +1,41 @@
+Command-line tools for processing biological sequencing data.  Barcode demultiplexing, adapter trimming, etc.
+
+Primarily written to support an Illumina based pipeline - but should work with any FASTQs.
+
+### Overview: ###
+
+  * [fastq-mcf](FastqMcf.md)
+> > Scans a sequence file for adapters, and, based on a log-scaled threshold, determines a set of clipping parameters and performs clipping.  Also does skewing detection and quality filtering.
+  * [fastq-multx](FastqMultx.md)
+> > Demultiplexes a fastq.  Capable of auto-determining barcode id's based on a master set fields.  Keeps multiple reads in-sync during demultiplexing.  Can verify that the reads are in-sync as well, and fail if they're not.
+  * [fastq-join](FastqJoin.md)
+> > Similar to audy's stitch program, but in C, more efficient and supports some automatic benchmarking and tuning. It uses the same "squared distance for anchored alignment" as other tools.
+  * [varcall](Varcall.md)
+> > Takes a pileup and calculates variants in a more easily parameterized manner than some other tools.
+
+### Other Stuff: ###
+
+  * [sam-stats](SamStats.md) - Basic sam/bam stats.  Like other tools, but produces what I want to look at, in a format suitable for passing to other programs.  (<a href='http://ea-utils.googlecode.com/svn/trunk/clipper/sam-stats.cpp'>View source</a>)
+
+  * fastq-stats - Basic fastq stats.  Counts duplicates. Option for per-cycle stats, or not (irrelevant for many sequencers).    (<a href='http://ea-utils.googlecode.com/svn/trunk/clipper/fastq-stats.cpp'>View source</a>)
+
+  * determine-phred - Returns the phred scale of the input file. Works with sams, fastq's or pileups and gzipped files.
+
+  * Chrdex.pm & Sqldex.pm - obsoleted by the cpan module Text::Tidx.   Sqldex may not actually be obsolete, because Tidx uses more ram and is slower for very small jobs.  But for Exome and RNA-Seq work, [Text::Tidx](http://search.cpan.org/~earonesty/Text-Tidx/) beats both.
+
+  * qsh - Runs a bash script file like a "cluster aware makefile"...only processing newer things, die'ing if things go wrong, and sending jobs to a queue manager if they're big.  That way you don't have to write makefiles, or wrap things in "qsub" calls for every little program.  Not really ready yet.
+
+  * grun - Fast, lightweight grid queue software.  Keeps the job queue on disk at all times.  Very fast.   Works well by now
+
+  * gwrap - Bash wrapper shell that downloads all dependencies that are not the local system.... good for EC2 nodes.  Linux only.   Will use it if we ever go to EC2.
+
+  * gtf2bed - Converter that bundles up a GFF's exons and makes a UCSC-styled bed file with thin/thick properly set from the start/stop sites.  (<a href='http://ea-utils.googlecode.com/svn/trunk/clipper/gtf2bed'>Click for source</a>)
+
+  * randomFQ - takes a fastq (can be gzipped or paired-end) and randomly subsets to a user defined number of reads (<a href='https://ea-utils.googlecode.com/svn/trunk/clipper/randomFQ'>Click for source</a>)
+
+### Citing: ###
+
+
+> Erik Aronesty (2011). _ea-utils_ : "Command-line tools for processing biological sequencing data"; http://code.google.com/p/ea-utils
+
+> Erik Aronesty (2013). _TOBioiJ_ : "Comparison of Sequencing Utility Programs", [DOI:10.2174/1875036201307010001](http://benthamscience.com/open/openaccess.php?tobioij/articles/V007/1TOBIOIJ.htm)