Skip to content
Bpipe - a tool for running and managing bioinformatics pipelines
Groovy Shell HTML Java
Branch: master
Clone or download

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin add new 'archive' and --autoarchive function to auto-cleanup pipelines Sep 8, 2019
docs document -u option Jan 17, 2020
examples add an example of a data driven pipeline Oct 2, 2018
gradle/wrapper update gradle, gpars, remove unneeded groovy-all from local-libs Apr 23, 2019
groovy-lib add missing groovy libs May 22, 2019
local-lib remove groovy-all file now relocated to groovy-lib May 21, 2019
src fix hang / failure to exit Jan 24, 2020
test-src/bpipe fix torque status parsing broken when qstat xml output contains newline Jan 18, 2020
tests add missing config file causing test failure Dec 2, 2019
.gitignore refactoring of even listener support to enable external monitoring Feb 6, 2017
.project Initial source checkin Oct 6, 2011
.travis.yml
LICENSE Added explicit license file for clarity of licensing Jul 15, 2015
README.md update README for 0.9.9.8 Oct 17, 2019
ReleaseNotes.txt update release notes for 0.9.9.8 Oct 17, 2019
build.gradle update default build version to 0.9.9.8 May 27, 2019
gradle.properties switch default groovy version back to groovy 2..4.8 Oct 17, 2019
gradlew reinstante groovy-all in local-lib as it turns out is needed Apr 23, 2019
mkdocs.yml add options statement for easy parsing of script specific command lin… Oct 16, 2019
requirements.txt Import docs from test repo Mar 14, 2015
settings.gradle Add settings.gradle to stop build failing if checked out to directory… Apr 16, 2016

README.md

Welcome to Bpipe travis

Bpipe provides a platform for running big bioinformatics jobs that consist of a series of processing stages - known as 'pipelines'.

Bpipe has been published in Bioinformatics! If you use Bpipe, please cite:

Sadedin S, Pope B & Oshlack A, Bpipe: A Tool for Running and Managing Bioinformatics Pipelines, Bioinformatics

Why Bpipe?

Many people working with bioinformatics data end up running jobs as shell scripts. While this makes running them easy it has a lot of limitations. For example, when scripts fail half way through it is often hard to tell where, or why they failed, and even harder to restart the job from the point of failure. There is no automatic log of the commands executed or automatic capture of console output to ensure it is possible to later on see what happened. Sometimes jobs fail half way through, leaving half created files that can get confused with good files. Modifying the pipeline is also time consuming and error prone - adding or removing a stage causes modifications in multiple places, meaning that it is all too easy to have a change of file name cause later commands to fail or worse, run on incorrect data. Bpipe tries to solve all these problems (and more!) while departing as little as possible from the simplicity of the shell script. In fact, your Bpipe scripts will often look very similar to the original shell scripts you might have started with.

By turning your shell scripts into Bpipe scripts, here are some of the features you can get:

  • Simple definition of tasks to run - Bpipe runs shell commands almost as-is - no programming required.
  • Transactional management of tasks - commands that fail get outputs cleaned up, log files saved and the pipeline cleanly aborted. No out of control jobs going crazy.
  • Automatic Connection of Pipeline Stages - Bpipe manages the file names for input and output of each stage in a systematic way so that you don't need to think about it. Removing or adding new stages "just works" and never breaks the flow of data.
  • Easy Restarting of Jobs - when a job fails cleanly restart from the point of failure.
  • Easy Parallelism - Bpipe makes it simple to split jobs into many pieces and run them all in parallel whether on a cluster or locally on your own machine
  • Audit Trail - Bpipe keeps a journal of exactly which commands executed and what their inputs and outputs were.
  • Integration with Cluster Resource Managers - if you use Torque PBS, Oracle Grid Engine or Platform LSF then Bpipe will make your life easier by allowing pure Bpipe scripts to run on your cluster virtually unchanged from how you run them locally.
  • Notifications by Email, Instant Message, Message Queuing Systems - Bpipe can send you alerts to tell you when your pipeline finishes or even as each stage completes.
  • See how Bpipe compares to similar tools

Take a look at the Overview to see Bpipe in action, work through the Basic Tutorial for simple first steps, see a step by step example of a realistic pipeline made using Bpipe, or take a look at the Reference to see all the documentation.

You can’t perform that action at this time.