Skip to content
Bpipe - a tool for running and managing bioinformatics pipelines
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin fix tail not working in main bpipe script Apr 23, 2019
docs add release notes for 0.9.9.7 / update docs Apr 23, 2019
examples
gradle/wrapper update gradle, gpars, remove unneeded groovy-all from local-libs Apr 23, 2019
local-lib
src fix region configured for cloud executor overriding region specified … Apr 23, 2019
test-src/bpipe some initial support for an AWS / EC2 executor Apr 23, 2019
tests fix erroneous messages printed about pre-existing files after run fin… Apr 23, 2019
.gitignore refactoring of even listener support to enable external monitoring Feb 6, 2017
.project Initial source checkin Oct 6, 2011
.travis.yml remove groovy from travis builds since built in groovy should suffice Apr 23, 2019
LICENSE Added explicit license file for clarity of licensing Jul 15, 2015
README.md
ReleaseNotes.txt updated release notes for 0.9.9.7 Apr 23, 2019
build.gradle
gradlew
mkdocs.yml add release notes for 0.9.9.7 / update docs Apr 23, 2019
requirements.txt Import docs from test repo Mar 14, 2015
settings.gradle Add settings.gradle to stop build failing if checked out to directory… Apr 16, 2016

README.md

Welcome to Bpipe travis

Bpipe provides a platform for running big bioinformatics jobs that consist of a series of processing stages - known as 'pipelines'.

Bpipe has been published in Bioinformatics! If you use Bpipe, please cite:

Sadedin S, Pope B & Oshlack A, Bpipe: A Tool for Running and Managing Bioinformatics Pipelines, Bioinformatics

Why Bpipe?

Many people working with bioinformatics data end up running jobs as shell scripts. While this makes running them easy it has a lot of limitations. For example, when scripts fail half way through it is often hard to tell where, or why they failed, and even harder to restart the job from the point of failure. There is no automatic log of the commands executed or automatic capture of console output to ensure it is possible to later on see what happened. Sometimes jobs fail half way through, leaving half created files that can get confused with good files. Modifying the pipeline is also time consuming and error prone - adding or removing a stage causes modifications in multiple places, meaning that it is all too easy to have a change of file name cause later commands to fail or worse, run on incorrect data. Bpipe tries to solve all these problems (and more!) while departing as little as possible from the simplicity of the shell script. In fact, your Bpipe scripts will often look very similar to the original shell scripts you might have started with.

By turning your shell scripts into Bpipe scripts, here are some of the features you can get:

  • Simple definition of tasks to run - Bpipe runs shell commands almost as-is - no programming required.
  • Transactional management of tasks - commands that fail get outputs cleaned up, log files saved and the pipeline cleanly aborted. No out of control jobs going crazy.
  • Automatic Connection of Pipeline Stages - Bpipe manages the file names for input and output of each stage in a systematic way so that you don't need to think about it. Removing or adding new stages "just works" and never breaks the flow of data.
  • Easy Restarting of Jobs - when a job fails cleanly restart from the point of failure.
  • Easy Parallelism - Bpipe makes it simple to split jobs into many pieces and run them all in parallel whether on a cluster or locally on your own machine
  • Audit Trail - Bpipe keeps a journal of exactly which commands executed and what their inputs and outputs were.
  • Integration with Cluster Resource Managers - if you use Torque PBS, Oracle Grid Engine or Platform LSF then Bpipe will make your life easier by allowing pure Bpipe scripts to run on your cluster virtually unchanged from how you run them locally.
  • Notifications by Email, Instant Message, Message Queuing Systems - Bpipe can send you alerts to tell you when your pipeline finishes or even as each stage completes.
  • See how Bpipe compares to similar tools

Take a look at the Overview to see Bpipe in action, work through the Basic Tutorial for simple first steps, see a step by step example of a realistic pipeline made using Bpipe, or take a look at the Reference to see all the documentation.

You can’t perform that action at this time.