Skip to content
Permalink
Browse files

add release notes for 0.9.9.7 / update docs

  • Loading branch information...
ssadedin committed Apr 23, 2019
1 parent 893d55e commit 37ae7edc58f6d84cbde1abf06ba735f07e96d3ea
Showing with 116 additions and 2 deletions.
  1. +2 −2 README.md
  2. +113 −0 docs/Releases/0.9.9.7.md
  3. +1 −0 mkdocs.yml
@@ -3,8 +3,8 @@ Welcome to Bpipe [![travis](https://travis-ci.org/ssadedin/bpipe.svg?branch=mas

Bpipe provides a platform for running big bioinformatics jobs that consist of a series of processing stages - known as 'pipelines'.

* July 21st, 2018 - New! [Bpipe 0.9.9.6](http://docs.bpipe.org/Releases/0.9.9.6) released!
* Download [latest](http://download.bpipe.org/versions/bpipe-0.9.9.6.tar.gz), [all](http://download.bpipe.org)
* April 25th, 2019 - New! [Bpipe 0.9.9.7](http://docs.bpipe.org/Releases/0.9.9.7/) released!
* Download [latest](http://download.bpipe.org/versions/bpipe-0.9.9.7.tar.gz), [all](http://download.bpipe.org)
* [Documentation](http://docs.bpipe.org)
* [Mailing List](https://groups.google.com/forum/#!forum/bpipe-discuss) (Google Group)

@@ -0,0 +1,113 @@
Bpipe Version 0.9.9.7
===============

## Summary

This release includes several major new features including prelimary support for
running Bpipe pipelines on cloud providers (Google Cloud, Amazon Web Services), a
new [merge point](../Language/MergePoints) operator for making it easier to
construct parallel pipelines using scatter-gather parallelism. In addition to these,
significant work has been done to dramatically improve performance and reduce
resource consumption on very highly parallel pipelines with large numbers of
input / output files.

## Features

- Preliminary support for executing pipelines on Google Cloud Services
(Compute Engine) and mounting storage for pipelines from Google Cloud
Storage

- Preliminary support for executing pipelines on Amazon Web Services
using EC2 and mounting storage for pipelines from S3

- The 'groovy' command can now run embedded groovy (executed outside
Bpipe) using the groovy runtime bundled with Bpipe

- Support aliasing to string values in addition to outputs

- Experimental support for beforeRun hook in command config: execute
arbitrary groovy code before a command executes

- Many performance improvements, esp. for large, highly
parallel pipelines

- Support configuration for number of retries for status
polling of HPC jobs (statusPollRetries setting)

- Support for 'optional' inputs in pipelines: to make input optional,
suffix with 'optional'. Also can add 'flag' to add flags
in commands eg: ${input.csv.optional.flag('--csv')}

- New operator: merge point operator (>>>) automatically configures a stage
to merge outputs from a previous parallel split

- Add region.bedFlag(flag) method for convenience when passing
regions to commands

- 'var' expressions may now be added in the main pipeline script,
not just pipeline stages. These define optional
variables, and provide a default.

- JMS support now responds to 'ping' message with 'pong' reply
if JMS 'Reply-To' is set to allow for status monitoring


## Fixes

- Fix incorrect "abnormal termination" messages
printed to console when pipeline stopped with 'bpipe stop'

- Fix incorrect 'pre-existing' printed for outputs that were
created by pipeline

- Fix genome not accessible in pipeline the first time downloaded,
printing error

- Re-execute checks if a commmand in the same stage has executed

- synchronize initialization of dir watcher to fix sporadic
ConcurrentModificationExceptions

- Fix empty embedded parallel stage list causing resolution of incorrect
downstream input

- Fix leak of 'var' variables across branches when 'using' applied to
pipeline stage

- Fix error if 4 or more arguments passed to "to" in transform

- Fix bpipe complaining spurious outputs not created on retry,
but not original run

- Fix some bugs where branch names were not being observed

- Fix branch name sometimes inserted without separating period for transforms

- Avoid redundantly putting branch name into files

- Improved detail in error / log messages in a few places

- Fix missing branch and '..' in filenames

- Change: globally defined variables must now be held constant
once pipeline starts

- Fix split regions not stable between runs, set region id as branch
name

- Fix bed.split producing different splits if run repeatedly on same bed

- Fix errors output if SLF4J referenced in user loaded libraries

- Fix npe / improve error message when filter used with mismatching output ext

- Fix error in stage body resulting in confusing 'no associated storage'
assertion failure

- Add 'allowForeign' option to 'from' to let it process non-outputs

- Lessen the retries and retry interval when file cannot be cleaned
up



@@ -83,4 +83,5 @@ pages:
- ['Releases/0.9.9.4.md','Releases','0.9.9.4']
- ['Releases/0.9.9.5.md','Releases','0.9.9.5']
- ['Releases/0.9.9.6.md','Releases','0.9.9.6']
- ['Releases/0.9.9.7.md','Releases','0.9.9.7']
#markdown_extensions: [gfm]

0 comments on commit 37ae7ed

Please sign in to comment.
You can’t perform that action at this time.