Permalink
Browse files

updated release notes for 0.9.9.5

  • Loading branch information...
ssadedin committed Jan 8, 2018
1 parent 82f7439 commit 310917a3ece02a68100476d4d0361673782ee7d5
Showing with 186 additions and 12 deletions.
  1. +85 −3 ReleaseNotes.txt
  2. +101 −9 docs/Releases/0.9.9.5.md
View
@@ -1,9 +1,91 @@
Changes for Bpipe Version 0.9.9.3
===================================
=========== Changes for Bpipe Version 0.9.9.5 ============
TODO
Features:
- Support for Bpipe 'agent' which can listen to requests via
JMS (activemq for now) to run pipelines
- Support sending json to JMS/ActiveMQ via send command
- Support for sending JSON content to URLs via send command
- Utilitity function 'asQuotedList' formats lists of inputs
as syntax compatible with js,python,groovy scripts
- Experimental support for "sequential" or power operator
- Print out the outputs to be created when run in test mode,
in addition to the command to run (existing behavior)
- Support for post-command hook to execute a script after every
command
- Add command start and stop time to output from 'bpipe query'
- Add start and stop time to display in 'bpipe errors'
- You can now specify java libraries as list in bpipe.config,
in addition to the existing string form
- Support for check review states, and improved check display
Fixes:
- Fix exception/error if double input extension overlaps
output with sng ext, eg. $input.foo.bar happens to match
upstream $output.foo
- Fix completed event sometimes not sent for failed stages
- Fix failed checks written to std out at end of run
- Fix situations where checks are re-executed when not
required
- Support supplying input files explicitly to run in
run([...])
- Support for JMS notifications of pipeline events
- Fix error displayed by 'bpipe log' if no pipeline has ever
been executed in dir
- Prevent recursive load caused defining a segment inside a
loaded file
- Add segment structure to JSON pipeline representation
Previously segments were expanded into their constituent stages
in the JSON representation, and thus the higher level pipeline
structure was lost. Now each node and edge in the group
is given a type, and both the segment and its intenral
structure are output as nodes / edges.
- Fix sometimes getting wrong last run pid (thinks still
running)
- Change to make error in loaded script terminate pipeline
instead of just printing error and continuing
- Fix bpipe log stops working after some commands until bpipe run
again
- Avoid printing out cleanup message if zero files to cleanup
- Move some of the docs out of Language and into Guides
- Prevent exception in bpipe errors if command state not saved
- Fix 'bpipe errors' not showing log output if hyphen in branch
- Fix status command, add regression test
- Groovy command in script now handles embedded quotes,
which previously caused syntax errors
- Fix error when nested segment made of single stage defined
Changes for Bpipe Version 0.9.9.2
===================================
View
@@ -3,22 +3,46 @@ Bpipe Version 0.9.9.5
## Summary
This release has a smattering of handy features including much more flexible
options to limit concurrency based on memory limits, native support for
BED file for parallelising over genomic regions, and some other minor features.
This release has a variety of new features with highlights being more flexible
options to limit concurrency based on memory usage, and support for integration
with message queuing systems (JMS) to both run pipelines remotely and also to
send outbound messages.
A range of important bug fixes are included as well. One minor behavior change may
be observed for pipelines using the `$threads` variable: this variable will now
only cause dynamic thread allocation if the `procs` config for that command is
either unset or is set to zero. The old behavior caused dynamic allocation of threads when
this value was set to 1, which was counterintuitive and confusing.
## Changes
## Features
- Bpipe now supports an 'agent' that can run in the background on a system
and accept remote instructions to run pipelines on your behalf. For now,
Bpipe listens to requests via JMS (activemq for now) to run pipelines, but
this will be expanded in the future.
- Improved support for the 'send' command to:
- Sending json to JMS/ActiveMQ queues
- Send (POST) json or other content to HTTP(s) URLs
- Utilitity function 'asQuotedList' formats lists of inputs
as syntax compatible with js,python,groovy scripts
- Experimental support for "sequential" or power operator
- Print out the outputs to be created when run in test mode,
in addition to the command to run (existing behavior)
- Support for check review states. Previously, checks were were either
passed, failed or overridden. Now when a check fails it enters 'pending
review' state, and becomes failed only when manually failed after review.
- Jobs are now automatically throttled based on memory if a memory
limit is set using `-m` flag or `limit { memory="<n>g" }` in bpipe.config.
- Inline memory variable can now be used to reference the amount of memory
specified in bpipe.config for a stage. Returns memory in GB.
For this to activate, you must define a "memory" config parameter for
commands. An inline / magic memory variable can now be used to reference the
amount of memory specified in bpipe.config for a command. Returns memory in GB.
eg: `exec "java -Xmx${memory}g SomeCommand"`
- Support for `memoryMargin` command config so that memory variable returns
less memory than allocated to a job (to allow for memory overhead)
@@ -36,14 +60,27 @@ this value was set to 1, which was counterintuitive and confusing.
- Check clauses can now omit the `otherwise` clause. Such checks are
executed after a stage exits.
- Add detailed command config to output displayed by `bpipe errors`
- Change: dynamic thread allocation is now invoked when procs=0
OR unspecified, not procs=1
- Preliminary support for jobs in queued / waiting state to be reallocated
if a pre-allocated job becomes available
## Minor Changes
- Support for post-command hook to execute a script after every
command
- Add command start and stop time to output from 'bpipe query'
- Add start and stop time to display in 'bpipe errors'
- You can now specify java libraries as list in bpipe.config,
in addition to the existing string form
- Add many more details to output shown by `bpipe errors`
## Fixes
- Fix possible failure of `cleanup` to identify files as eligible due to incorrectly
@@ -67,3 +104,58 @@ this value was set to 1, which was counterintuitive and confusing.
- Fix segments not correctly represented in json graph output
- Fix exception/error if double input extension overlaps
input extension overlaps
output with sng ext, eg. $input.foo.bar happens to match
upstream $output.foo
- Fix completed event sometimes not sent for failed stages
- Fix failed checks written to std out at end of run
- Fix situations where checks are re-executed when not
required
- Support supplying input files explicitly to run in
run([...])
- Support for JMS notifications of pipeline events
- Fix error displayed by 'bpipe log' if no pipeline has ever
been executed in dir
- Prevent recursive load caused defining a segment inside a
loaded file
- Add segment structure to JSON pipeline representation
Previously segments were expanded into their constituent stages
in the JSON representation, and thus the higher level pipeline
structure was lost. Now each node and edge in the group
is given a type, and both the segment and its intenral
structure are output as nodes / edges.
- Fix sometimes getting wrong last run pid (thinks still
running)
- Change to make error in loaded script terminate pipeline
instead of just printing error and continuing
- Fix bpipe log stops working after some commands until bpipe run
again
- Avoid printing out cleanup message if zero files to cleanup
- Move some of the docs out of Language and into Guides
- Prevent exception in bpipe errors if command state not saved
- Fix 'bpipe errors' not showing log output if hyphen in branch
- Fix status command, add regression test
- Groovy command in script now handles embedded quotes,
which previously caused syntax errors
- Fix error when nested segment made of single stage defined

0 comments on commit 310917a

Please sign in to comment.