updated release notes for 0.9.9.5

ssadedin · Jan 8, 2018 · 310917a · 310917a
1 parent 82f7439
commit 310917a
Show file tree

Hide file tree

Showing 2 changed files with 186 additions and 12 deletions.
diff --git a/ReleaseNotes.txt b/ReleaseNotes.txt
@@ -1,9 +1,91 @@
-Changes for Bpipe Version 0.9.9.3 
-===================================
+=========== Changes for Bpipe Version 0.9.9.5 ============
 
-TODO
+  Features:
 
+   - Support for Bpipe 'agent' which can listen to requests via
+     JMS (activemq for now) to run pipelines
+
+   - Support sending json to JMS/ActiveMQ via send command 
+
+   - Support for sending JSON content to URLs via send command
+
+   - Utilitity function 'asQuotedList' formats lists of inputs
+     as syntax compatible with js,python,groovy scripts 
+
+   - Experimental support for "sequential" or power operator 
 
+   - Print out the outputs to be created when run in test mode, 
+     in addition to the command to run (existing behavior)
+
+   - Support for post-command hook to execute a script after every 
+     command 
+
+   - Add command start and stop time to output from 'bpipe query' 
+
+   - Add start and stop time to display in 'bpipe errors' 
+
+   - You can now specify java libraries as list in bpipe.config,
+     in addition to the existing string form
+
+   - Support for check review states, and improved check display 
+
+
+  Fixes:
+
+
+   - Fix exception/error if double input extension overlaps 
+     output with sng ext, eg. $input.foo.bar  happens to match
+     upstream $output.foo
+
+   - Fix completed event sometimes not sent for failed stages 
+
+   - Fix failed checks written to std out at end of run 
+
+   - Fix situations where checks are re-executed when not 
+     required 
+
+   - Support supplying input files explicitly to run in 
+     run([...])
+
+   - Support for JMS notifications of pipeline events
+
+   - Fix error displayed by 'bpipe log' if no pipeline has ever 
+     been executed in dir 
+
+   - Prevent recursive load caused defining a segment inside a 
+     loaded file
+
+   - Add segment structure to JSON pipeline representation
+     Previously segments were expanded into their constituent stages 
+     in the JSON representation, and thus the higher level pipeline
+     structure was lost.  Now each node and edge in the group
+     is given a type, and both the segment and its intenral
+     structure are output as nodes / edges.  
+
+   - Fix sometimes getting wrong last run pid (thinks still 
+     running) 
+
+   - Change to make error in loaded script terminate pipeline 
+     instead of just printing error and continuing
+
+   - Fix bpipe log stops working after some commands until bpipe run 
+     again 
+
+   - Avoid printing out cleanup message if zero files to cleanup
+
+   - Move some of the docs out of Language and into Guides 
+
+   - Prevent exception in bpipe errors if command state not saved 
+
+   - Fix 'bpipe errors' not showing log output if hyphen in branch 
+
+   - Fix status command, add regression test 
+
+   - Groovy command in script now handles embedded quotes,
+     which previously caused syntax errors
+
+   - Fix error when nested segment made of single stage defined 
+
 
 Changes for Bpipe Version 0.9.9.2 
 ===================================

diff --git a/docs/Releases/0.9.9.5.md b/docs/Releases/0.9.9.5.md
@@ -3,22 +3,46 @@ Bpipe Version 0.9.9.5
 
 ## Summary
 
-This release has a smattering of handy features including much more flexible
-options to limit concurrency based on memory limits, native support for
-BED file for parallelising over genomic regions, and some other minor features.
+This release has a variety of new features with highlights being more flexible
+options to limit concurrency based on memory usage, and support for integration
+with message queuing systems (JMS) to both run pipelines remotely and also to
+send outbound messages. 
+
 A range of important bug fixes are included as well. One minor behavior change may 
 be observed for pipelines using the `$threads` variable: this variable will now
 only cause dynamic thread allocation if the `procs` config for that command is
 either unset or is set to zero. The old behavior caused dynamic allocation of threads when
 this value was set to 1, which was counterintuitive and confusing.
 
-## Changes
+## Features
+
+ - Bpipe now supports an 'agent' that can run in the background on a system 
+   and accept remote instructions to run pipelines on your behalf. For now, 
+   Bpipe listens to requests via JMS (activemq for now) to run pipelines, but
+   this will be expanded in the future.
+
+ - Improved support for the 'send' command to:
+  - Sending json to JMS/ActiveMQ queues
+  - Send (POST) json or other content to HTTP(s) URLs 
+
+ - Utilitity function 'asQuotedList' formats lists of inputs
+   as syntax compatible with js,python,groovy scripts 
+
+ - Experimental support for "sequential" or power operator 
+
+ - Print out the outputs to be created when run in test mode, 
+   in addition to the command to run (existing behavior)
+
+ - Support for check review states. Previously, checks were were either
+   passed, failed or overridden. Now when a check fails it enters 'pending
+   review' state, and becomes failed only when manually failed after review.
 
  - Jobs are now automatically throttled based on memory if a memory 
    limit is set using `-m` flag or `limit { memory="<n>g" }` in bpipe.config.
-
- - Inline memory variable can now be used to reference the amount of memory
-   specified in bpipe.config for a stage. Returns memory in GB.
+   For this to activate, you must define a "memory" config parameter for
+   commands.  An inline / magic memory variable can now be used to reference the 
+   amount of memory specified in bpipe.config for a command. Returns memory in GB.
+   eg: `exec "java -Xmx${memory}g SomeCommand"`
 
  - Support for `memoryMargin` command config so that memory variable returns
    less memory than allocated to a job (to allow for memory overhead)
@@ -36,14 +60,27 @@ this value was set to 1, which was counterintuitive and confusing.
  - Check clauses can now omit the `otherwise` clause. Such checks are 
    executed after a stage exits.
 
- - Add detailed command config to output displayed by `bpipe errors`
-
  - Change: dynamic thread allocation is now invoked when procs=0 
    OR unspecified, not procs=1 
 
  - Preliminary support for jobs in queued / waiting state to be reallocated
    if a pre-allocated job becomes available
 
+## Minor Changes
+
+ - Support for post-command hook to execute a script after every 
+   command 
+
+ - Add command start and stop time to output from 'bpipe query' 
+
+ - Add start and stop time to display in 'bpipe errors' 
+
+ - You can now specify java libraries as list in bpipe.config,
+   in addition to the existing string form
+
+ - Add many more details to output shown by `bpipe errors`
+
+
 ## Fixes
 
  - Fix possible failure of `cleanup` to identify files as eligible due to incorrectly
@@ -67,3 +104,58 @@ this value was set to 1, which was counterintuitive and confusing.
 
  - Fix segments not correctly represented in json graph output
 
+ - Fix exception/error if double input extension overlaps 
+   input extension overlaps 
+   output with sng ext, eg. $input.foo.bar  happens to match
+   upstream $output.foo
+
+ - Fix completed event sometimes not sent for failed stages 
+
+ - Fix failed checks written to std out at end of run 
+
+ - Fix situations where checks are re-executed when not 
+   required 
+
+ - Support supplying input files explicitly to run in 
+   run([...])
+
+ - Support for JMS notifications of pipeline events
+
+ - Fix error displayed by 'bpipe log' if no pipeline has ever 
+   been executed in dir 
+
+ - Prevent recursive load caused defining a segment inside a 
+   loaded file
+
+ - Add segment structure to JSON pipeline representation
+   Previously segments were expanded into their constituent stages 
+   in the JSON representation, and thus the higher level pipeline
+   structure was lost.  Now each node and edge in the group
+   is given a type, and both the segment and its intenral
+   structure are output as nodes / edges.  
+
+ - Fix sometimes getting wrong last run pid (thinks still 
+   running) 
+
+ - Change to make error in loaded script terminate pipeline 
+   instead of just printing error and continuing
+
+ - Fix bpipe log stops working after some commands until bpipe run 
+   again 
+
+ - Avoid printing out cleanup message if zero files to cleanup
+
+ - Move some of the docs out of Language and into Guides 
+
+ - Prevent exception in bpipe errors if command state not saved 
+
+ - Fix 'bpipe errors' not showing log output if hyphen in branch 
+
+ - Fix status command, add regression test 
+
+ - Groovy command in script now handles embedded quotes,
+   which previously caused syntax errors
+
+ - Fix error when nested segment made of single stage defined 
+
+