From 310917a3ece02a68100476d4d0361673782ee7d5 Mon Sep 17 00:00:00 2001 From: Simon Sadedin Date: Tue, 9 Jan 2018 09:28:56 +1100 Subject: [PATCH] updated release notes for 0.9.9.5 --- ReleaseNotes.txt | 88 +++++++++++++++++++++++++++++-- docs/Releases/0.9.9.5.md | 110 +++++++++++++++++++++++++++++++++++---- 2 files changed, 186 insertions(+), 12 deletions(-) diff --git a/ReleaseNotes.txt b/ReleaseNotes.txt index 4737d1dc..4889a6f1 100755 --- a/ReleaseNotes.txt +++ b/ReleaseNotes.txt @@ -1,9 +1,91 @@ -Changes for Bpipe Version 0.9.9.3 -=================================== +=========== Changes for Bpipe Version 0.9.9.5 ============ -TODO + Features: + - Support for Bpipe 'agent' which can listen to requests via + JMS (activemq for now) to run pipelines + + - Support sending json to JMS/ActiveMQ via send command + + - Support for sending JSON content to URLs via send command + + - Utilitity function 'asQuotedList' formats lists of inputs + as syntax compatible with js,python,groovy scripts + + - Experimental support for "sequential" or power operator + - Print out the outputs to be created when run in test mode, + in addition to the command to run (existing behavior) + + - Support for post-command hook to execute a script after every + command + + - Add command start and stop time to output from 'bpipe query' + + - Add start and stop time to display in 'bpipe errors' + + - You can now specify java libraries as list in bpipe.config, + in addition to the existing string form + + - Support for check review states, and improved check display + + + Fixes: + + + - Fix exception/error if double input extension overlaps + output with sng ext, eg. $input.foo.bar happens to match + upstream $output.foo + + - Fix completed event sometimes not sent for failed stages + + - Fix failed checks written to std out at end of run + + - Fix situations where checks are re-executed when not + required + + - Support supplying input files explicitly to run in + run([...]) + + - Support for JMS notifications of pipeline events + + - Fix error displayed by 'bpipe log' if no pipeline has ever + been executed in dir + + - Prevent recursive load caused defining a segment inside a + loaded file + + - Add segment structure to JSON pipeline representation + Previously segments were expanded into their constituent stages + in the JSON representation, and thus the higher level pipeline + structure was lost. Now each node and edge in the group + is given a type, and both the segment and its intenral + structure are output as nodes / edges. + + - Fix sometimes getting wrong last run pid (thinks still + running) + + - Change to make error in loaded script terminate pipeline + instead of just printing error and continuing + + - Fix bpipe log stops working after some commands until bpipe run + again + + - Avoid printing out cleanup message if zero files to cleanup + + - Move some of the docs out of Language and into Guides + + - Prevent exception in bpipe errors if command state not saved + + - Fix 'bpipe errors' not showing log output if hyphen in branch + + - Fix status command, add regression test + + - Groovy command in script now handles embedded quotes, + which previously caused syntax errors + + - Fix error when nested segment made of single stage defined + Changes for Bpipe Version 0.9.9.2 =================================== diff --git a/docs/Releases/0.9.9.5.md b/docs/Releases/0.9.9.5.md index 7243005d..d7148585 100644 --- a/docs/Releases/0.9.9.5.md +++ b/docs/Releases/0.9.9.5.md @@ -3,22 +3,46 @@ Bpipe Version 0.9.9.5 ## Summary -This release has a smattering of handy features including much more flexible -options to limit concurrency based on memory limits, native support for -BED file for parallelising over genomic regions, and some other minor features. +This release has a variety of new features with highlights being more flexible +options to limit concurrency based on memory usage, and support for integration +with message queuing systems (JMS) to both run pipelines remotely and also to +send outbound messages. + A range of important bug fixes are included as well. One minor behavior change may be observed for pipelines using the `$threads` variable: this variable will now only cause dynamic thread allocation if the `procs` config for that command is either unset or is set to zero. The old behavior caused dynamic allocation of threads when this value was set to 1, which was counterintuitive and confusing. -## Changes +## Features + + - Bpipe now supports an 'agent' that can run in the background on a system + and accept remote instructions to run pipelines on your behalf. For now, + Bpipe listens to requests via JMS (activemq for now) to run pipelines, but + this will be expanded in the future. + + - Improved support for the 'send' command to: + - Sending json to JMS/ActiveMQ queues + - Send (POST) json or other content to HTTP(s) URLs + + - Utilitity function 'asQuotedList' formats lists of inputs + as syntax compatible with js,python,groovy scripts + + - Experimental support for "sequential" or power operator + + - Print out the outputs to be created when run in test mode, + in addition to the command to run (existing behavior) + + - Support for check review states. Previously, checks were were either + passed, failed or overridden. Now when a check fails it enters 'pending + review' state, and becomes failed only when manually failed after review. - Jobs are now automatically throttled based on memory if a memory limit is set using `-m` flag or `limit { memory="g" }` in bpipe.config. - - - Inline memory variable can now be used to reference the amount of memory - specified in bpipe.config for a stage. Returns memory in GB. + For this to activate, you must define a "memory" config parameter for + commands. An inline / magic memory variable can now be used to reference the + amount of memory specified in bpipe.config for a command. Returns memory in GB. + eg: `exec "java -Xmx${memory}g SomeCommand"` - Support for `memoryMargin` command config so that memory variable returns less memory than allocated to a job (to allow for memory overhead) @@ -36,14 +60,27 @@ this value was set to 1, which was counterintuitive and confusing. - Check clauses can now omit the `otherwise` clause. Such checks are executed after a stage exits. - - Add detailed command config to output displayed by `bpipe errors` - - Change: dynamic thread allocation is now invoked when procs=0 OR unspecified, not procs=1 - Preliminary support for jobs in queued / waiting state to be reallocated if a pre-allocated job becomes available +## Minor Changes + + - Support for post-command hook to execute a script after every + command + + - Add command start and stop time to output from 'bpipe query' + + - Add start and stop time to display in 'bpipe errors' + + - You can now specify java libraries as list in bpipe.config, + in addition to the existing string form + + - Add many more details to output shown by `bpipe errors` + + ## Fixes - Fix possible failure of `cleanup` to identify files as eligible due to incorrectly @@ -67,3 +104,58 @@ this value was set to 1, which was counterintuitive and confusing. - Fix segments not correctly represented in json graph output + - Fix exception/error if double input extension overlaps + input extension overlaps + output with sng ext, eg. $input.foo.bar happens to match + upstream $output.foo + + - Fix completed event sometimes not sent for failed stages + + - Fix failed checks written to std out at end of run + + - Fix situations where checks are re-executed when not + required + + - Support supplying input files explicitly to run in + run([...]) + + - Support for JMS notifications of pipeline events + + - Fix error displayed by 'bpipe log' if no pipeline has ever + been executed in dir + + - Prevent recursive load caused defining a segment inside a + loaded file + + - Add segment structure to JSON pipeline representation + Previously segments were expanded into their constituent stages + in the JSON representation, and thus the higher level pipeline + structure was lost. Now each node and edge in the group + is given a type, and both the segment and its intenral + structure are output as nodes / edges. + + - Fix sometimes getting wrong last run pid (thinks still + running) + + - Change to make error in loaded script terminate pipeline + instead of just printing error and continuing + + - Fix bpipe log stops working after some commands until bpipe run + again + + - Avoid printing out cleanup message if zero files to cleanup + + - Move some of the docs out of Language and into Guides + + - Prevent exception in bpipe errors if command state not saved + + - Fix 'bpipe errors' not showing log output if hyphen in branch + + - Fix status command, add regression test + + - Groovy command in script now handles embedded quotes, + which previously caused syntax errors + + - Fix error when nested segment made of single stage defined + +