Skip to content
Compare
Choose a tag to compare
  • FEATURE: support for required attribute on notification
    channels to ensure pipeline failure if channel config fails

  • FEATURE: experimental support for virtual file extensions / file type
    mappings , defined using: filetype <virtual ext> : [ 'ext.one', 'ext.two',...]

  • FEATURE: agent will wait for commands to finish if it is asked to run
    multiple in the same directory at the same time

  • FEATURE: support for "queue" command : schedules pipeline to run after
    another one finishes in a directory

  • FIX: only match process owned by user when checking for running bpipe process

  • FEATURE(experimental): from clause can now be added in pipeline definition to
    cause inputs to prioritise from a given stage, with branch matching

  • FEATURE: update aws sdk to support latest instance types

  • FEATURE: EC2 commands now show name of stage via name tag, eg. in aws console

  • FEAUTURE: accept aws creds from environment if not defined in bpipe.config
    using standard AWS env variables for token / secret

  • FEATURE: support direct transfer of files via scp for AWSEC2 executor.
    If storage config setting is not defined and 'transfer' setting is true
    then Bpipe will utilise scp to transfer input/output files to EC2 instances

  • FEATURE: allow queue to be overridden within pipeline when sending
    to JMS channel via explicit 'queue' property

  • FEATURE: support replyTo as property for replying to JMS messages
    (rather than JMSReplyTo)

  • FEATURE: improved support for archiving: autoarchive and ability to run
    archive command from outside bpipe dir

  • FEATURE: implement support for 'required' and 'unique' as options on
    gitlab messages : ensures pipeline reports error status if
    gitlab calls fail

  • FEATURE: support for experimental "when" feature to make stages conditional

  • FEATURE: mask 'token' config property as sensitive info from log files

  • FEATURE: support for setting R_LIBS_USER with R { r_libs="..." }
    setting in bpipe.config

  • FEATURE: add checks to result message returned in reply-to on
    JMS messages

  • FEATURE: allow specification of gitlab projects by id when
    sending to gitlab

  • FEATURE: show custom title in bpipe startup when configured
    using title : '...' or title '...'

  • FEATURE: support to set env variables for commands via bpipe.config,
    by added env = : , ... to command config

  • FIX: http send not retried on bpipe retry if initial attempt failed

  • FIX: errors printed by trying to send JSON via SMTP

  • FIX: adjust bpipe agent jvm options to reduce footprint on large servers

  • FIX: fix transform of wildcard (eg: *.txt) to multiple outputs
    resulting in regex error

  • FIX: undefined pipeline variable somtimes shows bpipe internal stack trace

  • FIX: possible concurrentmodification errors when user-specificed config
    referenced concurrently in stages / reports

  • FIX: fix multilevel branch names not reflected in file names

  • FIX: error thrown when creating diagram using 'bpipe diagram ...'

  • Support for reservation of gpus with torque (set gpus=N in command config)

  • FIX: body of message relayed in JMS message header (causing parse fail in
    some clients / protocols)

  • FIX: fix regex in transform matching multi inputs only resolving single

  • FIX: fix error when literal regex used as transform input specifier

  • FIX: previous dirty state carrying over causing unexpected
    cleanup of good files

  • FIX: put retry around output dir watcher to handle race conditions
    / file system hiccups better

  • FIX: Explicitly set pwd for singularity as this seems not always
    to occur by default

  • FIX:: fix downstream reference to output created by check failing
    on retry

  • FIX: inability to create gitlab issue with empty content

  • FIX: json output double encoded in .bpipe/sent files

  • FIX: some commands not running with set -e by default

  • FIX:: fix warning printed for dirty files even though
    they don't exist, already notified

  • FIX: remove duplicates from printing of test outputs

  • FIX: remove printout of shebang stripped statement

Compare
Choose a tag to compare

This release includes a range of significant new features, including upgrade to
Groovy 2.5.13 as default groovy version, initial support for running commands
in docker or singularity containers, experimental cross-branch dependency support,
across the board improvements in accuracy and clarity of error messages,
robustness of error checking and efficiency, and improvements in support for
glcoud and AWS execution. Also included are many smaller features (see release
notes).

Release Notes

  • FEATURE: default groovy now changed to 2.5.13. To move back to
    old groovy, compile using -PGROOVY_VERSION=2.4.8

  • FEATURE: many improvments to efficiency, decrease load created by bpipe

  • FEATURE: initial singularity and docker container support

  • FIX: reference to $output in check causing incorrect output not
    found error

  • FIX: various fixes to AWS EC2 support

  • FEATURE: support for auth and headers in http send, and
    support to config http as notification channel

  • FIX: undefined variables referenced in stages resolving to
    value instead of throwing error

  • FIX: output.dir unexpectedly uses absolute path when inside run dir

  • FIX: branch outputs not resolved downstream when user terminates branch
    with success

  • FIX: error generating reports with latest groovy 2.5.x

  • FIX: ugly error if stage returns w/o output from annotated
    transform/filter

  • FIX: null pointer exception thrown for pipeline with no outputs
    for some pipelines

  • FIX: support custom project for gcloud executor

  • FEATURE: more flexible forwarding of local files

  • FIX: reports potentially overwriting each other inside
    transform with fixed outputs

  • FIX: storage system printed out even when local for not found files

  • FIX: legacy / manual output poller failing to see files outside
    root dir

  • FEATURE: config entries in bpipe.config defined as closure
    now receive optional arg containing config, eg: to configure
    memory based on procs

  • Initial implementation of channels to enable controlled async cross
    branch dependencies, using from(..., crossBranch:true) { ... }

  • FIX: in-progress files cleaned up when bpipe stats executed

  • FIX: resolving outputs across sibling branches now fails unless
    config option set

  • FIX: failure to resolve full path in from when duplicated
    in args

  • FIX: regression causing resolution of pre-existing files to fail

  • FIX: failure checking dependencies when only input is directory

  • FEATURE: support for environments in bpipe.config, selected via
    -e option, embed multiple configs for specific env inside named
    sections of bpipe.config

  • FEATURE: send can override template by supplying 'template' attribute
    after send, eg: send text(....) to template: 'report_template.html'

  • FIX: outputs sometimes not recognised as generated in printed output after
    pipeline finished

  • FEATURE: Support fo file attachments in gitlab notification / send

  • FIX: fix content of saved sent file null when sending json

  • FIX: crash / fail to send notification if pipeline input/output/file
    included in send of json

  • FEATURE: print queuing system job id in command status output

  • FEATURE: check for dirty files now enforced by markers in .bpipe/inprogress -
    error printed if present on retry.

Compare
Choose a tag to compare

Summary

This release includes a number of smaller features and improvements and
some important fixes.

NOTE: this version of Bpipe requires Java 1.8 or higher

Features

  • Ability to filter gitlab issues identified
    by title search for updating

  • Feature: support for 'debug' command that launches
    groovy shell. Add debug() to pipeline stage to launch
    interactive shell.

  • Revamp monitoring of command status for PBS/Torque
    based pipelines, by pooling calls to qstat between
    commands.

  • Feature: archive command to clean up .bpipe directory into
    restorable zip file

  • Support referencing params set on command line in
    bpipe.config

  • Bpipe pipelines can now accept the first line as a "shebang"
    so that bpipe pipelines become executable scripts. ie: make
    the first line of your pipeline: #!/path/to/bpipe

Fixes

  • Fix not completely robust reading of exit code for
    local commands (sometimes command succeeds when shouldn't)

  • Fix local commands not always terminating on explicit stop

  • Add setsid by default to local jobs on linux -
    prevents some cases where jobs abort when precessing
    ctrl-c after starting bpipe

  • Fix new gitlab issues with large content failing to send

  • Fix possible runaway killing of processes when
    local preallocated jobs are explicitly stopped with
    bpipe stop

  • Fix file notifications not always incrementing and
    therefore overwriting previous notification

  • Fix notification manager not respecting timeout to allow for
    setup of channels

  • Fix possible error due to race condition
    reading read PID with local command executor

  • Fix file handle leak on some file systems with large scale
    highly parallelised, long running pipelines

  • Fix the default file notification channel overwriting files from
    other send channels

  • Fix possible error if input is referenced in details of send

Compare
Choose a tag to compare

Changes for Bpipe Version 0.9.9.8

  • Feature: options statement for easy parsing of script specific
    command line arguments

  • Add workaround to fix possible failure on old pipelines checking
    file existence

  • BUG: Fix spurious error printed if broken symbolic link present

  • BUG: prevent agent launched commands hanging due to if bpipe prompts
    for information

  • Feature: add fileSystemSyncTimeMs which allows for customizable delay in
    file system sync before failing pipeline on missing output

  • BUG: fix regression - transform using regex does not substitute groups

    eg: transform('(.*)_R1.fastq') to("$1.txt") { ... }

  • Many fixes / optimisations to GCloud usage:

    • make workingDirectory specified for stage automatically
      placed within mounted storage

    • inherit region from main config inside
      filesystems, so it does not need to be double specified

    • support preemptible flag passed to google cloud
      instances

    • add sha1 hash of pipeline path / host to
      instanceid to make it unique b/w pipelines

    • fix region configured
      for cloud executor overriding region specified with -L

    • search for root input files in all filesystems, not only
      those configured as storages for stages

    • fix zone flag missing from gcloud commands, don't
      die mounting storage if instance not created

    • support using split / partitioned regions in cloud provider storage

  • BUG: Don't start executor pools when user is only testing the pipeline

  • BUG: ensure that executors clean up even if they error in stop
    command

  • Fix table output format so it shows correctly in GitHub markdown

  • Better behavior for bpipe test with send, and send for missing outputs

  • BUG: input patterns with dir in the pattern causing error / exception

  • Feature: support for reply -to in JMS bpipe agent

    If the reply-to or JMSReplyTo header is set, the bpipe agent will now
    send a message containing the original command and pipeline success / fail
    to specified reply-to address (allows eg: camel inOut routing)

  • BUG: Fix inverted logic causing incorrect detection of file existence
    when file is out of sync, causing file systems with high latency
    to fail sporadically

  • Better tolerance of file system sync issues for files referenced
    using file(...)

  • BUG: from sometimes failing to ensure specified input is resolved in
    preference to other available inputs preceding it in the
    pipeline

  • BUG: from sometimes selecting incorrect input in complex scenarios

  • BUG: fix file system watchers still created even if poller watcher specified,
    hence still resulting in file watcher exhaustion errors

  • BUG: -d option not respected in some situations, incorrect output dir may
    be used if default output already referenced

  • Feature: Support for Groovy 2.5.6 via optional gradle.properties setting

  • Feature: support for template dirs created by bpipe agent with incrementing
    values. Directories containing {inc} will be named with auto-incrementing
    counter.

  • Feature: 'terminateOnError' flag in send attributes causes
    notification failure as to be treated as error

  • BUG: improve unfriendly / incorrect behavior when duplicate mappings exist in transform

  • BUG: checks sometimes re-executing when not necessary

  • Feature: bpipe touch command updates files from current run
    of pipeline to new timestamps so they do not re-run on bpipe run.

Compare
Choose a tag to compare

Summary

This release includes several major new features including prelimary support for
running Bpipe pipelines on cloud providers (Google Cloud, Amazon Web Services), a
new merge point operator for making it easier to
construct parallel pipelines using scatter-gather parallelism. In addition to these,
significant work has been done to dramatically improve performance and reduce
resource consumption on very highly parallel pipelines with large numbers of
input / output files.

Features

  • Preliminary support for executing pipelines on Google Cloud Services
    (Compute Engine) and mounting storage for pipelines from Google Cloud
    Storage

  • Preliminary support for executing pipelines on Amazon Web Services
    using EC2 and mounting storage for pipelines from S3

  • The 'groovy' command can now run embedded groovy (executed outside
    Bpipe) using the groovy runtime bundled with Bpipe

  • Support aliasing to string values in addition to outputs

  • Experimental support for beforeRun hook in command config: execute
    arbitrary groovy code before a command executes

  • Many performance improvements, esp. for large, highly
    parallel pipelines

  • Support configuration for number of retries for status
    polling of HPC jobs (statusPollRetries setting)

  • Support for 'optional' inputs in pipelines: to make input optional,
    suffix with 'optional'. Also can add 'flag' to add flags
    in commands eg: ${input.csv.optional.flag('--csv')}

  • New operator: merge point operator (>>>) automatically configures a stage
    to merge outputs from a previous parallel split

  • Add region.bedFlag(flag) method for convenience when passing
    regions to commands

  • 'var' expressions may now be added in the main pipeline script,
    not just pipeline stages. These define optional
    variables, and provide a default.

  • JMS support now responds to 'ping' message with 'pong' reply
    if JMS 'Reply-To' is set to allow for status monitoring

Fixes

  • Fix incorrect "abnormal termination" messages
    printed to console when pipeline stopped with 'bpipe stop'

  • Fix incorrect 'pre-existing' printed for outputs that were
    created by pipeline

  • Fix genome not accessible in pipeline the first time downloaded,
    printing error

  • Re-execute checks if a commmand in the same stage has executed

  • synchronize initialization of dir watcher to fix sporadic
    ConcurrentModificationExceptions

  • Fix empty embedded parallel stage list causing resolution of incorrect
    downstream input

  • Fix leak of 'var' variables across branches when 'using' applied to
    pipeline stage

  • Fix error if 4 or more arguments passed to "to" in transform

  • Fix bpipe complaining spurious outputs not created on retry,
    but not original run

  • Fix some bugs where branch names were not being observed

  • Fix branch name sometimes inserted without separating period for transforms

  • Avoid redundantly putting branch name into files

  • Improved detail in error / log messages in a few places

  • Fix missing branch and '..' in filenames

  • Change: globally defined variables must now be held constant
    once pipeline starts

  • Fix split regions not stable between runs, set region id as branch
    name

  • Fix bed.split producing different splits if run repeatedly on same bed

  • Fix errors output if SLF4J referenced in user loaded libraries

  • Fix npe / improve error message when filter used with mismatching output ext

  • Fix error in stage body resulting in confusing 'no associated storage'
    assertion failure

  • Add 'allowForeign' option to 'from' to let it process non-outputs

  • Lessen the retries and retry interval when file cannot be cleaned
    up

Compare
Choose a tag to compare

Bpipe Version 0.9.9.6

Summary

This release includes some bug fixes that could cause confusing behahavior,
clearer and better messages in various error scenarios, and a couple of key
features such as support for gitlab, setting config parameters from within
your pipeline script, setting pipeline parameters from bpipe.config, and
others.

Changes

  • Fix: a send command encountered during bpipe test would not send the
    message but would record it as sent so that it would not be sent if
    the pipeline is then subsequently executed.

  • Fix: various issues with bpipe stats printing bad dates / times

  • you can now run bpipe stats while a pipeline is runnign to get realtime
    info

  • The bpipe agent now supports a ping command that will respond to confirm
    the agent is alive (via ActiveMQ).

  • You can now set parameters (variables) from within bpipe.config via
    a parameters section. This sets variables the same as if you set them
    via -p name=value when running bpipe.

  • You can now set pipeline limits (memory,concurrency,etc) from a config section
    within a pipeline script

  • Bpipe now supports gitlab through its send functionality. Issues can be created
    or updated added with syntax like: send issue(title: ...., assignee: ...) to gitlab

Compare
Choose a tag to compare

Release 0.9.9.5

Features:

  • Support for Bpipe 'agent' which can listen to requests via
    JMS (activemq for now) to run pipelines

  • Support sending json to JMS/ActiveMQ via send command

  • Utilitity function 'asQuotedList' formats lists of inputs
    as syntax compatible with js,python,groovy scripts

  • Experimental support for "sequential" or power operator

  • Print out the outputs to be created when run in test mode,
    in addition to the command to run (existing behavior)

  • Support for post-command hook to execute a script after every
    command

  • Add command start and stop time to output from 'bpipe query'

  • Add start and stop time to display in 'bpipe errors'

  • You can now specify java libraries as list in bpipe.config,
    in addition to the existing string form

  • Support for check review states, and improved check display

Fixes:

  • Fix exception/error if double input extension overlaps
    output with sng ext, eg. $input.foo.bar happens to match
    upstream $output.foo

  • Fix completed event sometimes not sent for failed stages

  • Fix failed checks written to std out at end of run

  • Fix situations where checks are re-executed when not
    required

  • Support supplying input files explicitly to run in
    run([...])

  • Support for JMS notifications of pipeline events

  • Fix error displayed by 'bpipe log' if no pipeline has ever
    been executed in dir

  • Prevent recursive load caused defining a segment inside a
    loaded file

  • Add segment structure to JSON pipeline representation
    Previously segments were expanded into their constituent stages
    in the JSON representation, and thus the higher level pipeline
    structure was lost. Now each node and edge in the group
    is given a type, and both the segment and its intenral
    structure are output as nodes / edges.

  • Fix sometimes getting wrong last run pid (thinks still
    running)

  • Change to make error in loaded script terminate pipeline
    instead of just printing error and continuing

  • Fix bpipe log stops working after some commands until bpipe run
    again

  • Avoid printing out cleanup message if zero files to cleanup

  • Move some of the docs out of Language and into Guides

  • Prevent exception in bpipe errors if command state not saved

  • Fix 'bpipe errors' not showing log output if hyphen in branch

  • Fix status command, add regression test

  • Groovy command in script now handles embedded quotes,
    which previously caused syntax errors

  • Fix error when nested segment made of single stage defined

Compare
Choose a tag to compare

Bpipe Version 0.9.9.4

This is a quick update that bundles a few new features with some critical fixes
to 0.9.9.3 based on feedback after it was released.

  • Fix issue #217
  • Add bpipe stats command to show runtimes of stages
  • Support config of concurrency limit in bpipe.config,
    via concurrency=<n>
  • Support for embedding config inside pipeline scripts via 'config'
    statement
  • Fix command status sometimes incorrect after stop preallocated
  • Fix pool commands in WAITING status not recognised as active
  • Fix trailing output from commands sometimes missing when output
    appears very fast at point where command terminates
Compare
Choose a tag to compare

Bpipe Version 0.9.9.3

Summary

Bpipe 0.9.9.3 contains a large number of bug fixes that affect
specific scenarios where unexpected behavior was being encountered. It also includes
some major enhancements including the ability to "pause" and resume a pipeline
without stopping in-progress jobs, and also support for pre-allocating
resources when running pipelines in environments where resources can
take time become available after they are requested. The new 'bpipe jobs'
command is also worth a look, especially the 'bpipe jobs -watch' option.

Download

Highlights

Enhancements

  • Support for preallocated resources for commands and allocated compute resources that
    persist between pipeline invocations. See the documentation
    for more details.

  • Support for using Groovy Grab/Grape to add libraries to pipeline
    (see documentation)

  • Improved output to 'bpipe query'

  • Greatly enhanced, real time display for 'bpipe jobs'

  • More informative display from 'bpipe status'

  • Generic custom options for queuing system jobs (see #193)

  • Improved behavior of stop and status when no pipeline running

  • Support to 'pause' a running pipeline via 'bpipe pause'

  • Tools can be located relative to pipeline for purpose of version probing

  • Experimental support for inline python, sqlite and groovy commands

  • Support to remake specific files via bpipe remake

  • Ability to split genomes into arbitrary sized parts specified in bases

  • Support for syntax of form ${output.withFlag("--someFlag")}

Bugfixes

  • Fix torque executor printing spurious errors when stopping jobs

  • fix spurious error produced by torque module request

  • Fix #209 - torque mem param variable missing in some situations

  • Fix sbatch/qsub failures not recognised correctly, producing
    misleading errors

  • Fix: files in output dirs not cleaned up if command fails

  • Fix #199: Stop command not halting jobs on some clusters

  • Fix timeout not treated as failure under SLURM

  • Fix split pattern on symbolic links in directory not resolving

  • Fix $outputs.ext (multiple outputs based on extension) broken

  • Fix genomes lacking 'chr' prefix not supported by 'chr' command

Compare
Choose a tag to compare

Changes for Bpipe Version 0.9.9.2

This release has focused primarily on performance, especially
for pipelines involving high concurrency and large numbers
of inputs and outputs. To achieve this, Bpipe now depends
on new Java APIs available in Java 1.7. This means that
Bpipe now requires Java 1.7 and will not work any longer
with Java 1.6.

Additionally, a range of small bugs have been fixed and
smaller features added.

Backwards Compatibility

Of note, there is one change that can
affect backwards compatibility. This relates to use of the
'transform' construct when multiple files of the same type are
transformed to multiple files of a different type. In the past
you could write a single transform such as:

transform('.txt') to('.csv') {
    ...
}

Previously this would transform every text file in the inputs
to a corresponding CSV file as output. From 0.9.9.2 forward
this will only transform a single input file. For multiple files, the
transform must now be specified as:

transform('*.txt') to('.csv') {
    ...
}

This change has been made to resolve ambiguity, since there was no
way previously to specify that a stage transforms exactly 1 input file
of a given type.

Change list

  • Large scale refactoring for performance with
    large numbers of inputs / outputs.
    Bpipe now requires Java 1.7!
  • Issue #180: bpipe doesn't allow empty branch list
  • Issue #184: chr not passed to nested parallel pipeline
  • fix wrong output referenced for 2nd double ext in
    produce eg: $output.bar.txt
  • support for modules in TorqueCommandExecutor
  • fix potential failure to cleanup files on error
  • Issue #175: Check state can be confused b/w branches
  • Handle errors in log init so that log fail doesn't prevent
    ability to run Bpipe (java logging tries to obtain exclusive
    locks which aren't supported on some file systems.
  • Update to groovy 2.4.6 (helps performance)
  • Fix ambiguous behavior of transform() when multiple inputs
    match transform pattern. New behavior requires transform('*.ext') to
    match upstream multiple files ending with .ext (minor possibilty for
    backwards incompatibility)
  • Fix output resolved as input for produce+double ext + numbered input
  • Fix exception when check saved without pass/fail set
  • Fix leak of tail / sed processes under OSX in some
    situations
  • Fix bootstrap theme for docs not recognised
  • Support to open a debugger Groovy console at any point
    in a pipeline using debug() function
  • Add settings.gradle to stop build failing if
    checked out to directory other than 'bpipe'
  • Fix possible hang when multi command used inside
    produce(...)
  • support for "mem_param" for torque executor
    this allows to choose which parameter Bpipe should use
    between "mem", "pvmem", etc. by setting mem_param="mem"
    in a command config
  • Issue #176: support multiple commands per config in multi
  • Various documentation fixes and cleanups
  • fix double file extension on input not resolving second and
    later input (eg: when referenced in form $input2.foo.bar)