Skip to content

@ssadedin ssadedin released this Apr 29, 2020 · 1 commit to master since this release

Summary

This release includes a number of smaller features and improvements and
some important fixes.

NOTE: this version of Bpipe requires Java 1.8 or higher

Features

  • Ability to filter gitlab issues identified
    by title search for updating

  • Feature: support for 'debug' command that launches
    groovy shell. Add debug() to pipeline stage to launch
    interactive shell.

  • Revamp monitoring of command status for PBS/Torque
    based pipelines, by pooling calls to qstat between
    commands.

  • Feature: archive command to clean up .bpipe directory into
    restorable zip file

  • Support referencing params set on command line in
    bpipe.config

  • Bpipe pipelines can now accept the first line as a "shebang"
    so that bpipe pipelines become executable scripts. ie: make
    the first line of your pipeline: #!/path/to/bpipe

Fixes

  • Fix not completely robust reading of exit code for
    local commands (sometimes command succeeds when shouldn't)

  • Fix local commands not always terminating on explicit stop

  • Add setsid by default to local jobs on linux -
    prevents some cases where jobs abort when precessing
    ctrl-c after starting bpipe

  • Fix new gitlab issues with large content failing to send

  • Fix possible runaway killing of processes when
    local preallocated jobs are explicitly stopped with
    bpipe stop

  • Fix file notifications not always incrementing and
    therefore overwriting previous notification

  • Fix notification manager not respecting timeout to allow for
    setup of channels

  • Fix possible error due to race condition
    reading read PID with local command executor

  • Fix file handle leak on some file systems with large scale
    highly parallelised, long running pipelines

  • Fix the default file notification channel overwriting files from
    other send channels

  • Fix possible error if input is referenced in details of send

Assets 4

@ssadedin ssadedin released this Oct 17, 2019 · 48 commits to master since this release

Changes for Bpipe Version 0.9.9.8

  • Feature: options statement for easy parsing of script specific
    command line arguments

  • Add workaround to fix possible failure on old pipelines checking
    file existence

  • BUG: Fix spurious error printed if broken symbolic link present

  • BUG: prevent agent launched commands hanging due to if bpipe prompts
    for information

  • Feature: add fileSystemSyncTimeMs which allows for customizable delay in
    file system sync before failing pipeline on missing output

  • BUG: fix regression - transform using regex does not substitute groups

    eg: transform('(.*)_R1.fastq') to("$1.txt") { ... }

  • Many fixes / optimisations to GCloud usage:

    • make workingDirectory specified for stage automatically
      placed within mounted storage

    • inherit region from main config inside
      filesystems, so it does not need to be double specified

    • support preemptible flag passed to google cloud
      instances

    • add sha1 hash of pipeline path / host to
      instanceid to make it unique b/w pipelines

    • fix region configured
      for cloud executor overriding region specified with -L

    • search for root input files in all filesystems, not only
      those configured as storages for stages

    • fix zone flag missing from gcloud commands, don't
      die mounting storage if instance not created

    • support using split / partitioned regions in cloud provider storage

  • BUG: Don't start executor pools when user is only testing the pipeline

  • BUG: ensure that executors clean up even if they error in stop
    command

  • Fix table output format so it shows correctly in GitHub markdown

  • Better behavior for bpipe test with send, and send for missing outputs

  • BUG: input patterns with dir in the pattern causing error / exception

  • Feature: support for reply -to in JMS bpipe agent

    If the reply-to or JMSReplyTo header is set, the bpipe agent will now
    send a message containing the original command and pipeline success / fail
    to specified reply-to address (allows eg: camel inOut routing)

  • BUG: Fix inverted logic causing incorrect detection of file existence
    when file is out of sync, causing file systems with high latency
    to fail sporadically

  • Better tolerance of file system sync issues for files referenced
    using file(...)

  • BUG: from sometimes failing to ensure specified input is resolved in
    preference to other available inputs preceding it in the
    pipeline

  • BUG: from sometimes selecting incorrect input in complex scenarios

  • BUG: fix file system watchers still created even if poller watcher specified,
    hence still resulting in file watcher exhaustion errors

  • BUG: -d option not respected in some situations, incorrect output dir may
    be used if default output already referenced

  • Feature: Support for Groovy 2.5.6 via optional gradle.properties setting

  • Feature: support for template dirs created by bpipe agent with incrementing
    values. Directories containing {inc} will be named with auto-incrementing
    counter.

  • Feature: 'terminateOnError' flag in send attributes causes
    notification failure as to be treated as error

  • BUG: improve unfriendly / incorrect behavior when duplicate mappings exist in transform

  • BUG: checks sometimes re-executing when not necessary

  • Feature: bpipe touch command updates files from current run
    of pipeline to new timestamps so they do not re-run on bpipe run.

Assets 4

@ssadedin ssadedin released this Apr 23, 2019 · 133 commits to master since this release

Summary

This release includes several major new features including prelimary support for
running Bpipe pipelines on cloud providers (Google Cloud, Amazon Web Services), a
new merge point operator for making it easier to
construct parallel pipelines using scatter-gather parallelism. In addition to these,
significant work has been done to dramatically improve performance and reduce
resource consumption on very highly parallel pipelines with large numbers of
input / output files.

Features

  • Preliminary support for executing pipelines on Google Cloud Services
    (Compute Engine) and mounting storage for pipelines from Google Cloud
    Storage

  • Preliminary support for executing pipelines on Amazon Web Services
    using EC2 and mounting storage for pipelines from S3

  • The 'groovy' command can now run embedded groovy (executed outside
    Bpipe) using the groovy runtime bundled with Bpipe

  • Support aliasing to string values in addition to outputs

  • Experimental support for beforeRun hook in command config: execute
    arbitrary groovy code before a command executes

  • Many performance improvements, esp. for large, highly
    parallel pipelines

  • Support configuration for number of retries for status
    polling of HPC jobs (statusPollRetries setting)

  • Support for 'optional' inputs in pipelines: to make input optional,
    suffix with 'optional'. Also can add 'flag' to add flags
    in commands eg: ${input.csv.optional.flag('--csv')}

  • New operator: merge point operator (>>>) automatically configures a stage
    to merge outputs from a previous parallel split

  • Add region.bedFlag(flag) method for convenience when passing
    regions to commands

  • 'var' expressions may now be added in the main pipeline script,
    not just pipeline stages. These define optional
    variables, and provide a default.

  • JMS support now responds to 'ping' message with 'pong' reply
    if JMS 'Reply-To' is set to allow for status monitoring

Fixes

  • Fix incorrect "abnormal termination" messages
    printed to console when pipeline stopped with 'bpipe stop'

  • Fix incorrect 'pre-existing' printed for outputs that were
    created by pipeline

  • Fix genome not accessible in pipeline the first time downloaded,
    printing error

  • Re-execute checks if a commmand in the same stage has executed

  • synchronize initialization of dir watcher to fix sporadic
    ConcurrentModificationExceptions

  • Fix empty embedded parallel stage list causing resolution of incorrect
    downstream input

  • Fix leak of 'var' variables across branches when 'using' applied to
    pipeline stage

  • Fix error if 4 or more arguments passed to "to" in transform

  • Fix bpipe complaining spurious outputs not created on retry,
    but not original run

  • Fix some bugs where branch names were not being observed

  • Fix branch name sometimes inserted without separating period for transforms

  • Avoid redundantly putting branch name into files

  • Improved detail in error / log messages in a few places

  • Fix missing branch and '..' in filenames

  • Change: globally defined variables must now be held constant
    once pipeline starts

  • Fix split regions not stable between runs, set region id as branch
    name

  • Fix bed.split producing different splits if run repeatedly on same bed

  • Fix errors output if SLF4J referenced in user loaded libraries

  • Fix npe / improve error message when filter used with mismatching output ext

  • Fix error in stage body resulting in confusing 'no associated storage'
    assertion failure

  • Add 'allowForeign' option to 'from' to let it process non-outputs

  • Lessen the retries and retry interval when file cannot be cleaned
    up

Assets 3

@ssadedin ssadedin released this Jul 20, 2018 · 360 commits to master since this release

Bpipe Version 0.9.9.6

Summary

This release includes some bug fixes that could cause confusing behahavior,
clearer and better messages in various error scenarios, and a couple of key
features such as support for gitlab, setting config parameters from within
your pipeline script, setting pipeline parameters from bpipe.config, and
others.

Changes

  • Fix: a send command encountered during bpipe test would not send the
    message but would record it as sent so that it would not be sent if
    the pipeline is then subsequently executed.

  • Fix: various issues with bpipe stats printing bad dates / times

  • you can now run bpipe stats while a pipeline is runnign to get realtime
    info

  • The bpipe agent now supports a ping command that will respond to confirm
    the agent is alive (via ActiveMQ).

  • You can now set parameters (variables) from within bpipe.config via
    a parameters section. This sets variables the same as if you set them
    via -p name=value when running bpipe.

  • You can now set pipeline limits (memory,concurrency,etc) from a config section
    within a pipeline script

  • Bpipe now supports gitlab through its send functionality. Issues can be created
    or updated added with syntax like: send issue(title: ...., assignee: ...) to gitlab

Assets 4

@ssadedin ssadedin released this Jan 14, 2018 · 395 commits to master since this release

Release 0.9.9.5

Features:

  • Support for Bpipe 'agent' which can listen to requests via
    JMS (activemq for now) to run pipelines

  • Support sending json to JMS/ActiveMQ via send command

  • Utilitity function 'asQuotedList' formats lists of inputs
    as syntax compatible with js,python,groovy scripts

  • Experimental support for "sequential" or power operator

  • Print out the outputs to be created when run in test mode,
    in addition to the command to run (existing behavior)

  • Support for post-command hook to execute a script after every
    command

  • Add command start and stop time to output from 'bpipe query'

  • Add start and stop time to display in 'bpipe errors'

  • You can now specify java libraries as list in bpipe.config,
    in addition to the existing string form

  • Support for check review states, and improved check display

Fixes:

  • Fix exception/error if double input extension overlaps
    output with sng ext, eg. $input.foo.bar happens to match
    upstream $output.foo

  • Fix completed event sometimes not sent for failed stages

  • Fix failed checks written to std out at end of run

  • Fix situations where checks are re-executed when not
    required

  • Support supplying input files explicitly to run in
    run([...])

  • Support for JMS notifications of pipeline events

  • Fix error displayed by 'bpipe log' if no pipeline has ever
    been executed in dir

  • Prevent recursive load caused defining a segment inside a
    loaded file

  • Add segment structure to JSON pipeline representation
    Previously segments were expanded into their constituent stages
    in the JSON representation, and thus the higher level pipeline
    structure was lost. Now each node and edge in the group
    is given a type, and both the segment and its intenral
    structure are output as nodes / edges.

  • Fix sometimes getting wrong last run pid (thinks still
    running)

  • Change to make error in loaded script terminate pipeline
    instead of just printing error and continuing

  • Fix bpipe log stops working after some commands until bpipe run
    again

  • Avoid printing out cleanup message if zero files to cleanup

  • Move some of the docs out of Language and into Guides

  • Prevent exception in bpipe errors if command state not saved

  • Fix 'bpipe errors' not showing log output if hyphen in branch

  • Fix status command, add regression test

  • Groovy command in script now handles embedded quotes,
    which previously caused syntax errors

  • Fix error when nested segment made of single stage defined

Assets 4

@ssadedin ssadedin released this Jun 10, 2017 · 519 commits to master since this release

Bpipe Version 0.9.9.4

This is a quick update that bundles a few new features with some critical fixes
to 0.9.9.3 based on feedback after it was released.

  • Fix issue #217
  • Add bpipe stats command to show runtimes of stages
  • Support config of concurrency limit in bpipe.config,
    via concurrency=<n>
  • Support for embedding config inside pipeline scripts via 'config'
    statement
  • Fix command status sometimes incorrect after stop preallocated
  • Fix pool commands in WAITING status not recognised as active
  • Fix trailing output from commands sometimes missing when output
    appears very fast at point where command terminates
Assets 3

@ssadedin ssadedin released this May 21, 2017 · 542 commits to master since this release

Bpipe Version 0.9.9.3

Summary

Bpipe 0.9.9.3 contains a large number of bug fixes that affect
specific scenarios where unexpected behavior was being encountered. It also includes
some major enhancements including the ability to "pause" and resume a pipeline
without stopping in-progress jobs, and also support for pre-allocating
resources when running pipelines in environments where resources can
take time become available after they are requested. The new 'bpipe jobs'
command is also worth a look, especially the 'bpipe jobs -watch' option.

Download

Highlights

Enhancements

  • Support for preallocated resources for commands and allocated compute resources that
    persist between pipeline invocations. See the documentation
    for more details.

  • Support for using Groovy Grab/Grape to add libraries to pipeline
    (see documentation)

  • Improved output to 'bpipe query'

  • Greatly enhanced, real time display for 'bpipe jobs'

  • More informative display from 'bpipe status'

  • Generic custom options for queuing system jobs (see #193)

  • Improved behavior of stop and status when no pipeline running

  • Support to 'pause' a running pipeline via 'bpipe pause'

  • Tools can be located relative to pipeline for purpose of version probing

  • Experimental support for inline python, sqlite and groovy commands

  • Support to remake specific files via bpipe remake

  • Ability to split genomes into arbitrary sized parts specified in bases

  • Support for syntax of form ${output.withFlag("--someFlag")}

Bugfixes

  • Fix torque executor printing spurious errors when stopping jobs

  • fix spurious error produced by torque module request

  • Fix #209 - torque mem param variable missing in some situations

  • Fix sbatch/qsub failures not recognised correctly, producing
    misleading errors

  • Fix: files in output dirs not cleaned up if command fails

  • Fix #199: Stop command not halting jobs on some clusters

  • Fix timeout not treated as failure under SLURM

  • Fix split pattern on symbolic links in directory not resolving

  • Fix $outputs.ext (multiple outputs based on extension) broken

  • Fix genomes lacking 'chr' prefix not supported by 'chr' command

Assets 2

@ssadedin ssadedin released this Jul 18, 2016 · 685 commits to master since this release

Changes for Bpipe Version 0.9.9.2

This release has focused primarily on performance, especially
for pipelines involving high concurrency and large numbers
of inputs and outputs. To achieve this, Bpipe now depends
on new Java APIs available in Java 1.7. This means that
Bpipe now requires Java 1.7 and will not work any longer
with Java 1.6.

Additionally, a range of small bugs have been fixed and
smaller features added.

Backwards Compatibility

Of note, there is one change that can
affect backwards compatibility. This relates to use of the
'transform' construct when multiple files of the same type are
transformed to multiple files of a different type. In the past
you could write a single transform such as:

transform('.txt') to('.csv') {
    ...
}

Previously this would transform every text file in the inputs
to a corresponding CSV file as output. From 0.9.9.2 forward
this will only transform a single input file. For multiple files, the
transform must now be specified as:

transform('*.txt') to('.csv') {
    ...
}

This change has been made to resolve ambiguity, since there was no
way previously to specify that a stage transforms exactly 1 input file
of a given type.

Change list

  • Large scale refactoring for performance with
    large numbers of inputs / outputs.
    Bpipe now requires Java 1.7!
  • Issue #180: bpipe doesn't allow empty branch list
  • Issue #184: chr not passed to nested parallel pipeline
  • fix wrong output referenced for 2nd double ext in
    produce eg: $output.bar.txt
  • support for modules in TorqueCommandExecutor
  • fix potential failure to cleanup files on error
  • Issue #175: Check state can be confused b/w branches
  • Handle errors in log init so that log fail doesn't prevent
    ability to run Bpipe (java logging tries to obtain exclusive
    locks which aren't supported on some file systems.
  • Update to groovy 2.4.6 (helps performance)
  • Fix ambiguous behavior of transform() when multiple inputs
    match transform pattern. New behavior requires transform('*.ext') to
    match upstream multiple files ending with .ext (minor possibilty for
    backwards incompatibility)
  • Fix output resolved as input for produce+double ext + numbered input
  • Fix exception when check saved without pass/fail set
  • Fix leak of tail / sed processes under OSX in some
    situations
  • Fix bootstrap theme for docs not recognised
  • Support to open a debugger Groovy console at any point
    in a pipeline using debug() function
  • Add settings.gradle to stop build failing if
    checked out to directory other than 'bpipe'
  • Fix possible hang when multi command used inside
    produce(...)
  • support for "mem_param" for torque executor
    this allows to choose which parameter Bpipe should use
    between "mem", "pvmem", etc. by setting mem_param="mem"
    in a command config
  • Issue #176: support multiple commands per config in multi
  • Various documentation fixes and cleanups
  • fix double file extension on input not resolving second and
    later input (eg: when referenced in form $input2.foo.bar)
Assets 3

@ssadedin ssadedin released this Feb 9, 2016 · 755 commits to master since this release

This release is only to correct the 0.9.9 release being created as .tar file instead of .tar.gz. Please see release notes for version 0.9.9.

Assets 3

@ssadedin ssadedin released this Jan 17, 2016 · 755 commits to master since this release

Changes for Bpipe Version 0.9.9

NOTE: see 0.9.9_beta_1 changes for further changes since previous release

  • SGE: Merge branch 'sge_fixes', including:
    • Fix #156, #142, Bpipe may hang or ignore job cancel / delete
      using SGE executor
    • add -terse to sge default command options
    • change to support separate options:
      'sge_pe' and 'procs' rather than combined.
      (Old combined form still supported for back compat.)
    • jobs are created using user modifiable command template,
      specified via jobTemplate configuration parameter
      which can be object with string value of path to template,
      or a closure which will be executed dynamically to obtain the
      path
    • fix incorrect reporting that stop command
      failed in sge
    • use Bpipe resolved working directory instead of $PWD since
      it is not guaranteed to be defined
    • fix cases of failure due to string procs variable
      getting overridden with integer (dynamic calculation
      of $threads variable)
    • fix incorrectly sending slots as well as pe for some
      configurations
    • support individual parameters for SGE options instead of compiling
      them into 'additional_options'
  • Fix choosing "terminate" not working for some executors
    (issue #162)
  • PBS-pro: combine multiple resource limits into single
    statement, add support for procs, memory requests
  • Experimental option for some executors to disable buffering of
    standard output, applies to torque, slurm, pbs-pro. Set "stdbuf" as
    option to have bpipe run commands using 'stdbuf' command to disable
    output buffering, when output from commands is delayed in appearing
    in Bpipe log.
  • Performance optimisations for large / complex pipelines:
    • Cache output graph in serialised form to avoid reading property
      files at startup
    • Reduce massive logging of outputs when large number of
      inputs and outputs in single stage
  • Add 'bpipe errors' and 'bpipe log -e' command to
    show output for failed commands
  • Add support to display per command output using the
    bpipe log -c
  • Fix 'bpipe test' printing exceptions and hanging on 'multi'
    statement
  • Fix regression causing possible failure when stage has
    outputs but no inputs
  • update gradle wrapper script, include changes omitted from
    2.0 upgrade
  • Fix files in subdirectory not resolved when whole file name
    matches input extension
  • Fix variable specified with 'using' not overriding global value
  • Fix 'check' results possibly leaking between branches
  • Fix reading output.dir causing potential modification of
    output directory
Assets 3
You can’t perform that action at this time.