Release 0.9.9.5

@ssadedin ssadedin released this Jan 14, 2018 · 28 commits to master since this release

Release 0.9.9.5

Features:

  • Support for Bpipe 'agent' which can listen to requests via
    JMS (activemq for now) to run pipelines

  • Support sending json to JMS/ActiveMQ via send command

  • Utilitity function 'asQuotedList' formats lists of inputs
    as syntax compatible with js,python,groovy scripts

  • Experimental support for "sequential" or power operator

  • Print out the outputs to be created when run in test mode,
    in addition to the command to run (existing behavior)

  • Support for post-command hook to execute a script after every
    command

  • Add command start and stop time to output from 'bpipe query'

  • Add start and stop time to display in 'bpipe errors'

  • You can now specify java libraries as list in bpipe.config,
    in addition to the existing string form

  • Support for check review states, and improved check display

Fixes:

  • Fix exception/error if double input extension overlaps
    output with sng ext, eg. $input.foo.bar happens to match
    upstream $output.foo

  • Fix completed event sometimes not sent for failed stages

  • Fix failed checks written to std out at end of run

  • Fix situations where checks are re-executed when not
    required

  • Support supplying input files explicitly to run in
    run([...])

  • Support for JMS notifications of pipeline events

  • Fix error displayed by 'bpipe log' if no pipeline has ever
    been executed in dir

  • Prevent recursive load caused defining a segment inside a
    loaded file

  • Add segment structure to JSON pipeline representation
    Previously segments were expanded into their constituent stages
    in the JSON representation, and thus the higher level pipeline
    structure was lost. Now each node and edge in the group
    is given a type, and both the segment and its intenral
    structure are output as nodes / edges.

  • Fix sometimes getting wrong last run pid (thinks still
    running)

  • Change to make error in loaded script terminate pipeline
    instead of just printing error and continuing

  • Fix bpipe log stops working after some commands until bpipe run
    again

  • Avoid printing out cleanup message if zero files to cleanup

  • Move some of the docs out of Language and into Guides

  • Prevent exception in bpipe errors if command state not saved

  • Fix 'bpipe errors' not showing log output if hyphen in branch

  • Fix status command, add regression test

  • Groovy command in script now handles embedded quotes,
    which previously caused syntax errors

  • Fix error when nested segment made of single stage defined

Bpipe 0.9.9.4

@ssadedin ssadedin released this Jun 10, 2017 · 152 commits to master since this release

Bpipe Version 0.9.9.4

This is a quick update that bundles a few new features with some critical fixes
to 0.9.9.3 based on feedback after it was released.

  • Fix issue #217
  • Add bpipe stats command to show runtimes of stages
  • Support config of concurrency limit in bpipe.config,
    via concurrency=<n>
  • Support for embedding config inside pipeline scripts via 'config'
    statement
  • Fix command status sometimes incorrect after stop preallocated
  • Fix pool commands in WAITING status not recognised as active
  • Fix trailing output from commands sometimes missing when output
    appears very fast at point where command terminates

0.9.9.3

@ssadedin ssadedin released this May 21, 2017 · 175 commits to master since this release

Bpipe Version 0.9.9.3

Summary

Bpipe 0.9.9.3 contains a large number of bug fixes that affect
specific scenarios where unexpected behavior was being encountered. It also includes
some major enhancements including the ability to "pause" and resume a pipeline
without stopping in-progress jobs, and also support for pre-allocating
resources when running pipelines in environments where resources can
take time become available after they are requested. The new 'bpipe jobs'
command is also worth a look, especially the 'bpipe jobs -watch' option.

Download

Highlights

Enhancements

  • Support for preallocated resources for commands and allocated compute resources that
    persist between pipeline invocations. See the documentation
    for more details.

  • Support for using Groovy Grab/Grape to add libraries to pipeline
    (see documentation)

  • Improved output to 'bpipe query'

  • Greatly enhanced, real time display for 'bpipe jobs'

  • More informative display from 'bpipe status'

  • Generic custom options for queuing system jobs (see #193)

  • Improved behavior of stop and status when no pipeline running

  • Support to 'pause' a running pipeline via 'bpipe pause'

  • Tools can be located relative to pipeline for purpose of version probing

  • Experimental support for inline python, sqlite and groovy commands

  • Support to remake specific files via bpipe remake

  • Ability to split genomes into arbitrary sized parts specified in bases

  • Support for syntax of form ${output.withFlag("--someFlag")}

Bugfixes

  • Fix torque executor printing spurious errors when stopping jobs

  • fix spurious error produced by torque module request

  • Fix #209 - torque mem param variable missing in some situations

  • Fix sbatch/qsub failures not recognised correctly, producing
    misleading errors

  • Fix: files in output dirs not cleaned up if command fails

  • Fix #199: Stop command not halting jobs on some clusters

  • Fix timeout not treated as failure under SLURM

  • Fix split pattern on symbolic links in directory not resolving

  • Fix $outputs.ext (multiple outputs based on extension) broken

  • Fix genomes lacking 'chr' prefix not supported by 'chr' command

0.9.9.2 Release

@ssadedin ssadedin released this Jul 18, 2016 · 318 commits to master since this release

Changes for Bpipe Version 0.9.9.2

This release has focused primarily on performance, especially
for pipelines involving high concurrency and large numbers
of inputs and outputs. To achieve this, Bpipe now depends
on new Java APIs available in Java 1.7. This means that
Bpipe now requires Java 1.7 and will not work any longer
with Java 1.6.

Additionally, a range of small bugs have been fixed and
smaller features added.

Backwards Compatibility

Of note, there is one change that can
affect backwards compatibility. This relates to use of the
'transform' construct when multiple files of the same type are
transformed to multiple files of a different type. In the past
you could write a single transform such as:

transform('.txt') to('.csv') {
    ...
}

Previously this would transform every text file in the inputs
to a corresponding CSV file as output. From 0.9.9.2 forward
this will only transform a single input file. For multiple files, the
transform must now be specified as:

transform('*.txt') to('.csv') {
    ...
}

This change has been made to resolve ambiguity, since there was no
way previously to specify that a stage transforms exactly 1 input file
of a given type.

Change list

  • Large scale refactoring for performance with
    large numbers of inputs / outputs.
    Bpipe now requires Java 1.7!
  • Issue #180: bpipe doesn't allow empty branch list
  • Issue #184: chr not passed to nested parallel pipeline
  • fix wrong output referenced for 2nd double ext in
    produce eg: $output.bar.txt
  • support for modules in TorqueCommandExecutor
  • fix potential failure to cleanup files on error
  • Issue #175: Check state can be confused b/w branches
  • Handle errors in log init so that log fail doesn't prevent
    ability to run Bpipe (java logging tries to obtain exclusive
    locks which aren't supported on some file systems.
  • Update to groovy 2.4.6 (helps performance)
  • Fix ambiguous behavior of transform() when multiple inputs
    match transform pattern. New behavior requires transform('*.ext') to
    match upstream multiple files ending with .ext (minor possibilty for
    backwards incompatibility)
  • Fix output resolved as input for produce+double ext + numbered input
  • Fix exception when check saved without pass/fail set
  • Fix leak of tail / sed processes under OSX in some
    situations
  • Fix bootstrap theme for docs not recognised
  • Support to open a debugger Groovy console at any point
    in a pipeline using debug() function
  • Add settings.gradle to stop build failing if
    checked out to directory other than 'bpipe'
  • Fix possible hang when multi command used inside
    produce(...)
  • support for "mem_param" for torque executor
    this allows to choose which parameter Bpipe should use
    between "mem", "pvmem", etc. by setting mem_param="mem"
    in a command config
  • Issue #176: support multiple commands per config in multi
  • Various documentation fixes and cleanups
  • fix double file extension on input not resolving second and
    later input (eg: when referenced in form $input2.foo.bar)

Re-Release of 0.9.9

@ssadedin ssadedin released this Feb 9, 2016 · 388 commits to master since this release

This release is only to correct the 0.9.9 release being created as .tar file instead of .tar.gz. Please see release notes for version 0.9.9.

0.9.9 Release

@ssadedin ssadedin released this Jan 17, 2016 · 388 commits to master since this release

Changes for Bpipe Version 0.9.9

NOTE: see 0.9.9_beta_1 changes for further changes since previous release

  • SGE: Merge branch 'sge_fixes', including:
    • Fix #156, #142, Bpipe may hang or ignore job cancel / delete
      using SGE executor
    • add -terse to sge default command options
    • change to support separate options:
      'sge_pe' and 'procs' rather than combined.
      (Old combined form still supported for back compat.)
    • jobs are created using user modifiable command template,
      specified via jobTemplate configuration parameter
      which can be object with string value of path to template,
      or a closure which will be executed dynamically to obtain the
      path
    • fix incorrect reporting that stop command
      failed in sge
    • use Bpipe resolved working directory instead of $PWD since
      it is not guaranteed to be defined
    • fix cases of failure due to string procs variable
      getting overridden with integer (dynamic calculation
      of $threads variable)
    • fix incorrectly sending slots as well as pe for some
      configurations
    • support individual parameters for SGE options instead of compiling
      them into 'additional_options'
  • Fix choosing "terminate" not working for some executors
    (issue #162)
  • PBS-pro: combine multiple resource limits into single
    statement, add support for procs, memory requests
  • Experimental option for some executors to disable buffering of
    standard output, applies to torque, slurm, pbs-pro. Set "stdbuf" as
    option to have bpipe run commands using 'stdbuf' command to disable
    output buffering, when output from commands is delayed in appearing
    in Bpipe log.
  • Performance optimisations for large / complex pipelines:
    • Cache output graph in serialised form to avoid reading property
      files at startup
    • Reduce massive logging of outputs when large number of
      inputs and outputs in single stage
  • Add 'bpipe errors' and 'bpipe log -e' command to
    show output for failed commands
  • Add support to display per command output using the
    bpipe log -c
  • Fix 'bpipe test' printing exceptions and hanging on 'multi'
    statement
  • Fix regression causing possible failure when stage has
    outputs but no inputs
  • update gradle wrapper script, include changes omitted from
    2.0 upgrade
  • Fix files in subdirectory not resolved when whole file name
    matches input extension
  • Fix variable specified with 'using' not overriding global value
  • Fix 'check' results possibly leaking between branches
  • Fix reading output.dir causing potential modification of
    output directory

Bpipe 0.9.9 Beta 1

@ssadedin ssadedin released this Aug 24, 2015 · 440 commits to master since this release

This release contains many, many fixes and an important change in how concurrency is handled in some situations which should allow for much better resource utilisation in some situations. Please help test this beta release!

=========== Changes for Bpipe Version 0.9.9_beta_1 ============

  • Fix content type not set causing errors with some smtp
    connections
  • Issue #155: Update to gradle 2.0 to avoid compile failure with JDK9
  • Performance optimisations for large numbers of inputs /
    outputs
  • Should report error if is referenced and
    finds no inputs
  • Give a hint about which 'from' clause
    pattern did not find a match in error message
  • Fix typo causing exception while logging error message
  • Fix: setting branch name from within stage causes incorrect
    branches downstream
  • Fix input resolved as output in rare circumstances
  • Fix transform ... to followed by double
    output variable extensions (.foo.bar) failing with confusing
    error message
  • Fix procs not set in torque job when using variable
  • Make bpipe return exit code reflecting whether pipeline
    succeeded or failed
  • Fix: bpipe may fail to cleanup files created in subdirectories
  • Better / fairer scheduling for dynamically allocated $thread variables
  • Support for ranges in procs parameter, eg: procs = 1..10, or
    uses(threads:1..10) { ... }
  • Fix: failure to parallelize stages inside a list added to
    another list, eg: [foo, bar] + [fubar,fug]
  • Issue #148: bpipe parsing commandline memory
    specification incorrectly
  • Issue #150: Cannot pass blank parameter via command line
  • Various documentation fixes
  • Fix: junit tests were not executed
  • Additional SGE options, SGE options moved inside script
  • fix: commands run using torque / pbs not using -e flag for bash,
    thus only treated as failed if last command fails
  • support for aliasing inputs to outputs via 'forward to
    ' to allow an input to be virtually treated as a named
    output within the pipeline
  • fix: bpipe terminating if ssh session terminates while showing bpipe log
    in foreground directly after initial launch
  • save output from LSF (to assist in debugging failed LSF commands)
  • fix stages not found when defined in bpipe lib folder and referenced in
    segment declaration if used after load
  • Switch to use html format for emails by default
  • allow user to configure email format with 'format' attribute in
    configuration
  • allow multiple templates per notification type with different formats
  • fix problems in using 'send'
    and 'file' to send explicit files based on templates
  • Fix high thread concurrency used when large number of concurrent branches
    created by one pipeline branch
  • Explicitly destroy process after waiting
    until streams closed to ensure file handles not leaked
  • fix some cases where Bpipe could report file as missing when
    created in slow syncing distributed file system
  • Allow more screen width when using 'bpipe checks', better formatting
    of check display
  • Add option to list checks rather than enter interactive mode
  • fix some cases where commands might not
    be stopped, and outputs not cleaned up when using bpipe
    stop. Update saved command info after cleanup.
  • ensure stop time is always set for commands
  • close possible file handle leak in (undocumented) capture command
  • fix: more robust checking for file existence on distributed file systems which
    may exhibit delays before files become visible locally
  • Avoid misleading termination message when user deliberately stops
    pipeline
  • Incorrect input used as basis for output file name in transforms inside
    'from', if competing input available from previous stage
  • fix: rare deadlocks - use dedicated thread pool per pipeline tier to
    avoid possibility of deadlocks in heavily nested pipelines

0.9.8.7 Release

@ssadedin ssadedin released this Mar 29, 2015 · 557 commits to master since this release

Changes for Bpipe Version 0.9.8.7

  • check /proc on linux rather than relying on kill
    0 to see if pipeline is running (enables cross
    user support for "bpipe log")
  • Addition of documentation to Bpipe distro
  • fix failure to cleanup files with no parent directory
    (regression)
  • Add future import for Python 2 and 3 compatibility
    (fix failure on Mac when Python3 is default)
  • switch to HTML format for default email. to switch back
    to text, use notifications { smtp { format="text" } } in
    bpipe.config.
  • Fix presence of subdirectory in bpipe output meta data folder
    causing bpipe to spew errors
  • Fix files sometimes not cleaned up on distributd file system
    where lag exists in synchronizing b/w nodes
  • Issue #128: use qsub -N to name job according to stage/filename
  • support for overriding a stage with an alternative implementation
    within a particular branch
  • fix: authentication failures on passwordless SMTP
    server
  • fix failure in cleanup if output properties file
    did not get optional canonicalPath property added
  • comment out verbose log message creating large bpipe
    logfiles for long runs
  • avoid encoding whole directory path into internal property file
    names (fix 'file path too long errors' on file systems allowing only
    256 chars in file names)
  • fix parent branches not visible in children via branch property
  • fix method missing error when multiplying by map by list
  • Reformatting and summarizing of errors at end of pipeline to make
    it easier to see where & why pipelines failed
  • Aggregating identical errors from
    multiple branches together in output so that failures from
    many branches are less verbose
  • Support for 'load' within a pipeline stage. Loads into the
    context of current branch rather than global. Pipeline stages
    not supported, only variables.
  • prevent error if files used by Bpipe are not writable but
    are deletable
  • prevent premature wildcard expansion of
    quoted arguments passed to Bpipe
  • support for cleanup as command in pipeline stage
  • prevent error if files used by Bpipe are not writable but
    are deletable
  • attempt to ensure log messages written
    even if exit due to outofmemory error
  • log message if file is not cleaned up because it doesn't
    exist: help to diagnose problems with files not cleaned up
  • better log info about where config files are
  • allow memory to be integer for torque/slurm/etc, fix error
    querying status in torque/slurm/etc, include queuing system
    command id in bpipe status output
  • show pending and waiting jobs with bpipe status, don't throw ugly error
    if no commands ever run
  • Fix: downstream references to inputs
    by index throwing "expected more inputs" error incorrectly
  • avoid confusing stack overflow when user defines branch
    variable with same name as pipeline stage
  • accept 'MB' or 'GB' in command line option for memory,
    cache runtime directory instead of creating new File objects
    all the time to figure it out
  • prevent null pointer exception when list containing null
    provided to 'from'
  • fix multiple inputs not
    forwarding correctly when referenced using "$inputs"
  • Bug fixes for lsf, added walltime option for Lsf
  • caching of dependency lookups by
    canonical path to speed up large, complex pipelines
  • caching of canonical path with check of basePath to ensure
    consistency
  • fix dependency issues after
    cleanup when non-default output directory used
  • Performance optimisations for systems where canonicalPath is
    very slow, better logging to resolve dependency issues

0.9.8.6 Release

@ssadedin ssadedin released this Oct 22, 2014 · 639 commits to master since this release

=========== Changes for Bpipe Version 0.9.8.6 ============

  • fix: errors may printed about files not existing when attempting
    to move to trash
  • fix bpipe jobs not printing jobs after user ctrl-c's out of
    run command
  • fix incorrect default for procs_mode variable for torque
  • fix ugly stack trace error when transform called without arguments
  • fix failing to escape trailing . in split patterns
  • branch.dir now ha a default value so that it is easier to put in reusable
    pipeline stages
  • Ensure walltime gets interpreted as
    integer even if passed as String / BigDecimal / Long etc
  • fix trash option not working in bpipe cleanup
  • Fix default output directory not applied to second
    produce argument if first specifies a directory
  • fix produce() with reference only to output.dir causing
    retry test to incorrectly indicate commands would run again
  • Fix incorrect file reported for errors when multiple
    external files loaded, added some comments, removed hard
    tabs
  • fix branch variable not overriding default value of
    variable specified with var
  • Fix files not checked after 'produce' if previously output and then
    removed outside of bpipe
  • Fix files not checked after 'produce' if
    previously output and then removed outside of bpipe
  • fix 'inputs' variable with double file extension not always resolving
    only files matching both extensions
  • fix output() function causing spurious errors about incompatible inputs
    when combined with output variable extensions
  • fix command not running when explicitly specified by produce() if another
    implicit output referenced by command
  • Support for produce using wildcard based on regex (via ~)
  • Add information about total run time to report and command log
  • support to specify branch filter on command line, to limit
    running to a specific branch (for testing, usually)
  • fix explicit output() combined with produce(...) resulting in
    inconsistent behavior
  • Fix stack traces printed out when errors occur to use actual name of user
    scripts instead of groovy generated ones
  • fix build.gradle not working with existing gradle wrapper
  • Implement default behavior for transform with
    more to() values than from() values instead of failing
  • Fix output referenced inside check causing check to be
    skipped / behave incorrectly