Bpipe 0.9.14

Bpipe 0.9.14 Release Notes

This is a major new release after 18 months of development. Significant focuses include improved AWS support for enterprise use, developer ergonomics (IDE, interactive dev-mode, shell, resource tracking, info and status output), improved container support, among many other changes.

New Commands & Enhancements

• bpipe completions: Added shell completion support — run eval "$(bpipe completions)" to enable bash tab-completion for Bpipe commands and options.
• bpipe jobs: Enhanced with new options including -all (show completed jobs), -age (filter by hours), -m (filter by directory name), -watch (continuous monitoring), pass/fail status display, and a script name column added to job info output.
• bpipe stats: Major overhaul — now reports per-stage evaluation statistics, input sizes, total CPU usage (including in-progress commands), region size per stage, and SLURM utilisation monitoring.
• bpipe dev: Runs pipeline in interactive "dev" mode, accepts debug as an interactive response to drop into a debug console for testing.
• bpipe history: Now includes timestamps for each historical run entry.
• bpipe generate-dsl: New command that generates GDSL (Groovy DSL descriptor) and DSLD (Eclipse DSL descriptor) files for IDE code completion in editors such as Eclipse and IntelliJ.

Execution & Compute

• GPU support in SLURM: SLURM executor now supports gpu as a managed resource, requesting GPU resources via --gres=gpu:N.
• Storage as a managed resource: Stages can now declare storage_space (e.g., "10 GB") as a managed resource — Bpipe will check available disk space and queue commands until sufficient space is free.
• SLURM OUT_OF_MEMORY detection: Improved error handling for SLURM jobs, including automatic detection of OUT_OF_MEMORY status and propagation as a meaningful pipeline error.
• Torque executor: Added @CompileStatic to the Torque executor for improved performance and reliability.
• Docker inherit_user: When using inherit_user = true, the Docker wrapper now also sets the user's group membership inside the container.
• Singularity: Added support for a custom path or executable name for the singularity command via configuration.
• --skip-checks option: New flag to bypass pipeline validation checks during development, speeding up the dev iteration cycle.
• bpipe run parallelism: Default concurrency increased for the stop command and general execution pipeline to improve shutdown performance.

Cloud / AWS

• Instance profiles: EC2 instances can now be launched with a configurable instanceProfile IAM role attached.
• Security groups & subnets: Added securityGroupId and subnetId configuration options for EC2 instance launches.
• Custom instance name tags: The Name tag of EC2 instances can now be customised via configuration.
• Private IP resolution: Bpipe can now resolve and use private IP addresses for EC2 instances that do not have public IPs assigned.
• IMDSv1/v2 control: Added imdsSupport option to choose between IMDSv1 and IMDSv2 when launching EC2 instances.
• Transfer speed monitoring: AWS S3 and EC2 file transfers now report transfer speed during execution.
• Default credentials provider: When accessKey is set to "auto", Bpipe now uses the AWS default credentials provider chain for more flexible authentication.
• SQS notifications: Added support for initialising SQS notification channels from environment variables and using temporary session credentials.
• EC2 role-based credentials: Automatic querying of role-based access credentials when running in EC2 environments.

Configuration & Environment

• BPIPE_ENV environment variable: Set the default Bpipe environment (bpipe.config profile) via the BPIPE_ENV environment variable instead of passing -e on every invocation.
• BPIPE_DB_DIR environment variable: The central job database directory (.bpipedb) location is now configurable via this environment variable.
• Agent environment per config: When running via the Bpipe agent, you can now configure multiple environments with different settings
• Config auto-reload: Configuration files (e.g., bpipe.config) are now automatically reloaded when modified if paused in a debug break at bpipe dev,
  avoiding the need to restart the pipeline to test config changes
• Java module warnings suppressed: Removed explicit XML API references that caused module system warnings on newer JDK versions.

Pipeline Authoring

• produce without closure: The produce statement now accepts a simple file name or pattern without requiring a closure block, simplifying output declarations.
• Multiple inputs with optional: The optional attribute now works correctly with multiple input sources (from ... and ...).
• $region properties: Region objects now expose size, from, to, and chromosomes properties for use in resource allocation expressions.
• byChromosome region splitting: New option to prevent cross-chromosome region grouping when splitting work across parallel branches — ensures all regions on a given chromosome are grouped together.
• sequential:true region splitting: New region splitting mode that keeps split regions adjacent and within a configurable maximum distance, useful for genomics workflows requiring localised grouping.
• Pipeline module logging: Added module loading messages in PipelineContext for better visibility of module dependencies during development.

Development & Debugging

• Stub mode: New interactive "stub" feature in dev mode that allows you to skip execution of stages by providing placeholder (stub) output files. Stubbed inputs are clearly labelled with "(stubbed)" in the output. You can also provide real files as stubs via stub <file> to continue downstream development without re-running expensive upstream stages.
• Tail output on failure: When a command fails, Bpipe now displays the last lines of its output automatically to help diagnose issues quickly.
• Memory and runtime assignment logging: Magic variables for memory allocation are now replaced and runtime assignment values are logged for better transparency.
• Timing and startup optimisation: Added bpipe dev timing and startup tracing (BPIPE_TRACE_TIMES) to help diagnose and reduce pipeline startup latency.
• File system skew allowance: Added tolerance for small file system clock skew when checking if input files were created before the pipeline started, preventing spurious errors on distributed filesystems.
• bpipe test info: The bpipe test command now prints the container or Conda environment in use for each command.

UV Environment Support

• UV for Python: Bpipe now supports UV as a Python package/environment manager alongside Conda, enabling fast Python environment resolution for pipeline tools. UV environment info is displayed in dev mode output.

Infrastructure & Internal

• Tiered compilation: Bpipe now uses JVM tiered compilation to reduce startup delay.
• Improved shutdown handling: Fixed several issues related to ActiveMQ connection failures causing hangs at shutdown, and improved cleanup of pipeline state after errors.

Bug Fixes

• Fixed NPE when a non-existent container is referenced in bpipe.config
• Fixed race condition causing agent hook replies to not be written
• Fixed recursive property references in configuration
• Fixed wrong output selected from produce when output suffix matches multiple candidates
• Fixed regression causing non-optional inputs to be treated as optional
• Fixed bpipe dev failure when non-String content is sent to output
• Fixed pipeline state pollution across segment evaluations
• Fixed spurious errors when pipeline errors before run and JMS agent hook is set
• Fixed broken clone() on JDK 17+
• Fixed badly formatted log messages and excessive environment variable logging
• Fixed missing .netrc file causing ugly stack traces
• Fixed test regressions in quiet_and_clean and client_and_clean tests
• Fixed Agent PID file errors when running generate-dsl in background