v1.7.0

Bacalhau v1.7.0 Release Notes

This release introduces enterprise support options, enhanced partitioning capabilities, improved networking, comprehensive authentication, and significant WebAssembly enhancements.

Enterprise Support through Expanso

• Enterprise support now available through Expanso while maintaining Bacalhau's open source core
• Free tier for everyone supporting up to 5 nodes with all core features
• Enterprise licensing option for organizations requiring professional support

Expanso Cloud Enhancements

• Comprehensive job submission capabilities through the Expanso Cloud web interface
• Library of pre-defined job templates for common use cases:
  • Log processing workflows
  • Data analysis with DuckDB
  • Apache Iceberg operations
  • And more
• Custom template creation capabilities with parameterization
• Simplified complex distributed computing tasks while maintaining flexibility

Enhanced Job Execution with Partitioned Workloads

• Distribution of work across compute nodes with built-in partition management
• Horizontal scaling and parallel processing for improved performance
• Granular failure recovery with automatic retry mechanisms for failed partitions
• Independent partition execution with context through environment variables

S3 Storage Input Partitioning

• Automatic data distribution across multiple executions with graceful failure handling
• Multiple partitioning strategies:
  • Object-Based Distribution: Even distribution without specific grouping
  • Date-Based Partitioning: Parallel processing for time-series analysis
  • Regex Pattern Matching: Distribution by patterns in file paths
  • Substring-Based Partitioning: Organization by customer segments
• Combined partitioned data with shared reference data support

S3 Publishing Enhancements

• Plain encoding support for direct access to job outputs without compression
• Individual file publishing rather than compressed archives
• Efficient access to individual files with no decompression overhead
• Default gzip encoding still available for complete result sets

Enhanced Networking Capabilities

• Networking enabled by default (shifted from opt-in to opt-out)
• Configuration option for restricting network access
• Experimental port mapping support
• Multiple network modes: bridge (default), host, and none

Comprehensive Authentication and Authorization

• Redesigned authentication system with multiple methods:
  • Basic authentication
  • API tokens
  • Single Sign-On (SSO)
• Fine-grained authorization controls
• New CLI commands for authentication management

Host Environment Variable Forwarding

• Access to specified environment variables from the host system
• Secure credential passing through a controlled allowlist mechanism
• Reference host variables using env: prefix in job specifications

Improved WebAssembly Support

• Refactored WASM job specification with better separation of concerns
• Remote modules defined as input sources for alignment with other executors
• HTTP networking support for WebAssembly through new host function implementation
• Configurable network access modes and security controls
• TinyGo-compatible client library for HTTP functionality

Docker DinD Integration

• Docker-in-Docker images for simplified containerized deployment
• Two image variants:
  • Standard base image for client usage and orchestrator nodes
  • DinD image for compute nodes running Docker workloads
• Pre-configured Docker daemon for simple integration with container ecosystems

Performance and Quality of Life Improvements

• Enhanced resource management and more efficient state tracking
• count field now defaults to 1 instead of 0
• NetworkDefault type for automatic networking configuration

Documentation & Resources

Explore the complete release details including FAQs, implementation guides and enterprise licensing information in our comprehensive v1.7.0 documentation. The full documentation includes detailed answers to common questions about partitioning strategies, authentication methods, S3 integration, and enterprise support options.

New Contributors

• @virajbhartiya made their first contribution in #4848
• @haixuanTao made their first contribution in #4857
• @Olg-K made their first contribution in #4873

Full Changelog: v1.6.0...v1.7.0

v1.7.0-rc2

What's Changed

• Make Authentication Bypass test more robust by @jamlo in #4901
• bacalhau auth info by @jamlo in #4902
• Redact sensitive info from agent config command by @jamlo in #4903
• Modify auth sso login syntax and add logout command by @jamlo in #4904
• Add "bacalhau auth sso token" command by @jamlo in #4905
• Fix SSO Login regression by @jamlo in #4907
• fix regression in not setting log level by @wdbaruni in #4910
• Avoid setting default network config when nil by @wdbaruni in #4911

Full Changelog: v1.7.0-rc1...v1.7.0-rc2

v1.7.0-rc1

Bacalhau v1.7.0 Release Notes

This release introduces enterprise support options, enhanced partitioning capabilities, improved networking, comprehensive authentication, and significant WebAssembly enhancements.

Enterprise Support through Expanso

• Enterprise support now available through Expanso while maintaining Bacalhau's open source core
• Free tier for everyone supporting up to 5 nodes with all core features
• Enterprise licensing option for organizations requiring professional support

Expanso Cloud Enhancements

• Comprehensive job submission capabilities through the Expanso Cloud web interface
• Library of pre-defined job templates for common use cases:
  • Log processing workflows
  • Data analysis with DuckDB
  • Apache Iceberg operations
  • And more
• Custom template creation capabilities with parameterization
• Simplified complex distributed computing tasks while maintaining flexibility

Enhanced Job Execution with Partitioned Workloads

• Distribution of work across compute nodes with built-in partition management
• Horizontal scaling and parallel processing for improved performance
• Granular failure recovery with automatic retry mechanisms for failed partitions
• Independent partition execution with context through environment variables

S3 Storage Input Partitioning

• Automatic data distribution across multiple executions with graceful failure handling
• Multiple partitioning strategies:
  • Object-Based Distribution: Even distribution without specific grouping
  • Date-Based Partitioning: Parallel processing for time-series analysis
  • Regex Pattern Matching: Distribution by patterns in file paths
  • Substring-Based Partitioning: Organization by customer segments
• Combined partitioned data with shared reference data support

S3 Publishing Enhancements

• Plain encoding support for direct access to job outputs without compression
• Individual file publishing rather than compressed archives
• Efficient access to individual files with no decompression overhead
• Default gzip encoding still available for complete result sets

Enhanced Networking Capabilities

• Networking enabled by default (shifted from opt-in to opt-out)
• Configuration option for restricting network access
• Experimental port mapping support
• Multiple network modes: bridge (default), host, and none

Comprehensive Authentication and Authorization

• Redesigned authentication system with multiple methods:
  • Basic authentication
  • API tokens
  • Single Sign-On (SSO)
• Fine-grained authorization controls
• New CLI commands for authentication management

Host Environment Variable Forwarding

• Access to specified environment variables from the host system
• Secure credential passing through a controlled allowlist mechanism
• Reference host variables using env: prefix in job specifications

Improved WebAssembly Support

• Refactored WASM job specification with better separation of concerns
• Remote modules defined as input sources for alignment with other executors
• HTTP networking support for WebAssembly through new host function implementation
• Configurable network access modes and security controls
• TinyGo-compatible client library for HTTP functionality

Docker DinD Integration

• Docker-in-Docker images for simplified containerized deployment
• Two image variants:
  • Standard base image for client usage and orchestrator nodes
  • DinD image for compute nodes running Docker workloads
• Pre-configured Docker daemon for simple integration with container ecosystems

Performance and Quality of Life Improvements

• Enhanced resource management and more efficient state tracking
• count field now defaults to 1 instead of 0
• NetworkDefault type for automatic networking configuration

New Contributors

• @virajbhartiya made their first contribution in #4848
• @haixuanTao made their first contribution in #4857
• @Olg-K made their first contribution in #4873

Full Changelog: v1.6.0...v1.7.0-rc1

v1.6.5

Feature Enhancements and Fixes

Important Networking Changes

• Networking Opt-Out Configuration: Added new JobAdmissionControl.RejectNetworkedJobs configuration to allow users to explicitly opt-out of networking. Currently, Bacalhau rejects networking by default and requires users to opt-in using JobAdmissionControl.AcceptNetworkedJobs=true. This new parameter is the first phase of a major change where networking will be enabled by default in v1.7 (planned for release in the coming weeks). Users who need to maintain disabled networking should begin transitioning to the new RejectNetworkedJobs=true configuration now to ensure a smooth upgrade path.

Job Configuration Enhancements

• Default Count Parameter: Fixed a significant usability issue where batch/service jobs would silently fail when no count was specified. The system now automatically sets the count field to 1 when not explicitly provided, eliminating confusion for new users. Explicit count=0 settings are still respected for intentionally stopping jobs.

Docker Image Handling

• Platform Compatibility Verification: Fixed the platform compatibility check for locally cached Docker images, ensuring that jobs correctly identify and utilize compatible images across different environments.

New Contributors

• @haixuanTao made their first contribution in #4857

Full Changelog: [v1.6.4...v1.6.5](v1.6.4...v1.6.5)

v1.6.4

S3 Publisher Enhancements

• Plain Encoding Support: Introduced an option to publish job results without gzip compression, facilitating more efficient data pipelines by allowing subsequent jobs to access individual files directly.

Environment Variable Management

• Host Environment Variable Forwarding: Added the ability to securely forward host environment variables to job executions. This feature enables passing credentials and secrets from the host to jobs through a controlled allowlist mechanism.

Docker Image Improvements

• CA Certificates Installation: Ensured that CA certificates are installed in our Docker images, enhancing security and enabling secure HTTPS communications within containerized environments.

These updates aim to enhance the functionality, security, and flexibility of Bacalhau.

Full Changelog: v1.6.3...v1.6.4

v1.6.3

Docker Image Enhancements

• New Docker-in-Docker (DinD) Image: Introduced a new Docker image variant specifically designed for compute nodes:

  • Added bacalhau:v1.6.3-dind and bacalhau:latest-dind images
  • Full Docker execution capability for compute nodes
  • Built-in Docker daemon support for container workloads
  • Requires privileged mode for Docker-in-Docker functionality

• Image Variants Clarification:

  • Base image (bacalhau:v1.6.3, bacalhau:latest): Optimized for client usage, orchestrator nodes, and non-Docker compute workloads
  • DinD image (bacalhau:v1.6.3-dind, bacalhau:latest-dind): Designed for compute nodes requiring Docker execution capabilities

Operational Improvements

• Docker Deployment:
  • Clearer separation of concerns between different node types
  • Improved documentation for Docker image usage

Full Changelog: v1.6.2...v1.6.3

v1.6.2

Performance and Stability Improvements

• Enhanced Network Performance: Implemented batching and rate limiting of executions per scheduling cycle, ensuring all executions are processed while maintaining system stability when orchestrating jobs across hundreds of nodes
• Improved Message Handling: Extended publish and subscribe timeout windows for more graceful network communication
• Critical Bug Fixes:
  • Resolved boltdb transaction panic issues during context cancellation
  • Fixed execution re-approval logic in the scheduler
  • Enhanced transaction handling for better stability

Observability Enhancements

• Comprehensive Metrics Coverage: Introduced OpenTelemetry metrics across multiple core components:
  • Job store metrics for storage monitoring
  • Scheduler metrics for job distribution insights
  • Planner metrics for execution planning visibility
  • NCL (Network Communication Layer) metrics
  • Message handler metrics for network communication monitoring

Operational Improvements

• Development Environment:
  • Enhanced devstack logging format
  • Added support for node joining existing networks
  • Improved configuration passing mechanisms

Full Changelog: v1.6.1...v1.6.2

v1.6.1

Major Improvements

• Partitioned Execution Support:
  Added support for splitting jobs across multiple executions with automatic partition management. The feature includes:

  • Partition assignment and tracking
  • Independent execution progress monitoring
  • Granular failure handling with retry of only failed partitions
  • Each execution receives its partition details through environment variables, enabling partition-aware processing when needed

• S3 Input Partitioning:
  Added automatic data distribution for S3 inputs across multiple executions using configurable strategies:

  • Multiple partitioning strategies: Users can choose between object-based distribution for even splitting, regex patterns for structured data, substring matching for fixed formats, or date-based partitioning for temporal data
  • Even distribution of data without requiring custom partition code
  • Support for shared data access through non-partitioned inputs
  • Automatic data subset assignment to each execution

Full Changelog: v1.6.0...v1.6.1

v1.6.0

Bacalhau v1.6.0 Release Notes

We are excited to announce the release of Bacalhau v1.6.0, introducing a new communication architecture that significantly improves the reliability and resilience of distributed compute networks.

Key Features and Improvements

New Bacalhau Messaging Protocol (BMP)

At the heart of this release is the new messaging protocol, a complete redesign of node communication that brings significant improvements to network reliability:

Key Benefits

• Self-Healing Network: Compute nodes and orchestrators automatically reconnect and sync after network interruptions
• Offline-First Operation: Compute nodes can start and operate even when disconnected from the orchestrator
• Automatic State Recovery: When nodes reconnect, they automatically share all missed job execution information and results
• Zero Data Loss: Ensures no job execution data or results are lost during network disruptions
• Seamless Recovery: Network interruptions are handled transparently without requiring manual intervention

Technical Improvements

• Reliable Message Delivery: Ordered, at-least-once message delivery between nodes
• Automatic Recovery: Built-in failure detection and recovery mechanisms
• Connection Health Monitoring: Proactive health checks and connection management
• Event-Based Architecture: Decoupled event processing from message delivery
• Efficient Checkpointing: Maintains system state for reliable recovery
• Backward Compatibility: Maintains compatibility with v1.5 orchestrators

Enhanced Web UI Experience

• Direct Result Downloads: Download job results directly from the interface
• Simplified Configuration: Automatic request routing eliminates manual IP configuration
• Improved Architecture: Streamlined backend setup while maintaining security

Operational Improvements

• Reverse Proxy Support: Added capability to run orchestrator behind a reverse proxy
• Agent Configuration: New bacalhau agent config command to inspect agent configuration
• TLS Support: Added TLS encryption support for NATS communication
• Better Logging: Implemented more human-readable logging patterns

Upgrade Notes and Backward Compatibility

Bacalhau v1.6.0 maintains backward compatibility while introducing the new BMP:

• Compute nodes maintain compatibility with v1.5 orchestrators, and vice versa
• Support for re-handshake from legacy clients

We're excited for you to experience the enhanced reliability and resilience provided by the BMP in Bacalhau v1.6.0. This release represents a significant architectural advancement in making distributed computing more robust and dependable.

v1.5.1

Major Improvements

• Enhanced Web UI Routing: Improved routing of Web UI requests without requiring backend address definition
• Faster Startup: Dramatically reduced node startup time from ~9 seconds to ~1.5 seconds by optimizing IMDS access
• Job Management: Added support for stopping jobs using short IDs
• Bug Fix: Resolved issues with default publishers functionality

Breaking Changes

• Removed exec command and job translation functionality

Additional Changes

• Added Docker compose support
• Improved API error handling

Links

• Full Changelog: v1.5.0...v1.5.1