Skip to content

v2.6.0

Latest
Latest

Choose a tag to compare

View all tags
@hansenmc hansenmc released this 20 Apr 23:29
· 5 commits to development since this release
v2.6.0
93e39a8

CoRB 2.6.0 Release Notes

Overview

CoRB 2.6.0 includes significant new features for job management, enhanced export capabilities, improved security, and a modernized web interface. This release adds support for restartable jobs, JSON file processing, export file splitting, and an interactive options builder UI.

Major New Features

Restartable Jobs (#65)

CoRB jobs can now be restarted from the point of interruption, avoiding reprocessing of already-completed URIs.

  • New Option: RESTARTABLE - Enable restart capability by persisting job state locally
  • New Option: RESTART-STATE-DIR - Configure where restart state files are stored (defaults to TEMP-DIR)
  • Provides best-effort at-least-once restart semantics
  • Restart files are automatically cleaned up after successful job completion
  • URIs that failed processing are marked as processed and skipped on restart

Usage:

RESTARTABLE=true
RESTART-STATE-DIR=/path/to/restart/state

JSON File Loaders (#96)

New loader classes enable processing JSON files similar to existing XML file loaders.

  • New Classes:
    • FileUrisJSONLoader - DOM-based JSON file processing
    • FileUrisStreamingJSONLoader - Memory-efficient streaming JSON processing
  • New Options:
    • JSON-FILE - Path to JSON file for URI loading
    • JSON-NODE - XPath expression to select JSON values (default: /*)
    • JSON-METADATA - XPath to metadata that will be passed to modules
    • JSON-TEMP-DIR - Temporary directory for streaming loader
    • METADATA-TO-PROCESS-MODULE - Pass metadata to process module
    • LOADER-BASE64-ENCODE - Base64 encode file content in envelope

Example:

URIS-LOADER=com.marklogic.developer.corb.FileUrisStreamingJSONLoader
JSON-FILE=/path/to/data.json
JSON-NODE=/items/*
LOADER-USE-ENVELOPE=true

Export File Splitting (#287)

Export files can now be automatically split into multiple files based on size or line count thresholds.

  • New Options:
    • EXPORT-FILE-SPLIT-MAX-LINES - Maximum lines per file before splitting
    • EXPORT-FILE-SPLIT-MAX-SIZE - Maximum file size before splitting (supports units: B, KB, MB, GB, TB)
    • EXPORT-FILE-AS-ZIP - Compress output files as ZIP
  • Split files use numeric suffixes (e.g., export.txt, 001_export.txt, 002_export.txt)
  • Automatic configuration of PreBatchUpdateFileTask and PostBatchUpdateFileTask when needed
  • Supports EXPORT-FILE-TOP-CONTENT and EXPORT-FILE-BOTTOM-CONTENT with splitting

Example:

EXPORT-FILE-NAME=report.csv
EXPORT-FILE-SPLIT-MAX-SIZE=100MB
EXPORT-FILE-AS-ZIP=true

Options Builder Web UI

New interactive web interface for building CoRB job configurations.

  • Build job configurations through a guided web form
  • Download generated .properties file
  • Launch jobs directly from the browser
  • Organized by functional groups: Connection, Security, Input, Processing, Execution, Export, Monitoring, Retry, Advanced
  • Search and filter options
  • Field validation and help text
  • Support for custom/additional properties

Access: Navigate to http://localhost:<port> on whatever port or range you specified when the Job Server is running and select "Options Builder" tab.

Enhanced Jasypt Support

Upgraded Jasypt decrypter with support for initialization vectors and configurable encrypters.

  • New Options:
    • JASYPT-PROPERTIES-FILE - Path to Jasypt properties file (default: jasypt.properties)
    • JASYPT-STRING-ENCRYPTER - Custom encrypter class (default: StandardPBEStringEncryptor)
    • JASYPT-IV-GENERATOR - IV generator class (requires Jasypt 1.9.3+)
  • Support for modern encryption algorithms with IV generators
  • Examples: RandomIvGenerator, StringFixedIvGenerator

Example:

DECRYPTER=com.marklogic.developer.corb.JasyptDecrypter
JASYPT-IV-GENERATOR=org.jasypt.iv.RandomIvGenerator
JASYPT-STRING-ENCRYPTER=org.jasypt.encryption.pbe.StandardPBEStringEncryptor

Configurable Host Key Decrypter Cipher

HostKeyDecrypter now supports custom cipher configuration.

  • New Option: HOST-KEY-DECRYPTER-CIPHER - Specify cipher transformation (default: AES)
  • Supports simple algorithm names (e.g., AES) or full transformation strings (e.g., AES/GCM/NoPadding)
  • Enables use of modern encryption modes like GCM

Example:

DECRYPTER=com.marklogic.developer.corb.HostKeyDecrypter
HOST-KEY-DECRYPTER-CIPHER=AES/GCM/NoPadding

Enhancements

Web UI Modernization

Complete rewrite of the Job Server dashboard using modern web technologies:

  • Replaced AngularJS + Bootstrap + jQuery with AlpineJS (lightweight 15KB framework)

Test Framework Improvements

  • Upgraded to JUnit 5.14.3
  • Upgraded to Mockito 4.11.0
  • Refactored test architecture to avoid System.exit() calls
  • Improved test isolation and maintainability
  • Added comprehensive test coverage for new features

Security Improvements

This release includes extensive security hardening based on security scan findings:

XML External Entity (XXE) Protection

  • Disabled DTD and external entity resolution in XML parsers
  • Applied secure XML processing configurations across all XML/XPath operations
  • Protection against XXE injection attacks

Resource Management

  • Ensured proper closing of streams and resources using try-with-resources
  • Fixed potential resource leaks
  • Thread-safe mutations and synchronization improvements

Secure Defaults

  • Standardized character encoding to UTF-8 across the application
  • Enhanced null safety checks throughout codebase
  • Protected finalize() methods to prevent exploitation via constructor exceptions
  • Serialization security improvements

Input Validation

  • Enhanced parameter validation in Job Server endpoints
  • Path traversal attack prevention
  • Restricted file access patterns

Dependency Updates

Runtime Dependencies

  • MarkLogic XCC: 11.3.1 → 12.0.0
  • AlpineJS: 3.15.8 (new dependency, replaces AngularJS/Bootstrap/jQuery)
  • Jasypt: Examples updated to 1.9.3

Build Dependencies

  • Gradle: 8.14.4
  • JUnit: 5.14.3
  • Mockito: 4.11.0
  • JetBrains Annotations: 26.1.0
  • External Sorting in Java: Updated to latest version

Bug Fixes and Improvements

Code Quality

  • Comprehensive Javadoc documentation added across all classes
  • Better handling of edge cases and null values

Task Auto-Configuration

Automatic task configuration for common patterns:

  • Auto-configure PreBatchUpdateFileTask when EXPORT-FILE-TOP-CONTENT is specified with ExportBatchToFileTask
  • Auto-configure PostBatchUpdateFileTask when using:
    • EXPORT-FILE-BOTTOM-CONTENT
    • EXPORT-FILE-SPLIT-MAX-LINES
    • EXPORT-FILE-SPLIT-MAX-SIZE
    • EXPORT-FILE-AS-ZIP
    • EXPORT-FILE-SORT with ExportBatchToFileTask

Job Server Improvements

  • Better handling of job lifecycle and context management
  • Improved metrics collection and reporting
  • Enhanced error handling and user feedback
  • Support for running jobs directly from the web UI

Platform Compatibility

  • Improved Windows compatibility for file handling and tests
  • Better UTF-8 character encoding support
  • Cross-platform path handling improvements

Breaking Changes

None. This release maintains backward compatibility with 2.5.x configurations.

Note: The EXPORT-FILE-AS-ZIP option now uses dash-separated naming for consistency. The underscore-separated variant is deprecated but still supported for backward compatibility.

Assets 8
Loading