Redesign pipeline command with unified config structure and YAML includes

[tomdurrant]

Summary

• Redesigns the rompy pipeline command to support a cleaner, unified configuration structure
• Adds YAML include functionality with !include directive for composable configurations
• Implements breaking changes to pipeline config format (backward compatibility warnings added)

Key Changes

New Configuration Structure

Pipeline configs now use three top-level keys:

• config: ModelRun configuration (portable, backend-agnostic)
• backend: Optional backend configuration (can be overridden with --backend-config)
• postprocessor: Optional postprocessor configuration (can be overridden with --processor-config)

YAML Include Loader

• Created IncludeLoader class with !include tag support
• Supports relative paths, absolute paths, and environment variable expansion
• Circular dependency detection and maximum depth limit (10 levels)
• Comprehensive test coverage (16 tests)

Breaking Changes

• Pipeline configs must use 'config', 'backend', 'postprocessor' keys (not old names)
• Backend configs must use 'type' instead of 'backend_type'
• --run-backend CLI flag removed (use --backend-config instead)
• LocalPipelineBackend.execute() signature changed from run_backend: str to backend_config: BackendConfig

Backward Compatibility

• Deprecation warnings for old parameter names
• Migration guide included in docs/PIPELINE_MIGRATION.md

Testing

• 16 tests for YAML include loader
• 12 tests for pipeline config structure
• 14 updated tests for pipeline backend
• All 42 tests passing

Files Changed

Created:

• src/rompy/core/yaml_loader.py - YAML include loader implementation
• tests/test_yaml_loader.py - YAML loader tests
• tests/test_pipeline_config.py - Pipeline config integration tests
• docs/PIPELINE_MIGRATION.md - Migration guide

Modified:

• src/rompy/cli.py - Updated load_config() and pipeline() command
• src/rompy/model.py - Updated pipeline() docstring
• src/rompy/pipeline/__init__.py - Updated LocalPipelineBackend.execute() signature
• src/rompy/postprocess/config.py - Added helper function
• examples/configs/basic_pipeline.yml - Updated to new structure
• examples/configs/pipeline_config.yml - 6 comprehensive examples including YAML includes
• tests/backends/test_enhanced_backends.py - Updated 14 tests

Example Configuration

config:        # ModelRun config
  run_id: example
  period: ...
  config: ...

backend:       # Backend config (optional)
  type: local
  timeout: 7200

postprocessor: # Postprocessor config (optional)
  type: ww3_transfer
  destinations: [...]

Design Principles

• Model configs remain portable (backends are optional plugins)
• Backend and postprocessor configs are NOT validated as part of ModelRun schema
• CLI overrides take precedence over inline configurations
• YAML includes support composable, reusable configurations