Add generic TransformStep with pluggable operation system

[plusplusoneplusplus]

Summary

Implements a flexible, generic data transformation primitive for the workflow system with a pluggable operation registry. This addresses the need for map-reduce workflows (multi-model consensus validation) while providing a extensible foundation for various data transformation use cases.

Features

Pluggable Operation System

• BaseOperation abstract class with validation interface
• OperationRegistry for dynamic operation registration
• TransformStep orchestrator (137 lines, clean and focused)
• Easy extensibility - register custom operations without modifying core code

Built-in Operations (8 total)

Comparison Operations:

• compare_results - Compare multiple AI model outputs with similarity scoring
• verify_consensus - Validate if comparison meets threshold criteria

Aggregation Operations:

• aggregate - 7 functions: sum, avg, count, min, max, group_by, concat

Filtering Operations:

• filter - 8 conditions: equals, not_equals, contains, not_contains, greater_than, less_than, regex, custom

Mapping Operations:

• map - 4 functions: extract, project, compute, transform

Architecture

TransformStep (orchestrator)
    ↓
OperationRegistry
    ├─ ComparisonOperation (compare_results, verify_consensus)
    ├─ AggregateOperation (sum, avg, count, min, max, group_by, concat)
    ├─ FilterOperation (equals, contains, greater_than, less_than, regex, etc.)
    └─ MapOperation (extract, project, compute, transform)

Testing

• 48 comprehensive tests - all passing ✅
• test_transform_operations.py - 30 operation-level tests
• test_transform_step.py - 18 integration tests
• Full coverage of validation, execution, and edge cases

Files Changed

New Files:

• plugins/automation/workflows/steps/transform_step.py (137 lines)
• plugins/automation/workflows/steps/operations/ (6 files, ~720 lines)
• plugins/automation/tests/test_transform_operations.py (401 lines)
• plugins/automation/tests/test_transform_step.py (367 lines)
• plugins/automation/workflows/examples/model_comparison_mapreduce.yaml

Modified Files:

• plugins/automation/workflows/README.md (consolidated, 58% reduction)
• plugins/automation/workflows/engine.py (TransformStep registration)
• plugins/automation/workflows/steps/__init__.py (exports)

Stats: 13 files changed, 2103 insertions(+), 766 deletions(-)

Example Usage

Map-Reduce (Multi-Model Validation)

# Map: Execute on different models
- id: model_1
  type: agent
  inputs: {question: "{{ inputs.prompt }}"}

- id: model_2
  type: agent
  inputs: {question: "{{ inputs.prompt }}"}

# Reduce: Compare and verify
- id: compare
  type: transform
  config: {operation: compare_results}
  inputs:
    model_1_result: "{{ steps.model_1.result }}"
    model_2_result: "{{ steps.model_2.result }}"
    threshold: 0.75

Data Pipeline

- id: filter
  type: transform
  config: {operation: filter, condition: equals, field: status, value: "active"}

- id: aggregate
  type: transform
  config: {operation: aggregate, function: avg}

Breaking Changes

None - this is a new feature addition.

Checklist

• All tests passing (48/48)
• Documentation updated
• Example workflow provided
• Code follows project style
• No sensitive information included
• Pre-commit hooks passed

🤖 Generated with Claude Code