Skip to content

Pre-execution validation is weaker than --validate-only validation #66

@spinje

Description

@spinje

Problem

When running a workflow normally, validation is weaker than when using --validate-only. This causes:

  1. Unknown node types to show ugly tracebacks instead of clean error messages
  2. Validation errors to be caught at compilation time instead of validation time
  3. Inconsistent user experience between validation modes

Steps to Reproduce

Create a workflow with an unknown node type:

{
  "ir_version": "0.1.0",
  "nodes": [
    {"id": "step1", "type": "shell", "params": {"command": "echo hello"}},
    {"id": "step2", "type": "nonexistent-node", "params": {}}
  ],
  "edges": [{"from": "step1", "to": "step2"}]
}

With --validate-only (clean error):

$ pflow --validate-only workflow.json
Validating workflow (static validation)...
✗ Static validation failed:
  • Unknown node type: 'nonexistent-node'

Normal execution (ugly traceback):

$ pflow workflow.json
ERROR: Node instantiation failed
Traceback (most recent call last):
  ...
pflow.runtime.compiler.CompilationError: compiler: Node type 'nonexistent-node' not found
...
❌ Planning failed: compiler: Node type 'nonexistent-node' not found

Root Cause

The CLI has two different validation paths:

Mode Validation Used
--validate-only WorkflowValidator.validate() - full 5-layer validation
Normal execution validate_ir() - schema-only validation

The normal execution path uses _validate_and_handle_workflow_errors() which only calls validate_ir() for schema validation. It does NOT validate:

  • Node types exist in registry
  • Data flow correctness (cycles, forward refs)
  • Template variable resolution
  • Output source validity

Expected Behavior

  • Normal execution should use the same validation as --validate-only
  • Unknown node types should be caught with clean error messages
  • Validation should happen BEFORE any nodes execute
  • Error messages should be consistent across all modes

Proposed Solution

  1. Replace schema-only validation with full WorkflowValidator.validate()
  2. Use real execution params (not dummy) for template validation
  3. Remove dead code (_validate_workflow_structure, _validate_and_handle_workflow_errors)
  4. Add tests verifying validation happens before execution

Impact

  • User Experience: Clean, consistent error messages
  • Safety: Validation catches more issues before execution starts
  • Code Quality: Removes duplicate validation logic (~65 lines)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions