Skip to content

LLM node swallows exception details; trace shows incorrect success status #6

@spinje

Description

@spinje

Summary

When an LLM node fails in a pflow workflow, the error message displayed to users omits the actual exception details, making debugging impossible. Additionally, the workflow trace file incorrectly reports success: true for failed nodes and final_status: "success" for failed workflows.

Issues

Issue 1: LLM Error Messages Swallow Exception Details

Severity: High - Blocks debugging

When an LLM call fails for reasons other than UnknownModelError or NeedsKeyException, the original exception message is discarded and replaced with a generic message.

Steps to Reproduce:

cat > /tmp/test-llm-fail.json << 'EOF2'
{
  "inputs": {},
  "nodes": [{"id": "test-llm", "type": "llm", "params": {"prompt": "Say hello", "model": "nonexistent-model"}}],
  "edges": []
}
EOF2
pflow /tmp/test-llm-fail.json

Actual behavior:

Message: LLM call failed after 3 attempts. Model: nonexistent-model

Expected behavior:

Message: LLM call failed after 3 attempts. Model: nonexistent-model. Error: UnknownModelError...

Root Cause: In src/pflow/nodes/llm/llm.py, the exec_fallback method captures error_msg = str(exc) but doesn't include it in the generic error case.

Issue 2: Trace File Reports Incorrect Success Status

Severity: Medium - Misleading debug information

The workflow trace file reports success: true even when the workflow fails via error action.

Actual trace output:

{
  "final_status": "success",
  "nodes": [{"node_id": "test-llm", "success": true}]
}

Expected trace output:

{
  "final_status": "failed",
  "nodes": [{"node_id": "test-llm", "success": false}]
}

Root Cause: In src/pflow/runtime/instrumented_wrapper.py, trace is recorded with success=True before checking if the node returned "error" action. The caching logic already checks result != "error" but trace recording doesn't.

Impact

  1. AI agents cannot debug workflow failures - No way to determine why LLM calls fail
  2. Trace files are unreliable - Cannot be trusted to indicate actual success/failure status

Environment

  • pflow version: current main
  • Platform: macOS/Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions