hotfix: KEEP-1541 reconcile spurious max-retries failures in workflow executor

[joelorzet]

Summary

Workflows are being incorrectly marked as error in workflow_executions even though all step logs show status: success and the SDK run shows status: completed. The error message is always: Step "step//...//stepName" exceeded max retries (X retries).

This affects sendWebhookStep and conditionStep most frequently.

Root Cause

The Workflow DevKit's "use step" durability layer tracks step completion through an internal event-sourcing mechanism. When the SDK's state tracking encounters a conflict (e.g., step already completed, state replay mismatch), it throws "exceeded max retries" even though:

1. The step function itself returned a success result
2. withStepLogging already recorded a status: success log in workflow_execution_logs
3. The actual side effect (webhook call, condition evaluation) completed correctly

The error is caught by executeNode()'s catch block and stored as results[nodeId] = { success: false, error: ... }, which causes finalSuccess to be false -- marking the entire workflow as error.

Fix

After all nodes finish execution but before computing finalSuccess, cross-reference any failed results that contain "exceeded max retries" errors against the actual execution logs in workflow_execution_logs:

• If every log entry for that node has status: success (no error logs at all), the SDK error was spurious -- override the in-memory results to success using the logged output
• If there is any error log for the node, keep the failure -- this prevents masking real failures where a retry legitimately failed

The in-memory results mutation happens before finalSuccess is computed, so triggerStep({ _workflowComplete: { status } }) writes the correct status to DB. No separate DB update is needed.

Architecture: HTTP Loopback Pattern

The workflow bundler rejects Node.js modules (like nanoid pulled in via db/schema), so .workflow.ts files cannot import DB modules -- not even dynamically (the bundler traces import() calls), and functions cannot be passed as parameters (the SDK serializes workflow arguments for durability). To work around this:

1. keeperhub/api/internal/execution-logs/route.ts -- internal API endpoint that queries workflow_execution_logs for a given execution + node IDs
2. keeperhub/lib/fetch-execution-logs.ts -- HTTP loopback helper (same pattern as execution-fallback.ts) that calls the internal endpoint via fetch(). No DB imports, safe for the workflow bundle
3. lib/workflow-executor.workflow.ts -- imports fetchExecutionLogs and calls it after node execution, before finalSuccess computation

This approach:

• Does not change maxRetries -- keeping it at 0 means no duplicate step executions (no double webhook calls)
• Does not modify the catch block -- the error is still caught and recorded normally
• Only reconciles at the end -- no interference with step execution flow
• Is conservative -- only overrides when there is zero evidence of actual failure in the logs
• Does not change WorkflowExecutionInput -- no caller modifications needed

Changes

• lib/workflow-executor.workflow.ts -- reconciles after node execution, before finalSuccess computation
• keeperhub/lib/fetch-execution-logs.ts -- HTTP loopback helper to fetch execution logs without DB imports
• keeperhub/api/internal/execution-logs/route.ts -- internal API endpoint for execution log queries
• app/api/internal/execution-logs/route.ts -- thin wrapper (re-exports from keeperhub)
• keeperhub/lib/max-retries-reconciler.ts -- pure reconciliation logic (used by tests)
• tests/unit/max-retries-reconciler.test.ts -- 8 unit tests covering all reconciliation scenarios

Scenarios Tested

Scenario	Logs in DB	Result	Behavior
Step genuinely fails (HTTP 500)	error log	success: false	Unchanged -- no max-retries error
Step fails + SDK throws max retries	success + error logs	success: false	Kept as failure (error log exists)
Step succeeds + SDK throws spurious max retries	success log only	success: true	Overridden to success
No logs exist for node	none	success: false	Kept as failure (no evidence of success)
Multiple nodes, mixed outcomes	varies per node	per-node	Each node reconciled independently

[github-actions]

🚀 PR Environment Deployed

Your PR environment has been successfully deployed!

Environment Details:

• 🌐 URL: https://app-pr-487.keeperhub.com
• 📦 Namespace: pr-487
• 🏷️ Image Tag: 13c58e1
• 🔢 Commit: 13c58e10f21ae67075658de7205fb0673a537c9a

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Scheduler Dispatcher (staging image)
• Scheduler Executor (staging image)
• SC Event Tracker (staging image)
• SC Event Worker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.

[github-actions]

🚀 PR Environment Deployed

Your PR environment has been successfully deployed!

Environment Details:

• 🌐 URL: https://app-pr-487.keeperhub.com
• 📦 Namespace: pr-487
• 🏷️ Image Tag: 82fb90f
• 🔢 Commit: 82fb90ff49bfed6e02e3e6d8b5f3dff2312e6732

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Scheduler Dispatcher (staging image)
• Scheduler Executor (staging image)
• SC Event Tracker (staging image)
• SC Event Worker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.

[github-actions]

🧹 PR Environment Cleaned Up

The PR environment has been successfully deleted.

Deleted Resources:

• Namespace: pr-487
• All Helm releases (Keeperhub, Scheduler, Event services)
• PostgreSQL Database (including data)
• LocalStack, Redis
• All associated secrets and configs

All resources have been cleaned up and will no longer incur costs.

[github-actions]

🚀 PR Environment Deployed

Your PR environment has been successfully deployed!

Environment Details:

• 🌐 URL: https://app-pr-487.keeperhub.com
• 📦 Namespace: pr-487
• 🏷️ Image Tag: 84a60da
• 🔢 Commit: 84a60dab35b638328b9f9b63688be15a74620f13

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Scheduler Dispatcher (staging image)
• Scheduler Executor (staging image)
• SC Event Tracker (staging image)
• SC Event Worker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.