fix: Preserve workflow task failure errors through illegal call validator rescues

[AnushkaKulkarni]

What was changed

The illegal call tracer's validator rescue blocks (IllegalCallTracer#initialize, both branches) now re-raise exceptions marked with Internal::WorkflowTaskFailureError instead of flattening them into the ", reason: #{e}" string of a Workflow::NondeterminismError.

Why?

The deadlock detector raises DeadlockError asynchronously into the workflow thread (Timeout.timeout in thread_pool.rb). If that raise lands while an IllegalWorkflowCallValidator block is executing — in our production incidents, the Thread::Mutex#synchronize validator hit via protobuf's ObjectCache during ScheduleActivity arg serialization — the tracer's rescue Exception converts it into a NondeterminismError string. The WorkflowTaskFailureError marker from #467 is destroyed, so Future#initialize stores the error instead of failing the workflow task, and the activation completes with a partial command batch (a consumed-but-unscheduled activity id). Replay then fails permanently with [TMPRL1100] Activity id of scheduled event 'N' does not match activity id of activity command 'N-1'.

With this change the marker class survives the validator path, the existing #467 machinery re-raises it out of the future, and the workflow task fails and retries — the same contract as when the deadlock lands outside a traced call.

Checklist

1. Closes [Bug] Illegal call tracer converts DeadlockError to NondeterminismError, bypassing #467 and committing partial command batches #503

2. How was this tested:

Added test_deadlock_during_illegal_call_validator_fails_workflow_task to deadlock_test.rb: a custom validator raises DeadlockError (standing in for the async watchdog raise landing mid-validator) inside a Workflow::Future that is never waited. Pre-fix, the error is silently stored and no workflow task ever fails; post-fix, the task fails with the deadlock message (and the test asserts the failure is not the laundered "Cannot access ..." form). I could not run the suite locally (no Rust toolchain for the bridge), so relying on CI here; the equivalent behavior change has been validated in production via a monkeypatch on 1.5.0.

3. Any docs updates needed?

N/A

Made with Cursor

[CLAassistant]

All committers have signed the CLA.