TestWorkflowMutableStateImpl - race condition between TimerFired event and CancelTimer command

[tsurdilo]

processCancelTimer throws invalid_argument exception here if timer is null
this timer however would have been already removed if this same workflow task receives a TIMER_FIRED event for this same timer:

https://github.com/temporalio/sdk-java/blob/master/temporal-test-server/src/main/java/io/temporal/internal/testservice/TestWorkflowMutableStateImpl.java#L1437

I think on cancel command, we should only throw if we check first that if timer is null if it was actually removed in same workflow task.

Issue does not allow workflow in test to complete / make progress.

Full error can look like:

[Workflow Executor taskQueue="flakyservice", namespace="default": 1] WARN io.temporal.internal.worker.WorkflowWorker - Failure while reporting workflow progress to the server. If seen continuously the workflow might be stuck. WorkflowId=flaky, RunId=6d9e3f8a-7a73-4aaf-8cdd-02a3cee750f1, startedEventId=22
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: invalid history builder state for action
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondWorkflowTaskCompleted(WorkflowServiceGrpc.java:6079)
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.lambda$sendTaskCompleted$0(WorkflowWorker.java:557)
at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:49)
at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:40)
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.sendTaskCompleted(WorkflowWorker.java:552)
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:409)
at io.temporal.internal.worker.WorkflowWorker$TaskHandlerImpl.handle(WorkflowWorker.java:336)
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:76)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

Still working on a reliable test for this. @Quinn-With-Two-Ns ping me and can point you to slack there where there is a reproduce that you may need to run a number of times to run into this.

[zekkv]

Hitting this from the TypeScript SDK (@temporalio/testing 1.20.3), which downloads and runs this same temporal-test-server binary — passing it a bad arg produces a java.base@23.0.2 stack trace, so it is this implementation.

Posting the trigger in case it is the reliable repro you mentioned still working on.

Condition: a very short timer that fires in the same Workflow Task in which the Workflow cancels and re-arms it — which is exactly the TIMER_FIRED-already-removed-it path you identified.

Our Workflow's main loop waits on condition(predicate, timeout) where the timeout is derived from the earliest of several absolute deadlines, so each iteration cancels the previous timer and starts a new one. When two of those deadlines land within a couple of milliseconds of each other, the loop arms a sub-20 ms timer and cancels it on the very next task. In three captured reproductions the timer immediately preceding the failure was 2 ms, 3 ms, and 13 ms.

Symptoms, in case they are useful for confirming a fix:

• INVALID_ARGUMENT: invalid history builder state for action on RespondWorkflowTaskCompleted, surfaced by Core as Error while completing workflow activation.
• The execution then wedges at WorkflowTaskStarted with no WorkflowTaskCompleted ever written, and never recovers — we extended the wait to 55 s specifically to rule out a workflow-task timeout. Non-retryable, so Core cannot re-drive it.
• Not time skipping running away: we instrumented the server clock and it tracked real time to within ±10 ms throughout.

Reproduction shape (~1 in 8 runs for us, in a real suite rather than a minimal case): drive a Workflow whose main loop re-arms a condition timeout from a set of absolute deadlines, then cause two of those deadlines to be scheduled a few milliseconds apart. In our case two derived-effect deliveries failed back to back, each arming its own retry deadline ~2 ms apart.

Our workaround was to ensure only one such deadline is ever pending, so no sub-20 ms timer is created.