[Bug] [Master] Task retry is delayed by the elapsed time instead of being scheduled at (endTime + retryInterval)

[SEPURI-SAI-KRISHNA]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When a task fails and still has retry attempts left, the master schedules a delayed
TaskRetryLifecycleEvent. The delay is computed in
TaskRetryLifecycleEvent#of:

final long remainingTime =
        TimeUnit.MINUTES.toMillis(delayTime) + System.currentTimeMillis() - taskInstance.getEndTime().getTime();

The intent is "retry the task at endTime + retryInterval", so the remaining delay should be

retryInterval - (now - endTime)

but the expression actually evaluates to

retryInterval + (now - endTime)

The elapsed time since the task ended is added to the delay instead of being
subtracted from it. The signs of now and endTime are swapped.

What you expected to happen

The retry should fire retryInterval after the task ended. If the retry interval has
already elapsed by the time the event is created, the task should be retried
immediately.

Impact

In the common path (onFailedEvent fires milliseconds after the task ended) the error is
only a few milliseconds and is invisible.

It becomes significant whenever the retry event is created long after the task
actually ended, because the delay then grows without bound:

• Master failover. WorkflowFailoverCommandHandler#assembleWorkflowExecutionGraph
  rebuilds the execution graph from the existing task instances, so a task instance
  left in FAILURE with retries remaining is preserved as-is. When the workflow is
  re-triggered, TaskFailureStateAction#onStartEvent republishes a
  TaskFailedLifecycleEvent carrying the stale endTime read from the database.
  TaskRetryLifecycleEvent#of then computes retryInterval + (now - endTime), so the
  retry is postponed by the whole outage duration on top of the configured interval.
  A task that failed an hour before the failover, with a 1-minute retry interval, waits
  ~61 minutes instead of being retried immediately.
• Any other path where the failure event is processed after a delay (event backlog,
  workflow paused/resumed) is skewed the same way.

The workflow simply looks stuck in RUNNING_EXECUTION while the retry sits in the delay
queue.

How to reproduce

1. Define a workflow with one task that fails, with failRetryTimes >= 1 and
   failRetryInterval = 1 (minute).
2. Run it and let the task fail so it enters FAILURE while waiting to retry.
3. Kill the master that owns the workflow instance before the retry fires.
4. Wait a few minutes, then let another master pick the instance up by failover.
5. The retry is scheduled 1 minute + <outage duration> in the future instead of firing
   right away.

A deterministic unit-level reproduction: call TaskRetryLifecycleEvent.of(taskExecution)
with a task instance whose endTime is two hours in the past and whose retryInterval
is 5 minutes. Expected delay: 0. Actual delay: ~2h05m.

Anything else

The same expression is the only place the retry delay is computed, so the fix is
confined to TaskRetryLifecycleEvent#of.

Version

dev

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct