Adds a retryable error for when we try to delete open executions #3405
Conversation
|
what is the case we will delete a open execution from transfer task? |
service/history/queues/executable.go
Outdated
| if err == consts.ErrTaskRetry || err == consts.ErrDeleteOpenExec { | ||
| e.taggedMetricsHandler.Counter(TaskStandbyRetryCounter).Record(1) |
There was a problem hiding this comment.
We don't want the TaskStandbyRetryCounter metric get emitted when encountering the task not completed error. Probably define a new counter metric for this error.
There was a problem hiding this comment.
Added a new counter metric.
MichaelSnowden
force-pushed
the
err-delete-open-exec
branch
from
7b14ff7
to
31eaaa1
Compare
It looks like there are a lot of places workflow close tasks are generated, like when it fails, times out, or completes. I think the race condition is when one of those tasks is generated, and the user simultaneously calls DeleteWorkflowExecution and the delete task is made too, so this ensures that the close task finishes before the delete task, if there is any pending close task. However, I'm not really sure--I'm just copying the logic from delete_manager, which had this check. |
MichaelSnowden
force-pushed
the
err-delete-open-exec
branch
from
31eaaa1
to
f30bc8d
Compare
…lose task is incomplete
MichaelSnowden
force-pushed
the
err-delete-open-exec
branch
from
f30bc8d
to
8ba82ce
Compare
Adds a retryable error for when we try to delete open executions. This was requested by @yycptt since we want a longer retry delay for this type of error.