Limit task processing time

[yycptt]

What changed?

• Limit task processing time to 3s
• 500ms for getting workflow execution lock
• For workflow reset task and standby history resend the timeout is 20s and 30s (default) respectively. The timeout for resend is controlled by a dynamicconfig.

Why?

• Host level pool require the processing for each task to be fast

How did you test it?

• Integration tests & canary

Potential risks

• In terms of short context time + retry adding more load to downstream, this should not be a problem as the new retry policy is much less aggressive compared to the previous one. If worker pool sees timeout errors, the task will skip in-place retry and be send back to the rescheduler and the exponential backoff policy will kick in.

Is hotfix candidate?

• No.