[Bug] [Master] When database restart or close, the workflow instance status will be incorrect.

[ruanwenjun]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I find when there is workflowInstance running, and the database restart. The workflow instance's status will be incorrect, it will be running all the time. This is due to when we receive the worker's response, we will update the taskInstance's meta, and update the database, but when we update database failed, we will not rollback the meta in memory, so the worker send response again, the master will skip the state, due to it think the state has already been handled.

What you expected to happen

The master can recover, when database restart.

How to reproduce

1. Running 1000 workflowIntsance
2. Restart database
3. Find some workflowInstance will always be running

Anything else

No response

Version

dev

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting