[Feature][Failover]Add fault tolerance strategy

[fuchanghai]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

• background:

When the master or worker is shut down and started again, the unfinished process instances will be executed from the beginning. but some time i just want that begin from need tolerance

• method

- restart
- start execution from the task that need tolerance

• question

Do I need to set this fault tolerance policy parameter to a service level, or do I need to set it to belong to each process definition? I prefer the second one

cc @ruanwenjun @EricGao888

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[EricGao888]

@fuchanghai
"When the master or worker is shut down and started again, the unfinished process instances will be executed from the beginning."

Is it how things work on dev branch? I remember that in 3.0.X, when dolphin performs failover for a process instance, it will check whether those task instances need to be failed over or not.

[EricGao888]

If as you says, when dolphin performs failover for a process instance, it starts from the very beginning no matter whether some task instances succeeded. It should be a BUG.

[fuchanghai]

hi @EricGao888 when restart server ,process instance will be set RECOVER_TOLERANCE_FAULT_PROCESS

The following is the logic of starting from the failure and restoring fault tolerance. The startNodeList is not set for restoring fault tolerance. Is there anything I haven't noticed? Please help me correct it.