[BUG] task is running endlessly

[oceanos]

Describe
When the worker goes down while the task is running, the task will run endlessly.

Reproduce

1. run processA on node1（specify the worker group）
2. shut down node1 (or node1 was crashed) （script or jar in the temporary directory was not generated）
3. restart node1
4. task will run endlessly
5. no button will be enabled on process instance list page after you click one of them

Expected behavior
1, Endless task should be able to fail (or other ops) manually.
2, Or endless task should be able to fail (or other ops) automatically.

Screenshots
Task Is Running Endlessly：

Version of Dolphin Scheduler
-[1.2.0]

Additional context
After restart the master，these endless tasks were all fault-tolerance handled.

Requirement or improvement

• I can't always restart the master if I have these problems. So i think DS should provide some way to get people involved.

[davidzollo]

thanks for your detail description，we will disscuss it tomorrow

[oceanos]

This is a default value, for we want to use in server module only. But as development goes, and api module uses the same config as server. so causes the issue.
We will refactor quartz in the future. but in current version, you can set it as below in api module。
org.quartz.threadPool.threadCount = 1

Your reply has nothing to do with this issue. Do you mean by #2351?

[Technoboy-]

This is a default value, for we want to use in server module only. But as development goes, and api module uses the same config as server. so causes the issue.
We will refactor quartz in the future. but in current version, you can set it as below in api module。
org.quartz.threadPool.threadCount = 1

Your reply has nothing to do with this issue. Do you mean by #2351?

sorry. mail list linked this issue.

[lenboo]

we have worker fault tolerance.
when a worker goes down, the master will change the status of the task to "need fault tolerance" , and wait for other workers to re-execute the task.
obviously this mechanism does not work, but i don't know what happened on this case, maybe require more tests.

[davidzollo]

This is a default value, for we want to use in server module only. But as development goes, and api module uses the same config as server. so causes the issue.
We will refactor quartz in the future. but in current version, you can set it as below in api module。
org.quartz.threadPool.threadCount = 1

Your reply has nothing to do with this issue. Do you mean by #2351?

sorry. mail list linked this issue.

i am sorry, mislead you

[oceanos]

we have worker fault tolerance.
when a worker goes down, the master will change the status of the task to "need fault tolerance" , and wait for other workers to re-execute the task.
obviously this mechanism does not work, but i don't know what happened on this case, maybe require more tests.

Task that has been specified to crashed workers，worker fault tolerance will not work. I think DS should check the status of target worker before executing the task.

[zhongjiajie]

Does this still exists in dev branch @caishunfeng @lenboo

[caishunfeng]

Hi @oceanos the failover logic was updated and you can check it again by the latest version.
This issue will be closed, you can reopen it if need.