[Bug][MASTER] DS kill all yarn apps

[zhangsaisai1995]

Describe the bug
When we restart the DS server, DS kills all the yarn apps.It happens when the task contains commands
To Reproduce

Expected behavior
A clear and concise description of what you expected to happen.

1. Create a DS shell task with the command: yarn application -list
2. Restart the DS server after starting the task
3. Then you will see all the yarn apps will be killed by DS.
   Screenshots

Which version of Dolphin Scheduler:
-[1.3.5]

Additional context

Requirement or improvement
DS will store the task execution log in the temporary log file, which contains the yarn app ids , so that DS service can restart the task after it is restarted.
The point is that DS only performs regular matching on yarn application id, and kill & restart yarn app ids. {@link org.apache.dolphinscheduler.server.utils.ProcessUtils#killYarnJob()},that method will get all the app ids from {@param taskExecutionContext} by {@method taskExecutionContext.getLogPath()}.
Repair suggestion: add salt value to the yarn app ids in the log, and get app ids after matching the salt value in LoggerUtils#getAppIds method when restarting.

[zhangsaisai1995]

@zhuangchong

[zhuangchong]

@zhangsaisai1995
This is a real problem, and the community is talking about a good solution.

[CalvinKirs]

We need to change the storage method of appid, it is best to store it in DB, If at the time of kill (usually when fault tolerance is needed), first call the API to determine the final execution result of the yarn task, whether it will be better.

[ruanwenjun]

Maybe it is better to use YarnClient or SparkLauncher to run the task install of using shell command, then we can easily get the applicationId.

[geosmart]

We need to change the storage method of appid, it is best to store it in DB, If at the time of kill (usually when fault tolerance is needed), first call the API to determine the final execution result of the yarn task, whether it will be better.

when worker shutdown, it's yarn task will failover, and the master get the yarn applicationId by the task log file ,and the task log is come from a logServer witch run in the shutdown worker...
so you just can't failover ,if your worker is down

so we realy need to store the applicationId to do failover.

does we already have any this yarn application failover pr?

[geosmart]

when worker failover happen why we have to kill the yarn application?

the problem is the worker not the yarn application...

if we query the yarn application and found it exist and the state is running, we should make the task running rather than kill it and submit a new one

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.