[Bug] [Module Name] Server-side exception: task instance is null or host is null

[touchhuang]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Task displays submission success by master, but not excute by worker. how to fix it?

What you expected to happen

task need done well

How to reproduce

Anything else

sometimes

Version

3.1.x

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[AmriStrong]

Please check if worker Group management is configured correctly?
security -> worker group manage

[touchhuang]

There are no errors in the configuration. but it work well after stop some error instance, I found a large number of errors msg with another workGroup in master-log when this error occurred. Does this have an impact?

[kezhenyang163]

I also encountered a similar problem. When I manually executed the task, this error occurred. After restarting the cluster, it returned to normal. My configuration is a single machine work node.

It seems that the task has not been assigned to the corresponding execution node for execution, and the task has been in the pending state.

[yihao-tcf]

I have also experienced such problems, which seriously affect the normal operation of scheduled tasks.
version：3.1.7

Reproduction steps

1. Create Work groups
   

2. Create Test Tasks(Three different working groups)
   Task 1 use '大数据集群' worker group

Task 2 use '算法集群' worker group

Task 3 use 'default' worker group

Set scheduled execution，online workflow and scheduling
* * * * * ? *
3. Stop the worker of the '算法集群'

After stopping, all tasks will be affected and an error will be reported as task instance is null or host is null.

After stopping woker group info:

After stopping task execution status:

error logs message：

After starting the stopped worker, the task resumes execution. A large number of backlogged tasks are resuming execution, which may cause the cluster to be in a high load state and lead to the downtime of other services in the cluster. This is very dangerous

I don't quite understand why the node downtime of the '算法集群' worker group affects the tasks of other worker group nodes

[kezhenyang163]

Log in to the DS machine and run [status-all.sh] to check the health status of DS cluster nodes，When I have this error, it is usually accompanied by a node exception in the cluster，You can prioritize whether it is a machine node problem

After I restarted the cluster, the machine node status returned to normal and the task could be executed normally. However, the cause of DS node death still needs to be further identified according to the specific situation

My problem was that the master and work nodes died abnormally due to insufficient machine resources
So it's not a Bug

[kezhenyang163]

@touchhuang This error is caused by the abnormal work node, so I think this issue can be closed

[yihao-tcf]

@touchhuang This error is caused by the abnormal work node, so I think this issue can be closed

I don't think this is a bug, if one of the worker nodes goes down for some reason, causing all tasks to fail to run. Is this a bug?

[yihao-tcf]

I have also experienced such problems, which seriously affect the normal operation of scheduled tasks. version：3.1.7

Reproduction steps

1. Create Work groups
   
2. Create Test Tasks(Three different working groups)
   Task 1 use '大数据集群' worker group

Task 2 use '算法集群' worker group

Task 3 use 'default' worker group

Set scheduled execution，online workflow and scheduling 3. Stop the worker of the '算法集群'* * * * * ? *

After stopping, all tasks will be affected and an error will be reported as task instance is null or host is null.

After stopping woker group info:

After stopping task execution status:

error logs message：

After starting the stopped worker, the task resumes execution. A large number of backlogged tasks are resuming execution, which may cause the cluster to be in a high load state and lead to the downtime of other services in the cluster. This is very dangerous

I don't quite understand why the node downtime of the '算法集群' worker group affects the tasks of other worker group nodes

@kezhenyang163
From the steps of replication, it can be seen that when the nodes corresponding to my "算法集群" workgroup were closed, the tasks of the "大数据集群" and "default" workgroups were also affected and could not work properly. I think this is very unreasonable, as one of the executor nodes goes down, causing all nodes in the current system to be affected. Do you think it's reasonable?

[yihao-tcf]

@kezhenyang163
I believe that if the node corresponding to the "算法集群" workgroup goes down, it should affect the tasks of its corresponding workgroup. And it will not affect the working groups of other normal worker nodes. If a single node goes down and all tasks cannot run, then I think it will deviate from the "High Performance Workflow" described by Dolphinscheduler