[Bug] [MasterServer] RefreshResourceTask error by NPE

[caishunfeng]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When master refresh the worker node, it may occur NPE when hearbeat is null.

[ERROR] 2021-10-29 20:21:45.977 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[152] - [] RefreshResourceTask error
java.lang.NullPointerException: null
        at org.apache.dolphinscheduler.common.utils.HeartBeat.decodeHeartBeat(HeartBeat.java:228)
        at org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager$RefreshResourceTask.getHostWeight(LowerWeightHostManager.java:157)
        at org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager$RefreshResourceTask.run(LowerWeightHostManager.java:141)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

What you expected to happen

No NPE Excetion.

How to reproduce

Restart the worker server.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Hi:

• Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.

[Yao-MR-zz]

hi @caishunfeng can you share more or all the master log ?

[caishunfeng]

hi @caishunfeng can you share more or all the master log ?

@Yao-MR That's the complete log I show. Can't it help you to find the problem?

[Yao-MR-zz]

@caishunfeng i suppose this is not a bug

maybe this log happen in this Scenes：

1: the master have already finish the load worknode from zk, and in this period
the master only load the worknode exclude the worknodeinfo,
2: the RefreshResourceTask start sync info before the LowerWeightHostManager start to sync form zk with the worknode info

[caishunfeng]

@caishunfeng i suppose this is not a bug

maybe this log happen in this Scenes：

1: the master have already finish the load worknode from zk, and in this period the master only load the worknode exclude the worknodeinfo, 2: the RefreshResourceTask start sync info before the LowerWeightHostManager start to sync form zk with the worknode info

Yes, you're right. It just occur at sometimes and not effect the process. But it's better to fix the NPE.

[Yao-MR-zz]

@caishunfeng can you help check whether we find the root cause and fix elegently ?

have already give a PR

[caishunfeng]

@caishunfeng can you help check whether we find the root cause and fix elegently ?

have already give a PR

Well done. May be it should update the heartbeat info when the client registry at first time. What do you think?

[Yao-MR-zz]

@caishunfeng can you help check whether we find the root cause and fix elegently ?
have already give a PR

Well done. May be it should update the heartbeat info when the client registry at first time. What do you think?

the LowerWeightHostManager in master will load the worknode and worknode group from zk when

the master start, and the detail worknodeinfo will update in a sepearte thread in the later,

so this is two solution

1. as you say, we should get all the info including the worknode and worknodeinfo when the client registry at first time, and also start a seperte thread to update the worknodeinfo in a sepearte thread
2. keep now logic, and just think this scene is no problem, and we will assign the task to later connected and give detail info worknode, but should elegently fix this npe,

[Yao-MR-zz]

hi @caishunfeng reslolved! and merge the code to dev branch

[caishunfeng]

hi @caishunfeng reslolved! and merge the code to dev branch

Well done. Thanks for your PR.