[Bug]: The flink optimizer often triggers OOM exceptions. In fact, the files processed by each task are not large.

[wardlican]

Why are the changes needed?

Close #3502.

Brief change log

1. In the close method, in addition to setting the worker thread exit flag, the worker thread is also gracefully exited.
2. If the worker thread does not exit, it may cause the problem of repeatedly creating multiple worker threads in some scenarios, which may cause OOM problems due to the existence of multiple worker threads.

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[Aireed]

Can you change the information according to the PR specification?You can refer to #3432

[zhoujinsong]

Thanks for the contribution!

It seems that some check style error you need to fix, you can run mvn spotless:apply in the root directory to do that.

[xxubai]

Can you explain this change. Why make the thread be a daemon thread

[wardlican]

Can you explain this change. Why make the thread be a daemon thread

This design not only ensures graceful exit during normal shutdown, but also provides an additional safety guarantee through the characteristics of daemon threads to prevent threads from preventing JVM exit under abnormal circumstances.

[xxubai]

This design not only ensures graceful exit during normal shutdown, but also provides an additional safety guarantee through the characteristics of daemon threads to prevent threads from preventing JVM exit under abnormal circumstances.

For data process task, maybe user thread is a better choice? WDYT @zhoujinsong

[zhoujinsong]

For data process task, maybe user thread is a better choice? WDYT @zhoujinsong

Setting the thread to daemon looks good to me. The process handling thread should not protect the TM from closing.

[zhoujinsong]

LGTM.