[Feature][Zeta Engine] Separate the responsibilities of Master and Worker to improve the stability of the cluster

[EricJoy2048]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

1. At present, the Zeta engine does not distinguish between master nodes and worker nodes. All nodes can serve as master and worker nodes, and only one node in the entire cluster is marked as a master node at the same time. The master is responsible for receiving tasks submitted by clients and scheduling tasks, while also taking on the task running work.
   When there are many tasks running in the cluster, the load on the master node can be very high, which may cause the master node to hang up and perform fault tolerance on the master node. Master node fault tolerance will further increase the responsibility of the cluster, especially newly selected master nodes have a greater chance of failure due to excessive load.

2. The goal of this proposal is to separate the roles of the master and worker nodes. The master node is only responsible for task scheduling, IMAP data storage, and access; Worker nodes are only responsible for the operation of tasks, and do not participate in the storage and access of IMAP data, nor do they participate in parameter elections.

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[liunaijie]

the createJoiner method please also consider ths kubernetes case.

https://github.com/apache/seatunnel/pull/6500/files#diff-02535afd997b497cf23932d7244a15c5a7a37799f64b07db573eab32555c21d9R33
the config used within k8s.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.