[Feature][Improvement] Support multi cluster environments

[lenboo]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

If there are Shanghai sh, Hangzhou hz, two clusters as an example. The data source of each cluster may be independent, especially multinational companies are restricted by local laws and policies. The same task may need to be performed in different data centers.

• In the case of multiple clusters, split common operations and cluster-related operations.
• Whether the button of a single cluster remains unchanged or the complete set is identified by the available clusters.

Use case

Workflow A are associated with two clusters: C1 / C2.
Run the workflow A, and two workflow instances would be generated:
instance1 with cluster C1;
instance2 with cluster C2.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[lenboo]

Here is my solution:

1. Redefine the concept of the environment, the environment supports env (current status), config/xml and other types, modify the environment meta data, and add the environment type field
2. The field 'environment' is added to the workflow definition, which supports the association of multiple clusters (environments). When the workflow runs, a workflow instance is generated for each cluster.
3. The field "environment" is added to the workflow instance. Each workflow instance specifies a certain cluster, but cannot override the environment specified by the task (environment priority: Task Configuration > Workflow Configuration)
4. How to use the environment configuration is determined by the task execution logic itself

[caishunfeng]

What about adding region for environment? user can change region from UI and see data of selected region.

[lenboo]

Region is a field of environment？

[BoYiZhang]

Supporting multiple environments is a good requirement.

Here are a few questions I can think of that might be involved:

1. The resource center on DS is currently stored on the same set of HDFS, and is pulled from HDFS when DS is executed.
   If it is a multi-cluster, you need to consider how to design file storage.
2. Currently ds supports kerberbos authentication. If kerberos authentication is enabled for multiple clusters, how does kerberos authenticate?
   Enable kerberbos mutual trust or use the same set of kerberbos authentication?
3. The client/configuration files of multiple cluster environments are different, how to manage them. For example, hadoop and spark versions are configured differently.
4. The general requirement to support multiple environments should be to push tasks from the development environment to the production environment. Is there a review and release process for this?

---

支持多环境这是一个好的需求.

这是我想到的可能涉及的几个问题:

1.DS上的资源中心目前是存储在同一套HDFS上, DS执行的时候从HDFS上拉取.
如果是多集群的话,需要考虑一下文件存储方面如何设计.
2.目前ds支持kerberbos认证, 如果是多个集群开启kerberos认证的话, kerberos如何认证?
开启kerberbos互信还是使用同一套kerberbos认证?
3.多个集群环境的客户端/配置文件不一样,如何管理. 比如hadoop,spark版本配置不同.
4.支持多个环境的常规诉求应该是是开发环境任务推送到生产环境. 这个是否有审核发布流程?