[Feature][Zeta] Add tag to node used to filter worker when submit job #7045

liunaijie · 2024-06-21T09:24:41Z

Purpose of this pull request

close #7036

Does this PR introduce any user-facing change?

yes, use can add tag in seatunnel config use to choose node by tag

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

…node

Hisoka-X

Thanks @liunaijie ! This feature is very important for SeaTunnel.

Hisoka-X · 2024-06-24T02:35:28Z

seatunnel-engine/seatunnel-engine-common/src/main/resources/hazelcast.yaml

+  member-attributes:
+    group:
+      type: string
+      value: platform
+    team:
+      type: string
+      value: team1


Hi, please do not change this. I believe this feature should not open by default.

Hisoka-X · 2024-06-24T02:40:19Z

...rc/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java

+                                        if (workerAttr == null || workerAttr.isEmpty()) {
+                                            return true;
+                                        }


Why workerAttr is empty it will be matched?

Hisoka-X · 2024-06-24T02:40:58Z

...ver/src/main/java/org/apache/seatunnel/engine/server/dag/physical/PhysicalPlanGenerator.java

@@ -150,7 +151,8 @@ public PhysicalPlanGenerator(
    }

    public Tuple2<PhysicalPlan, Map<Integer, CheckpointPlan>> generate() {
-
+        Map<String, Object> envOptions = jobImmutableInformation.getJobConfig().getEnvOptions();
+        Map<String, String> tags = JsonUtils.toMap(envOptions.getOrDefault("tag", "").toString());


Please add tag into

seatunnel/seatunnel-api/src/main/java/org/apache/seatunnel/api/env/EnvCommonOptions.java

Line 27 in 476d492

public interface EnvCommonOptions {

Hisoka-X · 2024-06-24T02:54:43Z

docs/en/seatunnel-engine/resource-manager.md

@@ -0,0 +1,81 @@
+---
+
+sidebar_position: 9


Add this page into

seatunnel/docs/sidebars.js

Line 172 in 476d492

"items": [

liunaijie · 2024-06-24T08:25:17Z

docs/images/architecture_diagram.png

move image folder from docs/en/images to docs/images

liunaijie · 2024-06-24T08:31:24Z

The tag filter can also apply to task level, but for now not implement this.

Hisoka-X

Please add a test case with end to end use tag filter feature too.

Hisoka-X · 2024-06-24T10:05:04Z

docs/sidebars.js

@@ -178,7 +178,8 @@ const sidebars = {
                "seatunnel-engine/checkpoint-storage",
                "seatunnel-engine/rest-api",
                "seatunnel-engine/tcp",
-                "seatunnel-engine/engine-jar-storage-mode"
+                "seatunnel-engine/engine-jar-storage-mode",
+                "seatunnel-engine/resource-manager",


how about named this part as Resource Isolation?

Hisoka-X · 2024-06-24T10:12:54Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/ReadonlyConfig.java

@@ -96,6 +96,10 @@ public void toMap(Map<String, String> result) {
        }
    }

+    public Map<String, Object> getMapConfig() {


Suggested change

public Map<String, Object> getMapConfig() {

public Map<String, Object> getSourceMap() {

Hisoka-X · 2024-06-24T10:21:00Z

seatunnel-examples/seatunnel-engine-examples/src/main/resources/examples/fake_to_console.conf

+  tag_filter {
+    group = "platform"
+    team = "team1"
+  }


Please revert.

Hisoka-X

LGTM except some minior problem.

Hisoka-X · 2024-06-25T07:08:28Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/env/EnvCommonOptions.java

+            Options.key("tag_filter")
+                    .mapType()
+                    .noDefaultValue()
+                    .withDescription("custom parameters for run engine");


Suggested change

.withDescription("custom parameters for run engine");

.withDescription("Define the worker where the job runs by tag");

Hisoka-X · 2024-06-25T07:10:16Z

...rc/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java

+                                            Map.Entry::getKey, Map.Entry::getValue));
+        }
+        if (matchedWorker.isEmpty()) {
+            log.error("No matched worker.");


Suggested change

log.error("No matched worker.");

log.error("No matched worker with tag filter {}.", tagFilter);

hailin0 · 2024-06-26T11:29:12Z

docs/en/seatunnel-engine/resource-isolation.md

+  tag_filter {
+    group = "platform"
+    team = "team1"
+  }


How to adapt to flink/spark？

env{} is visible to all engines

At now, it is fine to only take effect under the zeta engine. We should not mention it on other engine's doc.

…apache#7045)

liunaijie added 2 commits June 21, 2024 16:50

[Feature] add tag to node, job config. when submit use tag to filter …

77ae1ba

…node

[Feature] add tag to node, job config. when submit use tag to filter …

250f4ba

…node

Hisoka-X added the feature New feature label Jun 24, 2024

Hisoka-X changed the title ~~[Feature] add tag to node, job config. when submit use tag to filter node~~ [Feature][Zeta] Add tag to node used to filter worker when submit job Jun 24, 2024

Hisoka-X reviewed Jun 24, 2024

View reviewed changes

update

e3addf7

Hisoka-X added the Zeta label Jun 24, 2024

liunaijie added 2 commits June 24, 2024 14:09

update document

01cb36d

update document

b4fa988

liunaijie force-pushed the tag_node branch from ee95c26 to b4fa988 Compare June 24, 2024 06:18

liunaijie added 2 commits June 24, 2024 14:32

update

6b928f4

update

40263fe

liunaijie commented Jun 24, 2024

View reviewed changes

docs/images/architecture_diagram.png Outdated

Copy link

Contributor Author

liunaijie Jun 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move image folder from docs/en/images to docs/images

Hisoka-X reviewed Jun 24, 2024

View reviewed changes

liunaijie added 4 commits June 24, 2024 20:47

update

8afc1d1

update

7462467

update

0560422

update

eee410b

liunaijie force-pushed the tag_node branch from 631fc64 to eee410b Compare June 25, 2024 01:15

Hisoka-X reviewed Jun 25, 2024

View reviewed changes

update

a26b9f1

Hisoka-X approved these changes Jun 25, 2024

View reviewed changes

github-actions bot added approved reviewed labels Jun 25, 2024

hailin0 reviewed Jun 26, 2024

View reviewed changes

hailin0 approved these changes Jun 26, 2024

View reviewed changes

hailin0 merged commit 3af6b2f into apache:dev Jun 26, 2024
10 checks passed

hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Jul 1, 2024

[Feature][Zeta] Add tag to node used to filter worker when submit job (…

e4fde37

…apache#7045)

liunaijie deleted the tag_node branch July 3, 2024 10:46

This was referenced Jul 7, 2024

[Improve][RestAPI] overview api add tag filter #7123

Closed

SeaTunnel community meeting Topic collect #7134

Closed

Thomas-HuWei pushed a commit to zilliztech/seatunnel that referenced this pull request Jul 10, 2024

[Feature][Zeta] Add tag to node used to filter worker when submit job (…

bd908d1

…apache#7045)

hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Jul 13, 2024

[Feature][Zeta] Add tag to node used to filter worker when submit job (…

cac45fe

…apache#7045)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Zeta] Add tag to node used to filter worker when submit job #7045

[Feature][Zeta] Add tag to node used to filter worker when submit job #7045

liunaijie commented Jun 21, 2024

Hisoka-X left a comment

Hisoka-X Jun 24, 2024

liunaijie Jun 24, 2024

Hisoka-X Jun 24, 2024

liunaijie Jun 24, 2024

Hisoka-X Jun 24, 2024

liunaijie Jun 24, 2024

Hisoka-X Jun 24, 2024

liunaijie Jun 24, 2024

liunaijie Jun 24, 2024

liunaijie commented Jun 24, 2024

Hisoka-X left a comment

Hisoka-X Jun 24, 2024

Hisoka-X Jun 24, 2024

Hisoka-X Jun 24, 2024

Hisoka-X left a comment

Hisoka-X Jun 25, 2024

Hisoka-X Jun 25, 2024

hailin0 Jun 26, 2024

Hisoka-X Jun 26, 2024

hailin0 Jun 26, 2024

	public Map<String, Object> getMapConfig() {
	public Map<String, Object> getSourceMap() {

	.withDescription("custom parameters for run engine");
	.withDescription("Define the worker where the job runs by tag");

	log.error("No matched worker.");
	log.error("No matched worker with tag filter {}.", tagFilter);

		@@ -0,0 +1,81 @@
		---

		sidebar_position: 9

[Feature][Zeta] Add tag to node used to filter worker when submit job #7045

[Feature][Zeta] Add tag to node used to filter worker when submit job #7045

Conversation

liunaijie commented Jun 21, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liunaijie commented Jun 24, 2024

Hisoka-X left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment