[Feature][Engine] Unify job env parameters #6003

happyboy1024 · 2023-12-13T13:42:26Z

Purpose of this pull request

Adjust the flink parameter configuration mode by prefixing flink. and the parameter items provided by flink official
Clear outdated configurations in the md file and e2e
close [Feature][Engine] Unify job env parameters #5937

Does this PR introduce any user-facing change?

How was this patch tested?

add new test cases to separately verify that the parameters are consistent before and after adjustments

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

…sed abnormally (apache#5008)

…efore fetch task start (apache#5008)

hailin0 · 2023-12-14T06:58:13Z

Please check ci

# Conflicts: # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kudu-e2e/src/test/resources/kudu_to_assert_with_multipletable.conf

Hisoka-X · 2023-12-20T07:56:39Z

...rter-common/src/main/java/org/apache/seatunnel/core/starter/flink/utils/EnvironmentUtil.java

+    public static void initTableEnvironmentConfiguration(
+            Config config, Configuration configuration) {
+        /**
+         * flink table configuration items are prefixed with 'table.exec'. reference: {@link
+         * org.apache.flink.table.api.config.ExecutionConfigOptions}
+         */
+        String prefixConf = "flink.table.exec";
+        String replacePrefix = "flink.";
+        if (!config.isEmpty()) {
+            for (Map.Entry<String, ConfigValue> entryConfKey : config.entrySet()) {
+                String confKey = entryConfKey.getKey().trim();
                if (confKey.startsWith(prefixConf)) {
                    configuration.setString(
-                            confKey.replaceFirst(prefixConf, ""), entryConfKey.getValue().render());
+                            confKey.replaceFirst(replacePrefix, ""),
+                            entryConfKey.getValue().unwrapped().toString());
                }
            }
        }


please add test case for this.

ignore below
Look like it remove prefix of flink.table.exec.config to table.exec.config. Not worked on flink.pipeline.max-parallelism according doc https://github.com/apache/seatunnel/pull/6003/files#diff-8f9b660879c6a25ff9d0f64619025ea7be9a72cd0152c4b896843424cab6abbcR44

Hisoka-X · 2023-12-20T08:01:09Z

...rter-common/src/main/java/org/apache/seatunnel/core/starter/flink/utils/EnvironmentUtil.java

+    public static void initTableEnvironmentConfiguration(
+            Config config, Configuration configuration) {
+        /**
+         * flink table configuration items are prefixed with 'table.exec'. reference: {@link
+         * org.apache.flink.table.api.config.ExecutionConfigOptions}
+         */
+        String prefixConf = "flink.table.exec";
+        String replacePrefix = "flink.";
+        if (!config.isEmpty()) {
+            for (Map.Entry<String, ConfigValue> entryConfKey : config.entrySet()) {
+                String confKey = entryConfKey.getKey().trim();
                if (confKey.startsWith(prefixConf)) {
                    configuration.setString(
-                            confKey.replaceFirst(prefixConf, ""), entryConfKey.getValue().render());
+                            confKey.replaceFirst(replacePrefix, ""),
+                            entryConfKey.getValue().unwrapped().toString());
                }
            }
        }


Overall LGTM, but please add a test case for flink.table.exec. feature.

Overall LGTM, but please add a test case for flink.table.exec. feature.

I've added a test case for this that uses table env parallelism to override global parallelism. PTAL.

Hisoka-X · 2023-12-22T03:38:04Z

...or-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/UnifyEnvParameterIT.java

+@Slf4j
+@DisabledOnContainer(
+        value = {},
+        type = {EngineType.SEATUNNEL, EngineType.SPARK},


LGTM, but I think we should create new place to store test only for flink in the future.

LGTM, but I think we should create new place to store test only for flink in the future.

Agree with you, we can add a new module under seatunnel-engine-e2e, this module can be called seatunnel-engine-flink-e2e or seatunnel-flink-engine-e2e.

Yep. Would you mind to do this after this PR be merged?

Yep. Would you mind to do this after this PR be merged?

Of course not, after this pr merge, I will do this.

dengjunjie and others added 21 commits July 3, 2023 17:15

[Bug][connector-cdc-mysql] mysql connections and memory of jvm increa…

4756e79

…sed abnormally (apache#5008)

Merge branch 'apache:dev' into dev

2cb759a

Merge branch 'apache:dev' into dev

6ada441

Merge branch 'apache:dev' into dev

cbe375d

Merge branch 'apache:dev' into dev

d6275ec

[bugfix][connector-cdc-mysql] reset the listener of binaryLogClient b…

68dbb48

…efore fetch task start (apache#5008)

Merge branch 'apache:dev' into dev

5b7e4d5

Merge branch 'apache:dev' into dev

7769017

Merge branch 'apache:dev' into dev

0de3c21

Merge branch 'apache:dev' into dev

7a3467e

Merge branch 'apache:dev' into dev

7262bd9

Merge branch 'apache:dev' into dev

0d55ee3

Merge branch 'apache:dev' into dev

0bfa9b8

Merge branch 'apache:dev' into dev

22a9a38

Merge branch 'apache:dev' into dev

1ce9847

Merge branch 'apache:dev' into dev

8dfcc4b

Merge branch 'apache:dev' into dev

d1dc354

Merge branch 'apache:dev' into dev

3df3e1a

Merge branch 'apache:dev' into dev

4e7302b

Merge branch 'apache:dev' into dev

48ea3fe

[Feature][Engine] Unify job env parameters (apache#5937)

e40af71

Merge branch 'apache:dev' into dev

d9a2dbb

hailin0 added this to the 2.3.4 milestone Dec 17, 2023

happyboy1024 and others added 5 commits December 17, 2023 11:56

Merge branch 'apache:dev' into dev

f664b3a

Merge branch 'apache:dev' into dev

8b28cbb

Merge branch 'apache:dev' into dev

f0e7ca6

Merge remote-tracking branch 'origin/dev' into unify-parameter

345e48f

# Conflicts: # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kudu-e2e/src/test/resources/kudu_to_assert_with_multipletable.conf

fix ci error and adjust the new configuration file

d996cbb

Hisoka-X reviewed Dec 20, 2023

View reviewed changes

Hisoka-X approved these changes Dec 20, 2023

View reviewed changes

github-actions bot added approved reviewed labels Dec 20, 2023

Hisoka-X requested changes Dec 20, 2023

View reviewed changes

github-actions bot removed approved reviewed labels Dec 20, 2023

hailin0 mentioned this pull request Dec 20, 2023

[Improve][Core] Unified the parallelism config key #6048

Closed

4 tasks

Carl-Zhou-CN mentioned this pull request Dec 21, 2023

[Improve][Connector-V2] Clear expired profile and fix some profile expired parameters #5960

Closed

4 tasks

happyboy1024 and others added 3 commits December 21, 2023 21:11

add a test case for configuration of flink-table-env

5f7577e

Merge branch 'apache:dev' into dev

de0cf26

Merge remote-tracking branch 'origin/dev' into unify-parameter

0bd77a0

Hisoka-X approved these changes Dec 22, 2023

View reviewed changes

github-actions bot added approved reviewed labels Dec 22, 2023

hailin0 approved these changes Dec 22, 2023

View reviewed changes

hailin0 merged commit 2410ab3 into apache:dev Dec 22, 2023
11 checks passed

hailin0 mentioned this pull request Dec 24, 2023

[Feature][OssFile Connector] Make Oss implement source factory and sink factory #6062

Merged

4 tasks

alextinng pushed a commit to alextinng/seatunnel that referenced this pull request Dec 26, 2023

[Feature][Engine] Unify job env parameters (apache#6003)

2aa08ee

hailin0 mentioned this pull request Dec 26, 2023

[Feature][Connector-v2][PulsarSink]Add Pulsar Sink Connector. #4382

Merged

5 tasks

happyboy1024 deleted the unify-parameter branch March 12, 2024 06:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Engine] Unify job env parameters #6003

[Feature][Engine] Unify job env parameters #6003

happyboy1024 commented Dec 13, 2023 •

edited

Loading

hailin0 commented Dec 14, 2023

Hisoka-X Dec 20, 2023 •

edited

Loading

Hisoka-X Dec 20, 2023

happyboy1024 Dec 21, 2023

Hisoka-X Dec 22, 2023

happyboy1024 Dec 22, 2023

Hisoka-X Dec 22, 2023

happyboy1024 Dec 22, 2023

[Feature][Engine] Unify job env parameters #6003

[Feature][Engine] Unify job env parameters #6003

Conversation

happyboy1024 commented Dec 13, 2023 • edited Loading

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

hailin0 commented Dec 14, 2023

Hisoka-X Dec 20, 2023 • edited Loading

Choose a reason for hiding this comment

Hisoka-X Dec 20, 2023

Choose a reason for hiding this comment

happyboy1024 Dec 21, 2023

Choose a reason for hiding this comment

Hisoka-X Dec 22, 2023

Choose a reason for hiding this comment

happyboy1024 Dec 22, 2023

Choose a reason for hiding this comment

Hisoka-X Dec 22, 2023

Choose a reason for hiding this comment

happyboy1024 Dec 22, 2023

Choose a reason for hiding this comment

happyboy1024 commented Dec 13, 2023 •

edited

Loading

Hisoka-X Dec 20, 2023 •

edited

Loading