Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Zeta] Added the metrics information of table statistics in multi-table mode #7212

Merged
merged 9 commits into from
Jul 26, 2024
Merged

[Feature][Zeta] Added the metrics information of table statistics in multi-table mode #7212

merged 9 commits into from
Jul 26, 2024

Conversation

hawk9821
Copy link
Contributor

…ics in multi-table mode #6959

Purpose of this pull request

Solve the problem of historical submission confusion #7088

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@hawk9821
Copy link
Contributor Author

@Hisoka-X

Hisoka-X
Hisoka-X reviewed Jul 22, 2024
View reviewed changes
...e-server/src/main/java/org/apache/seatunnel/engine/server/task/SeaTunnelSourceCollector.java Outdated Show resolved Hide resolved
...e-server/src/main/java/org/apache/seatunnel/engine/server/task/SeaTunnelSourceCollector.java Outdated Show resolved Hide resolved
.github/workflows/backend.yml Outdated
@@ -313,7 +313,7 @@ jobs:
- name: run updated modules integration test (part-1)
if: needs.changes.outputs.api == 'false' && needs.changes.outputs.it-modules != ''
run: |
sub_modules=`python tools/update_modules_check/update_modules_check.py sub_update_it_module ${{needs.changes.outputs.it-modules}} 8 0`
sub_modules=`python tools/update_modules_check/update_modules_check.py sub_update_it_module ${{needs.changes.outputs.it-modules}} 10 0`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert ci config change, the dev branch CI work fine at now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

Copy link
Contributor Author

@hawk9821 hawk9821 Jul 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after then revert ci config，ci workflow time out. details as follows
follow the 10 ci workflow, the workflow named updated-modules-integration-test-part is not executed. details as follows

...e-server/src/main/java/org/apache/seatunnel/engine/server/task/SeaTunnelSourceCollector.java Outdated Show resolved Hide resolved
...engine-server/src/main/java/org/apache/seatunnel/engine/server/task/SourceSeaTunnelTask.java Outdated Show resolved Hide resolved
@github-actions github-actions bot added CI&CD and removed CI&CD labels Jul 23, 2024
@Hisoka-X
Copy link
Member

please resolve conflicts

@github-actions github-actions bot removed the CI&CD label Jul 24, 2024
@Hisoka-X Hisoka-X changed the title [Feature][Zeta engine] Added the metrics information of table statistics in multi-table mode [Feature][Zeta] Added the metrics information of table statistics in multi-table mode Jul 24, 2024
Hisoka-X
Hisoka-X reviewed Jul 24, 2024
View reviewed changes
...e-e2e/connector-seatunnel-e2e-base/src/test/resources/batch_fake_multi_table_to_console.conf Outdated Show resolved Hide resolved
...e-e2e/connector-seatunnel-e2e-base/src/test/resources/batch_fake_multi_table_to_console.conf Outdated Show resolved Hide resolved
seatunnel-engine/seatunnel-engine-server/pom.xml Outdated Show resolved Hide resolved
...engine-server/src/main/java/org/apache/seatunnel/engine/server/task/SourceSeaTunnelTask.java Outdated
Comment on lines 91 to 95
tableIdentifier ->
TablePath.of(
tableIdentifier.getDatabaseName(),
tableIdentifier.getSchemaName(),
tableIdentifier.getTableName()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
tableIdentifier ->
TablePath.of(
tableIdentifier.getDatabaseName(),
tableIdentifier.getSchemaName(),
tableIdentifier.getTableName()))
tableIdentifier ->
tableIdentifier.toTablePath()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.map(
tableIdentifier ->
TablePath.of(
tableIdentifier.getDatabaseName(),
tableIdentifier.getSchemaName(),
tableIdentifier.getTableName()))
change :
.map(TableIdentifier::toTablePath)

Hisoka-X
Hisoka-X reviewed Jul 24, 2024
View reviewed changes
...erver/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java
@@ -79,7 +82,9 @@
public class RestHttpGetCommandProcessor extends HttpCommandProcessor<HttpGetCommand> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @liugddx

hawk9821 and others added 5 commits July 24, 2024 12:30
@hawk9821 @Hisoka-X
Update seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-bas…
5017b9f 
…e/src/test/resources/batch_fake_multi_table_to_console.conf

Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
@hawk9821 @Hisoka-X
Update seatunnel-engine/seatunnel-engine-server/pom.xml
f5b8cf6 
Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
@hawk9821 @Hisoka-X
Update seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-bas…
be9289b 
…e/src/test/resources/batch_fake_multi_table_to_console.conf

Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
@hawk9821
[Feature][Zeta engine] Added the metrics information of table statist…
e265654 
…ics in multi-table mode apache#6959
@hawk9821
Revert "The isNullable attribute is true when the primary key field i…
f448234 
…n the Paimon table converts the Column object. apache#7231"

This reverts commit 83b3b4f.
@github-actions github-actions bot removed the CI&CD label Jul 25, 2024
Hisoka-X
Hisoka-X approved these changes Jul 25, 2024
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if ci passed.

@hailin0 hailin0 merged commit d003bd8 into apache:dev Jul 26, 2024
5 checks passed
hawk9821 added a commit to hawk9821/seatunnel that referenced this pull request Jul 27, 2024
@hawk9821
[Feature][Zeta] Added the metrics information of table statistics in …
43dd0e6 
…multi-table mode (apache#7212)
@hawk9821 hawk9821 deleted the 6959 branch August 3, 2024 03:02
jovezhong added a commit to jovezhong/seatunnel that referenced this pull request Aug 12, 2024
@jovezhong
Squashed commit of the following:
e8e887b 
commit 9d161d2
Author: lizhenglei <127465317+jackyyyyyssss@users.noreply.github.com>
Date:   Mon Aug 12 14:03:21 2024 +0800

    [Fix] Fix http e2e case (apache#7356)

    * 1

    * fix

    * fix

    * fix

    * fix

    * fix

    ---------

    Co-authored-by: lizhenglei <673421862@qq.com>

commit 2489f64
Author: hailin0 <wanghailin@apache.org>
Date:   Mon Aug 12 13:36:15 2024 +0800

    [Improve][Connector] Add multi-table sink option check (apache#7360)

    * [Improve][Connector] Add multi-table sink option check

    * fix

commit fa34ac9
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Mon Aug 12 13:35:43 2024 +0800

    [Improve][API] Check catalog table fields name legal before send to downstream (apache#7358)

    * [Improve][API] Check catalog table fields name legal before send to downstream

    * update

commit 068c5e3
Author: Tyrantlucifer <TyrantLucifer@gmail.com>
Date:   Mon Aug 12 12:55:41 2024 +0800

    [Core][Flink] refactor flink proxy source/sink (apache#7355)

commit e23e3ac
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Aug 11 17:40:02 2024 +0800

    Bump org.apache.activemq:activemq-client (apache#7323)

    Bumps org.apache.activemq:activemq-client from 5.14.5 to 5.15.16.

commit d46cf16
Author: Jarvis <liunaijie1996@163.com>
Date:   Sat Aug 10 23:47:12 2024 +0800

    [Feature] Split transform and move jar into connectors directory (apache#7218)

commit 862e205
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Sat Aug 10 15:21:36 2024 +0800

    [Improve][Document] Change deprecated connector name in setup.md (apache#7366)

commit 2769ed5
Author: hailin0 <wanghailin@apache.org>
Date:   Sat Aug 10 09:58:03 2024 +0800

    [Hotfix][Connector] Fix jdbc compile error (apache#7359)

commit 3130ae0
Author: xxsc0529 <93303124+xxsc0529@users.noreply.github.com>
Date:   Fri Aug 9 20:02:06 2024 +0800

    [Fix][Connector-V2][OceanBase] Remove OceanBase catalog's dependency on mysql driver (apache#7311)

commit 9216627
Author: chaos <71205599+chaos-cn@users.noreply.github.com>
Date:   Fri Aug 9 13:53:28 2024 +0800

    Update ConsoleSinkFactory.java (apache#7350)

    Add existing rules to the optionRule

commit dc3c239
Author: dailai <dailai@chinatelecom.cn>
Date:   Fri Aug 9 13:11:30 2024 +0800

    [Improve][Jdbc] Skip all index when auto create table to improve performance of write (apache#7288)

commit 16950a6
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Thu Aug 8 17:27:47 2024 +0800

    [Improve][Connector-V2] Improve doris error msg (apache#7343)

commit a18fca8
Author: Alex Ting <alextinng@hotmail.com>
Date:   Thu Aug 8 11:22:41 2024 +0800

    [Fix][Connector-tdengine] Fix sql exception and concurrentmodifyexception when connect to taos and read data

commit 064fcad
Author: lizhenglei <127465317+jackyyyyyssss@users.noreply.github.com>
Date:   Wed Aug 7 21:57:57 2024 +0800

    [Improve][Transform] Improve DynamicCompile transform (apache#7319)

commit 16eeb1c
Author: He Wang <wanghechn@qq.com>
Date:   Wed Aug 7 14:54:46 2024 +0800

    [Improve][Jdbc] Remove MysqlType references in JdbcDialect (apache#7333)

commit 191d9e1
Author: ChunFuWu <319355703@qq.com>
Date:   Wed Aug 7 14:53:51 2024 +0800

    [FIX][SFTP] Fix username parameter error in sftp sink document (apache#7334)

commit 764d8b0
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Wed Aug 7 12:46:30 2024 +0800

    [Improve][API] Make sure the table name in TablePath not be null (apache#7252)

commit 8916523
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:46:12 2024 +0800

    [Hotfix][Connector] Fix kafka consumer log next startup offset (apache#7312)

commit 819c685
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:45:54 2024 +0800

    [Improve][Jdbc] Merge user config primary key when create table (apache#7313)

commit 7c3cd99
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:44:41 2024 +0800

    [Improve] Remove unused code (apache#7324)

commit 460e73e
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:44:24 2024 +0800

    [E2E] Enable fakesource e2e of spark/flink (apache#7325)

commit f6a1e51
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:44:13 2024 +0800

    [E2E] Enable JdbcPostgresIdentifierIT (apache#7326)

commit a12786b
Author: Carl-Zhou-CN <1058249259@qq.com>
Date:   Wed Aug 7 12:27:50 2024 +0800

    [DOC][Oss] fix document configuration is rectified when the oss is selected as the checkpoint base (apache#7332)

commit a4db64d
Author: zhangdonghao <39961809+hawk9821@users.noreply.github.com>
Date:   Wed Aug 7 11:52:19 2024 +0800

    [Improve][E2E] Support windows for the e2e of paimon (apache#7329)

commit 855254e
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Wed Aug 7 11:03:06 2024 +0800

    [Feature][Transform] Add LLM transform (apache#7303)

    * [Feature][Transform] Add LLM transform

    * update

    * update

    * retrigger

commit f0cefbe
Author: Arin <136636751+asapekia@users.noreply.github.com>
Date:   Tue Aug 6 19:23:38 2024 +0530

    [Feature] [Activemq] Added activemq sink  (apache#7251)

commit bb2c912
Author: Guangdong Liu <804167098@qq.com>
Date:   Tue Aug 6 21:46:12 2024 +0800

    [Fix][Doris] Fix the abnormality of deleting data in CDC scenario. (apache#7315)

commit c94ea32
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Tue Aug 6 11:23:52 2024 +0800

    [Fix][Doc] Fix miss sink-options-placeholders.md in sidebars (apache#7310)

commit 73632ba
Author: hailin0 <wanghailin@apache.org>
Date:   Tue Aug 6 10:08:45 2024 +0800

    [Hotfix][Zeta] Fix task cannot be stopped when system is busy (apache#7292)

commit 9d56cc3
Author: virvle <55478661+virvle@users.noreply.github.com>
Date:   Mon Aug 5 17:51:26 2024 +0800

    Update Oracle-CDC.md (apache#7285)

commit 4f120ff
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Mon Aug 5 14:34:01 2024 +0800

    [Improve] Update snapshot version to 2.3.7 (apache#7305)

commit 2fd4eec
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Mon Aug 5 11:35:02 2024 +0800

    [Fix][Doc] Fix hybrid cluster deployment document display error (apache#7306)

commit 82f5d8c
Author: hailin0 <wanghailin@apache.org>
Date:   Sat Aug 3 12:02:19 2024 +0800

    [Improve][SQL-Transform] Remove escape identifier from output fields (apache#7297)

commit 34a6b8e
Author: Jast <admin@hadoop.wiki>
Date:   Wed Jul 31 14:43:11 2024 +0800

    [hotfix][connector-v2-hbase]fix and  optimize hbase source problem (apache#7148)

    * [hotfix][improve][doc]optimize connector hbase source

    * [doc]add dependent document

    * [doc]update dependent document

    * [improve]improve static use

    * [hotfix]add test case

    * [hotfix]add test case

    ---------

    Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

commit 9df557c
Author: lizhenglei <127465317+jackyyyyyssss@users.noreply.github.com>
Date:   Tue Jul 30 21:37:04 2024 +0800

    [Improve][Transform] Improve DynamicCompile transform  (apache#7264)

commit b9acb57
Author: hailin0 <wanghailin@apache.org>
Date:   Tue Jul 30 16:12:57 2024 +0800

    [Feature][Transforms] Support cast to bytes function of sql (apache#7284)

commit ec1c319
Author: Nothing. <oswinwuo.o@gmail.com>
Date:   Mon Jul 29 21:53:01 2024 +0800

    [Docs]translate event-listener doc into chinese (apache#7274)

commit f012b2a
Author: dailai <dailai@chinatelecom.cn>
Date:   Mon Jul 29 21:51:59 2024 +0800

    [Improve][Connector-v2] Optimize the way of databases and tables are checked for existence (apache#7261)

commit 4897491
Author: 卢宗柱 <luzongzhu@sina.com>
Date:   Mon Jul 29 17:29:23 2024 +0800

    fix [Bug] Unable to create a source for identifier 'Iceberg'. apache#7182 (apache#7279)

    Co-authored-by: luzongzhu <luzongzhu@semi-tech.com>

commit 21c4f52
Author: corgy-w <73771213+corgy-w@users.noreply.github.com>
Date:   Mon Jul 29 17:28:10 2024 +0800

    [Improve][Core] Improve base on plugin name of lookup strategy (apache#7278)

    * [bug][plugin-discovery] fix multi plugin discovery

    * [bug][plugin-discovery] optimize code

    ---------

    Co-authored-by: wangchao <wangchao001@zhongan.com>

commit 3ccc6a8
Author: hailin0 <wanghailin@apache.org>
Date:   Mon Jul 29 10:41:09 2024 +0800

    [Hotfix][Zeta] Fix task cannot be stopped when system is busy (apache#7280)

commit d886495
Author: Guangdong Liu <804167098@qq.com>
Date:   Fri Jul 26 10:45:32 2024 +0800

    [Fix][Connector-V2] Fix doris TRANSFER_ENCODING header error (apache#7267)

commit 31e59cd
Author: Xuzz <90698333+Xuzhengz@users.noreply.github.com>
Date:   Fri Jul 26 10:44:50 2024 +0800

    [Feature][Jdbc] Support hive compatibleMode add inceptor dialect (apache#7262)

commit d003bd8
Author: zhangdonghao <39961809+hawk9821@users.noreply.github.com>
Date:   Fri Jul 26 10:44:09 2024 +0800

    [Feature][Zeta] Added the metrics information of table statistics in multi-table mode (apache#7212)

commit 876d2f0
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Thu Jul 25 10:24:55 2024 +0800

    [Fix][Dist] Remove aws jar dependency in release package (apache#7257)

    * [Fix][Dist] Remove aws jar dependency in release package

    * [Fix][Dist] Remove aws jar dependency in release package

commit 0d08b20
Author: dailai <dailai@chinatelecom.cn>
Date:   Thu Jul 25 10:00:13 2024 +0800

    [Improve][Connector-v2] Optimize the count table rows for jdbc-oracle and oracle-cdc (apache#7248)

commit cc59499
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Tue Jul 23 14:27:07 2024 +0800

    [Improve][Core] Move MultiTableSink to seatunnel-api module (apache#7243)

    * [Improve][Core] Move MultiTableSink to seatunnel-api module

    * [Improve][Core] Move MultiTableSink to seatunnel-api module

commit fe0c477
Author: Eric <gaojun2048@gmail.com>
Date:   Tue Jul 23 08:36:58 2024 +0800

    [Hotfix][Zeta] Fix taskgroup failed log lost (apache#7241)

commit b0fe432
Author: hawk9821 <39961809+hawk9821@users.noreply.github.com>
Date:   Mon Jul 22 16:12:12 2024 +0800

    The isNullable attribute is true when the primary key field in the Paimon table converts the Column object. apache#7231 (apache#7242)

commit 4b3af9b
Author: Guangdong Liu <804167098@qq.com>
Date:   Mon Jul 22 13:04:19 2024 +0800

    [Improve][Doris Connector] Unified serialization method,Use RowToJsonConverter and TextSerializationSchema (apache#7229)

    * 1

    * 1

    * 1

    * Update seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java

    Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

    ---------

    Co-authored-by: gdliu3 <gdliu3@iflytek.com>
    Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

commit 4ec25f3
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Mon Jul 22 12:01:36 2024 +0800

    [Fix][Zeta] Fix release slot resource twice (apache#7236)

commit 7dc3fa8
Author: litiliu <38579068+litiliu@users.noreply.github.com>
Date:   Mon Jul 22 11:41:33 2024 +0800

    [Fix][Connector kafka]Fix Kafka consumer stop fetching after TM node restarted (apache#7233)

commit 063f83c
Author: Guangdong Liu <804167098@qq.com>
Date:   Mon Jul 22 10:40:58 2024 +0800

    [Improve][Restapi] Add ip and port to monitoring information (apache#7203)

commit c4ca741
Author: hailin0 <wanghailin@apache.org>
Date:   Sat Jul 20 21:00:06 2024 +0800

    [Feature][Core] Support using upstream table placeholders in sink options and auto replacement (apache#7131)

    、

commit 821dfc8
Author: lizhenglei <127465317+jackyyyyyssss@users.noreply.github.com>
Date:   Sat Jul 20 20:25:50 2024 +0800

    [Feature][Transform] DynamicCompile add transform  (apache#7170)

commit 1e7c78d
Author: hilo <1964736605@qq.com>
Date:   Fri Jul 19 19:44:18 2024 +0800

    [Improve][Doc] Add IMap OSS config jars into document (apache#7029)

commit d39ad93
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Fri Jul 19 19:42:03 2024 +0800

    [Fix][Zeta] Fix hybrid deployment can not get worker when init (apache#7235)

commit 2058f5d
Author: ChunFuWu <319355703@qq.com>
Date:   Fri Jul 19 14:21:23 2024 +0800

    [Chore][NOTICE] Correct year in NOTICE file (apache#7232)

commit 6e33a97
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Thu Jul 18 23:03:27 2024 +0800

    [Fix] Fix Hana type converter decimal scale is 0 convert to int error (apache#7167)

commit d1b4a7f
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Thu Jul 18 15:31:30 2024 +0800

    [Improve][Zeta] Add check for submit duplicate job id (apache#7021)

commit e44e8b9
Author: hailin0 <wanghailin@apache.org>
Date:   Thu Jul 18 15:30:49 2024 +0800

    [Improve][Jdbc] Support write unicode text into sqlserver (apache#7159)

commit f40f11a
Author: tcodehuber <tcodehuber@gmail.com>
Date:   Thu Jul 18 00:03:33 2024 +0800

    [Docs] Update engine related docs info (apache#7228)
chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024
@hawk9821
[Feature][Zeta] Added the metrics information of table statistics in …
c5c3008 
…multi-table mode (apache#7212)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hisoka-X Hisoka-X Hisoka-X approved these changes

@hailin0 hailin0 hailin0 approved these changes

Assignees
No one assigned
Labels
api approved e2e reviewed Zeta Rest API Zeta
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hawk9821 @Hisoka-X @hailin0