Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix][Doris] Fix the abnormality of deleting data in CDC scenario. #7315

Merged
merged 6 commits into from
Aug 6, 2024
Merged

[Fix][Doris] Fix the abnormality of deleting data in CDC scenario. #7315

merged 6 commits into from
Aug 6, 2024

Conversation

liugddx
Copy link
Member

@liugddx liugddx commented Aug 6, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@Hisoka-X Hisoka-X added this to the 2.3.7 milestone Aug 6, 2024
@Hisoka-X Hisoka-X changed the title [Bugfix](doris) Fix the abnormality of deleting data in CDC scenario. [Fix][Doris] Fix the abnormality of deleting data in CDC scenario. Aug 6, 2024
gdliu3 added 2 commits August 6, 2024 15:45
Hisoka-X
Hisoka-X approved these changes Aug 6, 2024
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liugddx !

@hailin0 hailin0 merged commit bb2c912 into apache:dev Aug 6, 2024
5 checks passed
luzongzhu pushed a commit to luzongzhu/seatunnel that referenced this pull request Aug 9, 2024
@liugddx
[Fix][Doris] Fix the abnormality of deleting data in CDC scenario. (a…
17a9bd0 
…pache#7315)
jovezhong added a commit to jovezhong/seatunnel that referenced this pull request Aug 12, 2024
@jovezhong
Squashed commit of the following:
e8e887b 
commit 9d161d2
Author: lizhenglei <127465317+jackyyyyyssss@users.noreply.github.com>
Date:   Mon Aug 12 14:03:21 2024 +0800

    [Fix] Fix http e2e case (apache#7356)

    * 1

    * fix

    * fix

    * fix

    * fix

    * fix

    ---------

    Co-authored-by: lizhenglei <673421862@qq.com>

commit 2489f64
Author: hailin0 <wanghailin@apache.org>
Date:   Mon Aug 12 13:36:15 2024 +0800

    [Improve][Connector] Add multi-table sink option check (apache#7360)

    * [Improve][Connector] Add multi-table sink option check

    * fix

commit fa34ac9
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Mon Aug 12 13:35:43 2024 +0800

    [Improve][API] Check catalog table fields name legal before send to downstream (apache#7358)

    * [Improve][API] Check catalog table fields name legal before send to downstream

    * update

commit 068c5e3
Author: Tyrantlucifer <TyrantLucifer@gmail.com>
Date:   Mon Aug 12 12:55:41 2024 +0800

    [Core][Flink] refactor flink proxy source/sink (apache#7355)

commit e23e3ac
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Aug 11 17:40:02 2024 +0800

    Bump org.apache.activemq:activemq-client (apache#7323)

    Bumps org.apache.activemq:activemq-client from 5.14.5 to 5.15.16.

commit d46cf16
Author: Jarvis <liunaijie1996@163.com>
Date:   Sat Aug 10 23:47:12 2024 +0800

    [Feature] Split transform and move jar into connectors directory (apache#7218)

commit 862e205
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Sat Aug 10 15:21:36 2024 +0800

    [Improve][Document] Change deprecated connector name in setup.md (apache#7366)

commit 2769ed5
Author: hailin0 <wanghailin@apache.org>
Date:   Sat Aug 10 09:58:03 2024 +0800

    [Hotfix][Connector] Fix jdbc compile error (apache#7359)

commit 3130ae0
Author: xxsc0529 <93303124+xxsc0529@users.noreply.github.com>
Date:   Fri Aug 9 20:02:06 2024 +0800

    [Fix][Connector-V2][OceanBase] Remove OceanBase catalog's dependency on mysql driver (apache#7311)

commit 9216627
Author: chaos <71205599+chaos-cn@users.noreply.github.com>
Date:   Fri Aug 9 13:53:28 2024 +0800

    Update ConsoleSinkFactory.java (apache#7350)

    Add existing rules to the optionRule

commit dc3c239
Author: dailai <dailai@chinatelecom.cn>
Date:   Fri Aug 9 13:11:30 2024 +0800

    [Improve][Jdbc] Skip all index when auto create table to improve performance of write (apache#7288)

commit 16950a6
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Thu Aug 8 17:27:47 2024 +0800

    [Improve][Connector-V2] Improve doris error msg (apache#7343)

commit a18fca8
Author: Alex Ting <alextinng@hotmail.com>
Date:   Thu Aug 8 11:22:41 2024 +0800

    [Fix][Connector-tdengine] Fix sql exception and concurrentmodifyexception when connect to taos and read data

commit 064fcad
Author: lizhenglei <127465317+jackyyyyyssss@users.noreply.github.com>
Date:   Wed Aug 7 21:57:57 2024 +0800

    [Improve][Transform] Improve DynamicCompile transform (apache#7319)

commit 16eeb1c
Author: He Wang <wanghechn@qq.com>
Date:   Wed Aug 7 14:54:46 2024 +0800

    [Improve][Jdbc] Remove MysqlType references in JdbcDialect (apache#7333)

commit 191d9e1
Author: ChunFuWu <319355703@qq.com>
Date:   Wed Aug 7 14:53:51 2024 +0800

    [FIX][SFTP] Fix username parameter error in sftp sink document (apache#7334)

commit 764d8b0
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Wed Aug 7 12:46:30 2024 +0800

    [Improve][API] Make sure the table name in TablePath not be null (apache#7252)

commit 8916523
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:46:12 2024 +0800

    [Hotfix][Connector] Fix kafka consumer log next startup offset (apache#7312)

commit 819c685
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:45:54 2024 +0800

    [Improve][Jdbc] Merge user config primary key when create table (apache#7313)

commit 7c3cd99
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:44:41 2024 +0800

    [Improve] Remove unused code (apache#7324)

commit 460e73e
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:44:24 2024 +0800

    [E2E] Enable fakesource e2e of spark/flink (apache#7325)

commit f6a1e51
Author: hailin0 <wanghailin@apache.org>
Date:   Wed Aug 7 12:44:13 2024 +0800

    [E2E] Enable JdbcPostgresIdentifierIT (apache#7326)

commit a12786b
Author: Carl-Zhou-CN <1058249259@qq.com>
Date:   Wed Aug 7 12:27:50 2024 +0800

    [DOC][Oss] fix document configuration is rectified when the oss is selected as the checkpoint base (apache#7332)

commit a4db64d
Author: zhangdonghao <39961809+hawk9821@users.noreply.github.com>
Date:   Wed Aug 7 11:52:19 2024 +0800

    [Improve][E2E] Support windows for the e2e of paimon (apache#7329)

commit 855254e
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Wed Aug 7 11:03:06 2024 +0800

    [Feature][Transform] Add LLM transform (apache#7303)

    * [Feature][Transform] Add LLM transform

    * update

    * update

    * retrigger

commit f0cefbe
Author: Arin <136636751+asapekia@users.noreply.github.com>
Date:   Tue Aug 6 19:23:38 2024 +0530

    [Feature] [Activemq] Added activemq sink  (apache#7251)

commit bb2c912
Author: Guangdong Liu <804167098@qq.com>
Date:   Tue Aug 6 21:46:12 2024 +0800

    [Fix][Doris] Fix the abnormality of deleting data in CDC scenario. (apache#7315)

commit c94ea32
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Tue Aug 6 11:23:52 2024 +0800

    [Fix][Doc] Fix miss sink-options-placeholders.md in sidebars (apache#7310)

commit 73632ba
Author: hailin0 <wanghailin@apache.org>
Date:   Tue Aug 6 10:08:45 2024 +0800

    [Hotfix][Zeta] Fix task cannot be stopped when system is busy (apache#7292)

commit 9d56cc3
Author: virvle <55478661+virvle@users.noreply.github.com>
Date:   Mon Aug 5 17:51:26 2024 +0800

    Update Oracle-CDC.md (apache#7285)

commit 4f120ff
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Mon Aug 5 14:34:01 2024 +0800

    [Improve] Update snapshot version to 2.3.7 (apache#7305)

commit 2fd4eec
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Mon Aug 5 11:35:02 2024 +0800

    [Fix][Doc] Fix hybrid cluster deployment document display error (apache#7306)

commit 82f5d8c
Author: hailin0 <wanghailin@apache.org>
Date:   Sat Aug 3 12:02:19 2024 +0800

    [Improve][SQL-Transform] Remove escape identifier from output fields (apache#7297)

commit 34a6b8e
Author: Jast <admin@hadoop.wiki>
Date:   Wed Jul 31 14:43:11 2024 +0800

    [hotfix][connector-v2-hbase]fix and  optimize hbase source problem (apache#7148)

    * [hotfix][improve][doc]optimize connector hbase source

    * [doc]add dependent document

    * [doc]update dependent document

    * [improve]improve static use

    * [hotfix]add test case

    * [hotfix]add test case

    ---------

    Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

commit 9df557c
Author: lizhenglei <127465317+jackyyyyyssss@users.noreply.github.com>
Date:   Tue Jul 30 21:37:04 2024 +0800

    [Improve][Transform] Improve DynamicCompile transform  (apache#7264)

commit b9acb57
Author: hailin0 <wanghailin@apache.org>
Date:   Tue Jul 30 16:12:57 2024 +0800

    [Feature][Transforms] Support cast to bytes function of sql (apache#7284)

commit ec1c319
Author: Nothing. <oswinwuo.o@gmail.com>
Date:   Mon Jul 29 21:53:01 2024 +0800

    [Docs]translate event-listener doc into chinese (apache#7274)

commit f012b2a
Author: dailai <dailai@chinatelecom.cn>
Date:   Mon Jul 29 21:51:59 2024 +0800

    [Improve][Connector-v2] Optimize the way of databases and tables are checked for existence (apache#7261)

commit 4897491
Author: 卢宗柱 <luzongzhu@sina.com>
Date:   Mon Jul 29 17:29:23 2024 +0800

    fix [Bug] Unable to create a source for identifier 'Iceberg'. apache#7182 (apache#7279)

    Co-authored-by: luzongzhu <luzongzhu@semi-tech.com>

commit 21c4f52
Author: corgy-w <73771213+corgy-w@users.noreply.github.com>
Date:   Mon Jul 29 17:28:10 2024 +0800

    [Improve][Core] Improve base on plugin name of lookup strategy (apache#7278)

    * [bug][plugin-discovery] fix multi plugin discovery

    * [bug][plugin-discovery] optimize code

    ---------

    Co-authored-by: wangchao <wangchao001@zhongan.com>

commit 3ccc6a8
Author: hailin0 <wanghailin@apache.org>
Date:   Mon Jul 29 10:41:09 2024 +0800

    [Hotfix][Zeta] Fix task cannot be stopped when system is busy (apache#7280)

commit d886495
Author: Guangdong Liu <804167098@qq.com>
Date:   Fri Jul 26 10:45:32 2024 +0800

    [Fix][Connector-V2] Fix doris TRANSFER_ENCODING header error (apache#7267)

commit 31e59cd
Author: Xuzz <90698333+Xuzhengz@users.noreply.github.com>
Date:   Fri Jul 26 10:44:50 2024 +0800

    [Feature][Jdbc] Support hive compatibleMode add inceptor dialect (apache#7262)

commit d003bd8
Author: zhangdonghao <39961809+hawk9821@users.noreply.github.com>
Date:   Fri Jul 26 10:44:09 2024 +0800

    [Feature][Zeta] Added the metrics information of table statistics in multi-table mode (apache#7212)

commit 876d2f0
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Thu Jul 25 10:24:55 2024 +0800

    [Fix][Dist] Remove aws jar dependency in release package (apache#7257)

    * [Fix][Dist] Remove aws jar dependency in release package

    * [Fix][Dist] Remove aws jar dependency in release package

commit 0d08b20
Author: dailai <dailai@chinatelecom.cn>
Date:   Thu Jul 25 10:00:13 2024 +0800

    [Improve][Connector-v2] Optimize the count table rows for jdbc-oracle and oracle-cdc (apache#7248)

commit cc59499
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Tue Jul 23 14:27:07 2024 +0800

    [Improve][Core] Move MultiTableSink to seatunnel-api module (apache#7243)

    * [Improve][Core] Move MultiTableSink to seatunnel-api module

    * [Improve][Core] Move MultiTableSink to seatunnel-api module

commit fe0c477
Author: Eric <gaojun2048@gmail.com>
Date:   Tue Jul 23 08:36:58 2024 +0800

    [Hotfix][Zeta] Fix taskgroup failed log lost (apache#7241)

commit b0fe432
Author: hawk9821 <39961809+hawk9821@users.noreply.github.com>
Date:   Mon Jul 22 16:12:12 2024 +0800

    The isNullable attribute is true when the primary key field in the Paimon table converts the Column object. apache#7231 (apache#7242)

commit 4b3af9b
Author: Guangdong Liu <804167098@qq.com>
Date:   Mon Jul 22 13:04:19 2024 +0800

    [Improve][Doris Connector] Unified serialization method,Use RowToJsonConverter and TextSerializationSchema (apache#7229)

    * 1

    * 1

    * 1

    * Update seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java

    Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

    ---------

    Co-authored-by: gdliu3 <gdliu3@iflytek.com>
    Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

commit 4ec25f3
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Mon Jul 22 12:01:36 2024 +0800

    [Fix][Zeta] Fix release slot resource twice (apache#7236)

commit 7dc3fa8
Author: litiliu <38579068+litiliu@users.noreply.github.com>
Date:   Mon Jul 22 11:41:33 2024 +0800

    [Fix][Connector kafka]Fix Kafka consumer stop fetching after TM node restarted (apache#7233)

commit 063f83c
Author: Guangdong Liu <804167098@qq.com>
Date:   Mon Jul 22 10:40:58 2024 +0800

    [Improve][Restapi] Add ip and port to monitoring information (apache#7203)

commit c4ca741
Author: hailin0 <wanghailin@apache.org>
Date:   Sat Jul 20 21:00:06 2024 +0800

    [Feature][Core] Support using upstream table placeholders in sink options and auto replacement (apache#7131)

    、

commit 821dfc8
Author: lizhenglei <127465317+jackyyyyyssss@users.noreply.github.com>
Date:   Sat Jul 20 20:25:50 2024 +0800

    [Feature][Transform] DynamicCompile add transform  (apache#7170)

commit 1e7c78d
Author: hilo <1964736605@qq.com>
Date:   Fri Jul 19 19:44:18 2024 +0800

    [Improve][Doc] Add IMap OSS config jars into document (apache#7029)

commit d39ad93
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Fri Jul 19 19:42:03 2024 +0800

    [Fix][Zeta] Fix hybrid deployment can not get worker when init (apache#7235)

commit 2058f5d
Author: ChunFuWu <319355703@qq.com>
Date:   Fri Jul 19 14:21:23 2024 +0800

    [Chore][NOTICE] Correct year in NOTICE file (apache#7232)

commit 6e33a97
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Thu Jul 18 23:03:27 2024 +0800

    [Fix] Fix Hana type converter decimal scale is 0 convert to int error (apache#7167)

commit d1b4a7f
Author: Jia Fan <fanjiaeminem@qq.com>
Date:   Thu Jul 18 15:31:30 2024 +0800

    [Improve][Zeta] Add check for submit duplicate job id (apache#7021)

commit e44e8b9
Author: hailin0 <wanghailin@apache.org>
Date:   Thu Jul 18 15:30:49 2024 +0800

    [Improve][Jdbc] Support write unicode text into sqlserver (apache#7159)

commit f40f11a
Author: tcodehuber <tcodehuber@gmail.com>
Date:   Thu Jul 18 00:03:33 2024 +0800

    [Docs] Update engine related docs info (apache#7228)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hailin0 hailin0 hailin0 approved these changes

@Hisoka-X Hisoka-X Hisoka-X approved these changes

Assignees
No one assigned
Labels
approved connectors-v2 doris e2e reviewed
Projects
None yet
Milestone
2.3.7
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@liugddx @hailin0 @Hisoka-X