Cherry pick bugfix commits for Release 2.4.2 by GOODBOY008 · Pull Request #2591 · apache/flink-cdc

GOODBOY008 · 2023-10-30T03:41:20Z

Cherry pick bugfix commits for Release 2.4.2.

GOODBOY008 · 2023-10-30T03:44:39Z

@Jiabao-Sun Becasuse the first commit depends on the under three commits, so I add the under three commit to 2.4.2.

leonardBang

Thanks @GOODBOY008 for the great work, I left some comments about API change commits, I think we should only introduce the bugfix commits for bugfix version.

leonardBang · 2023-10-30T05:20:37Z

+ * produces full changelog mode structure {@link RowData}.
+ */
+@PublicEvolving
+public class MongoDBConnectorFullChangelogDeserializationSchema


This is a feature with public API change, I'm not sure we can pick this or not, I tend to skip this one

Thanks @leonardBang, it makes sense to me.
The supports of MongoDB 6.0 will indeed affect the API, but it will maintain compatibility with the old version under the default configuration. I tend to skip this one as well.

cc @GOODBOY008.

leonardBang · 2023-10-30T05:22:15Z

+     * <p>See more
+     * https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished.
+     */
+    public PostgresSourceBuilder<T> closeIdleReaders(boolean closeIdleReaders) {


This should be a API change feature?

There is no API change, here is something omitted here at 2.3

leonardBang · 2023-10-30T05:24:10Z

-        /** VTCtld server config. */
-        public Builder<T> vtctldConfig(VtctldConfig vtctldConfig) {
-            this.vtctldConfig = vtctldConfig;
+        /**


Looks like we pick some features

These changes belong to [vitess] Migrate connector properties to debezium 1.9.7 (#2260). It's ok to merge this commit.

I understand your concern and drop this commit.

GOODBOY008 · 2023-10-30T07:05:48Z

@Jiabao-Sun Becasuse the first commit depends on the under three commits, so I add the under three commit to 2.4.2.

@leonardBang About mongodb hotfix, I added under three commits because upper bugfix depend on them. I confirmed with @Jiabao-Sun , he think it's ok to add this three commits.

leonardBang · 2023-10-30T07:12:37Z

@GOODBOY008 @Jiabao-Sun Keep API compatible is pretty important, we need ensure that all users who use Flink CDC 2.4.0& 2.4.1 can bump to Flink CDC 2.4.2 without any SQL/code change, I'm not sure this PR would break this constraint or not, looks like this PR would break this?

…xx = database, schema or table) (apache#2212) (cherry picked from commit ba5fcbc)

(cherry picked from commit 0714688)

This closes apache#2334. (cherry picked from commit 883b684)

…m for Oracle CDC Connector This closes apache#2325. Co-authored-by: skylines <34996528+gtk96@users.noreply.github.com> (cherry picked from commit 65bdbfd)

(cherry picked from commit 6c2fe08)

(cherry picked from commit bca60a4)

(cherry picked from commit 0b9ca59)

This closes apache#2345. (cherry picked from commit 61b9a2a)

…ailedSource (apache#2384) * [hotfix][mysql] fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource * [hotfix][mysql] fix failed test LegacyMySqlSourceTest#testSnapshotOnFailedSource (cherry picked from commit b8f468e)

This closes #apache#2362. Co-authored-by: TJX2014 <tangjinxin@deepexi.com> (cherry picked from commit 094fff4)

…pleTest#testConsumingAllEvents (apache#2364) (cherry picked from commit a3ddc48)

apache#2388) Co-authored-by: DESKTOP-TQDONF3\狗狗jian <zhangjian@adream.org> (cherry picked from commit f7df47e)

…ner reuse bug (apache#2394) (cherry picked from commit 90ed963)

…tikv disk full` failure (apache#2402) (cherry picked from commit 8098c82)

… tables (apache#2160) (cherry picked from commit 92aca18)

(cherry picked from commit 420fe11)

… parsing MySQL DDL (apache#2418) (cherry picked from commit 8e7378d)

* [sqlserver] Fix backfill stream task hang * [sqlserver] Fix backfill stream task hang (cherry picked from commit 6fa02ef)

…#2369) Co-authored-by: pgm-rookie <669768751@qq.com> (cherry picked from commit 54df7e9)

* [oracle] Correct the naming error * Fix comments * fix name error (cherry picked from commit 8feb51b)

…able name with dot (apache#2443) (cherry picked from commit 7801ac9)

…pache#2399) (cherry picked from commit 84e97df)

… calculation This closes apache#2421. Co-authored-by: Leonard Xu <leonard@apache.org> (cherry picked from commit dddab4d)

(cherry picked from commit 07db169)

(cherry picked from commit 7172695)

…stream split (apache#2425) (cherry picked from commit 82e3053)

(cherry picked from commit 928ccf1)

GOODBOY008 · 2023-10-30T10:09:50Z

@Jiabao-Sun I just pick the two commits, PTAL.

) (cherry picked from commit aec2a91)

lvyanquan · 2023-10-31T02:40:25Z

LGTM, +1

leonardBang · 2023-10-31T07:40:17Z

The CI failed even I triggered retry, @GOODBOY008 @lvyanquan could you take a look ?

GOODBOY008 · 2023-10-31T07:47:27Z

The CI failed even I triggered retry, @GOODBOY008 @lvyanquan could you take a look ?

Yes,I am trying to find out reason.

(cherry picked from commit 2f79b24)

(cherry picked from commit 6f9a064)

leonardBang

Thank all for this PR, especially thanks @GOODBOY008 for preparing the PR and
troubleshooting the CI failure, LGTM

GOODBOY008 requested review from Jiabao-Sun and leonardBang October 30, 2023 03:47

leonardBang reviewed Oct 30, 2023

View reviewed changes

GOODBOY008 force-pushed the release-2.4 branch from d239f72 to 473e749 Compare October 30, 2023 07:17

senlizishi and others added 23 commits October 30, 2023 15:39

[hotfix] update dbz option name 'xx. whitelist' to 'xx.include.list'(…

8352936

…xx = database, schema or table) (apache#2212) (cherry picked from commit ba5fcbc)

[build] Add vcs.xml for IDEA (apache#2064)

a4ba02a

(cherry picked from commit 0714688)

[mongodb] [tests] Loosen E2E timeout limitations

b1b0825

This closes apache#2334. (cherry picked from commit 883b684)

[docs][oracle] Add an example of Incremental Snapshot based DataStrea…

f6f13c1

…m for Oracle CDC Connector This closes apache#2325. Co-authored-by: skylines <34996528+gtk96@users.noreply.github.com> (cherry picked from commit 65bdbfd)

[doc] fix errors in PostgresParallelSourceExample (apache#2324)

60483e8

(cherry picked from commit 6c2fe08)

[polish] Add type parameter for MySqlRecords(apache#2300)

4ea4aa0

(cherry picked from commit bca60a4)

[mongodb][hotfix] Fix SourceRecord range check (apache#2343)

980f1b1

(cherry picked from commit 0b9ca59)

[db2] [tests] Fix unstable Db2ConnectorITCase due to cleanup failure

6855aec

This closes apache#2345. (cherry picked from commit 61b9a2a)

[hotfix][mysql] Catch underlying throwable in snapshot reading.

9f4e65f

This closes #apache#2362. Co-authored-by: TJX2014 <tangjinxin@deepexi.com> (cherry picked from commit 094fff4)

[test][mysql] Provide the server-time-zone setting in MySqlSourceExam…

76f9f39

…pleTest#testConsumingAllEvents (apache#2364) (cherry picked from commit a3ddc48)

[mysql] quota the field names when executing mysql query (apache#2381) (

2476638

apache#2388) Co-authored-by: DESKTOP-TQDONF3\狗狗jian <zhangjian@adream.org> (cherry picked from commit f7df47e)

[test][postgres] Close postgres containers after tests and fix contai…

70a2bec

…ner reuse bug (apache#2394) (cherry picked from commit 90ed963)

[test] Fix TiDBE2eITCase by updating the container version to avoid `…

cf390f9

…tikv disk full` failure (apache#2402) (cherry picked from commit 8098c82)

[mysql] Filter databases that do not need to be read when discovering…

0c2cbd0

… tables (apache#2160) (cherry picked from commit 92aca18)

[Doc] Use redo log instead of binlog in Oracle document (apache#2408)

06a521e

(cherry picked from commit 420fe11)

[hotfix][mysql] Trim default value for BIGINT and SMALLINT types when…

dbd33e6

… parsing MySQL DDL (apache#2418) (cherry picked from commit 8e7378d)

[hotfix][sqlserver] Fix backfill stream task hang (apache#2374)

a6e4380

* [sqlserver] Fix backfill stream task hang * [sqlserver] Fix backfill stream task hang (cherry picked from commit 6fa02ef)

[sqlserver] Add table filter to speed up SqlServerSchema read (apache…

0fe1cbd

…#2369) Co-authored-by: pgm-rookie <669768751@qq.com> (cherry picked from commit 54df7e9)

[oracle] Correct the naming error (apache#2405)

29816f9

* [oracle] Correct the naming error * Fix comments * fix name error (cherry picked from commit 8feb51b)

[cdc-base] Fix parsing error when serializing and deserializing the t…

b6d87d4

…able name with dot (apache#2443) (cherry picked from commit 7801ac9)

[mysql] Add finished unack splits to state for the MysqlSourceReader (a…

7d5e197

…pache#2399) (cherry picked from commit 84e97df)

[hotfix][mysql] Keep assigned splits in order to fix wrong meta group…

f6173f2

… calculation This closes apache#2421. Co-authored-by: Leonard Xu <leonard@apache.org> (cherry picked from commit dddab4d)

wuzhenhua01 and others added 4 commits October 30, 2023 15:40

[postgres] Close idle readers when snapshot finished (apache#2400)

cdc5990

(cherry picked from commit 07db169)

[cdc-base] Fix TM hangs caused by uncaught exception (apache#2511)

929f853

(cherry picked from commit 7172695)

[postgres] Close jdbc connection after creating replication slot for …

039b2ab

…stream split (apache#2425) (cherry picked from commit 82e3053)

[postgres] Not drop replication slot for stream split (apache#2436)

31b7b79

(cherry picked from commit 928ccf1)

GOODBOY008 force-pushed the release-2.4 branch from 473e749 to 31b7b79 Compare October 30, 2023 07:41

[e2e] Remove db docker image after test cases are executed. (apache#2508

92e642f

) (cherry picked from commit aec2a91)

GOODBOY008 force-pushed the release-2.4 branch from e850811 to 92e642f Compare October 30, 2023 10:28

mergify Bot mentioned this pull request Oct 30, 2023

[TEST] [WIP] reproduce oracle issue #2167

Closed

GOODBOY008 added 3 commits October 31, 2023 16:57

[hotfix] Fix vitess docker image tag

e02cac6

[sqlserver] Add support bigint type as table split column (apache#2416)

0ba8626

(cherry picked from commit 2f79b24)

[sqlserver] Fix get latest lsn (apache#2551)

e8fca81

(cherry picked from commit 6f9a064)

leonardBang approved these changes Oct 31, 2023

View reviewed changes

leonardBang merged commit 80cd840 into apache:release-2.4 Oct 31, 2023

GOODBOY008 deleted the release-2.4 branch January 11, 2024 15:58

Conversation

GOODBOY008 commented Oct 30, 2023

Uh oh!

GOODBOY008 commented Oct 30, 2023

Uh oh!

leonardBang left a comment

Choose a reason for hiding this comment

Uh oh!

leonardBang Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

Jiabao-Sun Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

Jiabao-Sun Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

leonardBang Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

Jiabao-Sun Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

leonardBang Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

GOODBOY008 Oct 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GOODBOY008 Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

GOODBOY008 commented Oct 30, 2023

Uh oh!

leonardBang commented Oct 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GOODBOY008 commented Oct 30, 2023

Uh oh!

lvyanquan commented Oct 31, 2023

Uh oh!

leonardBang commented Oct 31, 2023

Uh oh!

GOODBOY008 commented Oct 31, 2023

Uh oh!

leonardBang left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

GOODBOY008 Oct 30, 2023 •

edited

Loading

leonardBang commented Oct 30, 2023 •

edited

Loading