Skip to content

[INLONG-6636][Sort] Keep metric computing consistent for source MySQL and sink HBase#6637

Merged
EMsnap merged 2 commits into
apache:masterfrom
gong:INLONG-6636
Nov 28, 2022
Merged

[INLONG-6636][Sort] Keep metric computing consistent for source MySQL and sink HBase#6637
EMsnap merged 2 commits into
apache:masterfrom
gong:INLONG-6636

Conversation

@gong

@gong gong commented Nov 25, 2022

Copy link
Copy Markdown
Contributor

Prepare a Pull Request

Motivation

It generates four RowKind data: insert, update_before, update_after, delete when source is mysql cdc.
We will count all RowKind data when computing metric of mysql cdc. HBase use upsert to write data and only need three RowKind data: insert, update_after, delete. Flink runtime will optimize code when sync mysql-cdc to HBase. Flink will use DropUpdateBeforeFunction#filter to drop update_before data. We need HBase computing four RowKind data to keep consistent with mysql-cdc. So we will change return value to all RowKind type of HBaseDynamicTableSink#getChangelogMode.

Modifications

  • Modify HBaseDynamicTableSink#getChangelogMode.
  • HBase skip update_before data .
  • MySQL cdc metric computing four Rowkind data.

@gong gong requested review from EMsnap and yunqingmoswu and removed request for EMsnap November 25, 2022 15:35
@EMsnap EMsnap merged commit 16ddf5f into apache:master Nov 28, 2022
featzhang pushed a commit to featzhang/inlong that referenced this pull request Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improve][Sort] Keep metric computing consistent for source MySQL and sink HBase

6 participants