[INLONG-6636][Sort] Keep metric computing consistent for source MySQL and sink HBase#6637
Merged
Conversation
healchow
approved these changes
Nov 26, 2022
woofyzhao
approved these changes
Nov 28, 2022
EMsnap
approved these changes
Nov 28, 2022
yunqingmoswu
approved these changes
Nov 28, 2022
thesumery
approved these changes
Nov 28, 2022
featzhang
pushed a commit
to featzhang/inlong
that referenced
this pull request
Nov 28, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prepare a Pull Request
[INLONG-6636][Sort] Keep metric computing consistent for source MySQL and sink HBase
Fixes [Improve][Sort] Keep metric computing consistent for source MySQL and sink HBase #6636
Motivation
It generates four
RowKinddata: insert, update_before, update_after, delete when source is mysql cdc.We will count all
RowKinddata when computing metric of mysql cdc. HBase use upsert to write data and only need threeRowKinddata: insert, update_after, delete. Flink runtime will optimize code when sync mysql-cdc to HBase. Flink will useDropUpdateBeforeFunction#filterto dropupdate_beforedata. We need HBase computing fourRowKinddata to keep consistent with mysql-cdc. So we will change return value to allRowKindtype ofHBaseDynamicTableSink#getChangelogMode.Modifications
update_beforedata .Rowkinddata.