[INLONG-7249][Sort] JDBC accurate dirty data archive and metric calculation#7580
Merged
dockerzhang merged 12 commits intoapache:masterfrom Apr 11, 2023
Merged
Conversation
gong
reviewed
Mar 13, 2023
...rs/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/TableMetricStatementExecutor.java
Outdated
Show resolved
Hide resolved
8bc1170 to
d3d7a36
Compare
added 2 commits
March 13, 2023 17:21
gong
reviewed
Mar 13, 2023
...ectors/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcBatchingOutputFormat.java
Show resolved
Hide resolved
...ectors/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcBatchingOutputFormat.java
Show resolved
Hide resolved
...ectors/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcBatchingOutputFormat.java
Outdated
Show resolved
Hide resolved
...ectors/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcBatchingOutputFormat.java
Outdated
Show resolved
Hide resolved
...s/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcMultiBatchingOutputFormat.java
Show resolved
Hide resolved
...s/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcMultiBatchingOutputFormat.java
Show resolved
Hide resolved
Yizhou-Yang
commented
Mar 17, 2023
...rs/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/TableMetricStatementExecutor.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
@gong @yunqingmoswu Thanks for the review. I've addressed your comments. |
added 2 commits
March 28, 2023 11:31
Contributor
|
@Yizhou-Yang pls resolve conflict |
gong
reviewed
Apr 4, 2023
...s/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcMultiBatchingOutputFormat.java
Outdated
Show resolved
Hide resolved
EMsnap
approved these changes
Apr 4, 2023
added 2 commits
April 4, 2023 14:19
healchow
reviewed
Apr 8, 2023
...s/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcMultiBatchingOutputFormat.java
Outdated
Show resolved
Hide resolved
healchow
reviewed
Apr 8, 2023
...s/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcMultiBatchingOutputFormat.java
Outdated
Show resolved
Hide resolved
healchow
reviewed
Apr 8, 2023
...s/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcMultiBatchingOutputFormat.java
Outdated
Show resolved
Hide resolved
healchow
reviewed
Apr 8, 2023
...s/jdbc/src/main/java/org/apache/inlong/sort/jdbc/internal/JdbcMultiBatchingOutputFormat.java
Outdated
Show resolved
Hide resolved
dockerzhang
reviewed
Apr 10, 2023
...t/sort-connectors/base/src/test/java/org/apache/inlong/sort/base/dirty/RegexReplaceTest.java
Outdated
Show resolved
Hide resolved
...t/sort-connectors/base/src/test/java/org/apache/inlong/sort/base/dirty/RegexReplaceTest.java
Outdated
Show resolved
Hide resolved
dockerzhang
approved these changes
Apr 10, 2023
healchow
approved these changes
Apr 10, 2023
yunqingmoswu
approved these changes
Apr 11, 2023
EMsnap
approved these changes
Apr 11, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prepare a Pull Request
Motivation
JDBC needs to archive dirty data. In this pr, I tried to explore to possibility of accurate dirty data archive for jdbc by modifying the executors. The modifications are well-tested and stable in most cases, and in other cases it will give a warning "jdbc enhance failed for class:{}", and won't affect normal code.
Modification
Support dirty data accurate archive by using reflection to replace some kinds of executors, and replace the tablesimplestatementexecutor with one that adds metrics.
To-do and to-improve
1.There are 8 total implementations of flink-cdc-executors.
The currently supported sink executor types are:
TableBufferedStatementExecutor
TableBufferReducedStatementExecutor
TableSimpleStatementExecutor
unsupported executor types: (will not archive dirty data, but will not break code)
KeyedBatchStatementExecutor
NoOPStatementExecutor
TableInsertOrUpdateStatementExecutor
no need to support:
InsertOrUpdateJdbcExecutor (TableInsertOrUpdateStatementExecutor is enough)
SimpleBatchStatementExecutor (TableSimpleStatementExecutor is enough)
2.This pr uses Java reflection depending on flink-cdc-connectors. This design can be improved by introducing the executors into Inlong-jdbc-connector, and modifying the builder to directly change the executor class, instead of doing all this reflection. Also, throughput can increase by refactoring too.
Even Though this pr still has imperfections, I think it can be merged for the following reasons: