-
Notifications
You must be signed in to change notification settings - Fork 505
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[INLONG-6617][Sort] Add common process for dirty data sink and supports log sink #6618
Conversation
...rt/sort-connectors/base/src/test/java/org/apache/inlong/sort/base/dirty/FormatUtilsTest.java
Show resolved
Hide resolved
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
Show resolved
Hide resolved
...-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/DirtyOptions.java
Outdated
Show resolved
Hide resolved
Is this dirty sink added to integration tasks as one of the sinks in the sink list? Like whenever data is dirty during the process, write to the given sink...I feel that there is much work to do when adapting this dirty sink to individual adaptors and identifying exactly where errors would occur. What happens if errors occur during read/writes to the dirty sink? will adding this dirty sink (I want to say "side output" if i understood correctly) induce a high performance cost? |
...t-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/sink/log/LogDirtySink.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how can the connector be applied to the existed connector, can you describe it in the description ?
Maybe, add some decription in issue, how to use ? |
It has located in pr describe. |
It is necessary to create DirtySink through DirtyFactory in the connector, and then call DirtySink's invoke method where dirty data occurs. |
@yunqingmoswu Maybe, add a tool class like |
|
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/Constants.java
Outdated
Show resolved
Hide resolved
...sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/PatternReplaceUtils.java
Show resolved
Hide resolved
inlong-sort/sort-connectors/base/src/main/java/org/apache/inlong/sort/base/util/LabelUtils.java
Outdated
Show resolved
Hide resolved
...t-connectors/base/src/main/java/org/apache/inlong/sort/base/dirty/sink/log/LogDirtySink.java
Show resolved
Hide resolved
@yunqingmoswu please git rebase master |
done |
LGTM |
Prepare a Pull Request
(Change the title refer to the following example)
Title: [INLONG-6617][Sort] Add common process for dirty data sink and supports log sink
(The following XYZ should be replaced by the actual GitHub Issue number)
Motivation
Add common process for dirty data sink and supports log sink.
This piece is designed as follows:
Modifications
1.Add dirty process model
2.Add abstract for DirtySinkFactory, DirtySink,DirtyData,DirtyType
3.Add DirtyOptions as dirty common options
4.Add utils class for dirty sink such as LabelUtils, PatternReplaceUtils, and etc.
5.Add common option define, such as 'dirty.ignore',dirty.side-output.enable', 'dirty.side-output.connector', 'dirty.side-output.labels', 'dirty.side-output.format', 'dirty.side-output.log-tag', 'dirty.side-output.field-delimiter', and etc.
Verifying this change
(Please pick either of the following options)
This change is a trivial rework/code cleanup without any test coverage.
This change is already covered by existing tests, such as:
(please describe tests)
This change added tests and can be verified as follows:
(example:)
Documentation