-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Transform-V2 Filter] support exclude columns in the filter transform #6960
Conversation
|
cc @hailin0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Only one small problem. Thanks @litiliu , I like this PR.
...-transforms-v2/src/main/java/org/apache/seatunnel/transform/filter/FilterFieldTransform.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for you contribution.
Only some minor issues.
...-transforms-v2/src/main/java/org/apache/seatunnel/transform/filter/FilterFieldTransform.java
Outdated
Show resolved
Hide resolved
Co-authored-by: hailin0 <hailin088@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. cc @hailin0
Purpose of this pull request
This close #6910
Does this PR introduce any user-facing change?
Yes.
The previous behavior of the
Transform-V2 Filteris keeping the columns in thefieldsfield. This PR adds an optional option namedmode, the default value isKEEP, which keeps the columns in thefieldsfield. If themodeis set toDELETE, it will delete the columns in thefieldsfield. Ep:Before this PR:
If The data read from source is a table like this:
If we want to keep the field named
name,card, we can add aFilterTransform like below:After this PR:
We can delete the field named
ageby adding aFilterTransform with the mode field set toDELETElike below:It will be useful when you want to delete a small number of fields from a large table with tons of fields.
How was this patch tested?
Added Addition UT cases.
Check list
New License Guide
release-note.