Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Transform-V2 Filter] support exclude columns in the filter transform #6960

Merged
merged 13 commits into from
Jun 13, 2024

Conversation

litiliu
Copy link
Contributor

@litiliu litiliu commented Jun 7, 2024

Purpose of this pull request

This close #6910

Does this PR introduce any user-facing change?

Yes.
The previous behavior of the Transform-V2 Filter is keeping the columns in the fields field. This PR adds an optional option named mode, the default value is KEEP, which keeps the columns in the fields field. If the mode is set to DELETE, it will delete the columns in the fields field. Ep:

Before this PR:

If The data read from source is a table like this:

name age card
Joy Ding 20 123
May Ding 20 123
Kin Dom 20 123
Joy Dom 20 123

If we want to keep the field named name, card, we can add a Filter Transform like below:

transform {
  Filter {
    source_table_name = "fake"
    result_table_name = "fake1"
    fields = [name, card]
  }
}

After this PR:

We can delete the field named age by adding a Filter Transform with the mode field set to DELETE like below:

transform {
  Filter {
    source_table_name = "fake"
    result_table_name = "fake1"
    fields = [age]
    mode = "DELETE"
  }
}

It will be useful when you want to delete a small number of fields from a large table with tons of fields.

How was this patch tested?

Added Addition UT cases.

Check list

@litiliu litiliu marked this pull request as ready for review June 7, 2024 08:04
@Hisoka-X
Copy link
Member

Hisoka-X commented Jun 7, 2024

cc @hailin0

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Only one small problem. Thanks @litiliu , I like this PR.

Copy link
Member

@EricJoy2048 EricJoy2048 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for you contribution.

Only some minor issues.

@litiliu litiliu requested a review from EricJoy2048 June 12, 2024 09:26
Co-authored-by: hailin0 <hailin088@gmail.com>
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. cc @hailin0

@EricJoy2048 EricJoy2048 merged commit 4ba6637 into apache:dev Jun 13, 2024
7 checks passed
litiliu added a commit to litiliu/seatunnel that referenced this pull request Jul 11, 2024
@litiliu litiliu deleted the feat-6910 branch July 11, 2024 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Transform-V2 Filter] support exclude columns in the filter transform
5 participants