Skip to content

Create projection plugin #2000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 17, 2021
Merged

Create projection plugin #2000

merged 3 commits into from
Dec 17, 2021

Conversation

6yozo
Copy link
Contributor

@6yozo 6yozo commented Dec 10, 2021

📔 Description

Projection: removes one or more columns

📝 Checklist

  • All user-facing changes have changelog entries.
  • The changes are reflected on docs.tenzir.com/vast, if necessary.
  • The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

Review this pull request commit-by-commit.

@6yozo 6yozo self-assigned this Dec 10, 2021
Copy link
Member

@mavam mavam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you also looked at the Arrow-native functions for projection?

I would be surprised if projection is not easily doable out of the box. There is a lot there: https://github.com/apache/arrow/search?l=C%2B%2B&q=project

@6yozo 6yozo force-pushed the story/sc-29694/projection branch 6 times, most recently from 9551897 to 242125e Compare December 14, 2021 13:53
@dominiklohmann dominiklohmann added blocked Blocked by an (external) issue feature New functionality labels Dec 14, 2021
@6yozo 6yozo force-pushed the story/sc-29694/projection branch from 242125e to ac4acfd Compare December 14, 2021 16:26
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial thoughts on the code. I've verified that the functionality works as expected for both Arrow- and MsgPack-encoded table slices.

@6yozo 6yozo force-pushed the story/sc-29694/projection branch 3 times, most recently from 91a970e to 4229c9d Compare December 15, 2021 12:08
@6yozo 6yozo force-pushed the story/sc-29694/projection branch from 519c720 to 440b53b Compare December 15, 2021 17:13
@6yozo 6yozo force-pushed the story/sc-29694/projection branch from 440b53b to 292a17c Compare December 15, 2021 19:42
@6yozo 6yozo marked this pull request as ready for review December 16, 2021 08:20
@6yozo 6yozo requested a review from dominiklohmann December 16, 2021 08:50
@dominiklohmann dominiklohmann removed the blocked Blocked by an (external) issue label Dec 16, 2021
Copy link
Member

@dominiklohmann dominiklohmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both delete and project work nicely for me locally. I just have some minor notes.

@6yozo 6yozo force-pushed the story/sc-29694/projection branch from 292a17c to a3b4862 Compare December 17, 2021 08:12
Gyozo Gaspar added 2 commits December 17, 2021 09:32
It calls arrow transform also when there is no generic transform, and the slice
is not arrow encoded. In this case, the slice is converted to arrow first.
@6yozo 6yozo force-pushed the story/sc-29694/projection branch from a3b4862 to a0b5812 Compare December 17, 2021 08:44
@6yozo 6yozo merged commit c8f0b7c into master Dec 17, 2021
@6yozo 6yozo deleted the story/sc-29694/projection branch December 17, 2021 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants