feat: Support projection #192
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
evenyag
force-pushed
the
feat/storage-projection
branch
from
0f3ec71
to
9a48944
Compare
The btree memtable use `is_needed()` to filter unneeded value columns, then use `ProjectedSchema::batch_from_parts()` to construct batch, so it don't need to known the layout of internal columns.
Also returns error if the `projected_columns` used to build the `ProjectedSchema` is empty.
This fix the issue that the metadata refer to the wrong timestamp column if datafusion reorder the fields of the arrow schema.
evenyag
force-pushed
the
feat/storage-projection
branch
from
899cc9d
to
9957e2d
Compare
Codecov Report
@@ Coverage Diff @@
## develop #192 +/- ##
===========================================
+ Coverage 75.45% 75.91% +0.46%
===========================================
Files 237 245 +8
Lines 18670 19650 +980
===========================================
+ Hits 14088 14918 +830
- Misses 4582 4732 +150
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
v0y4g3r
reviewed
Aug 25, 2022
Also fix some typo
clickme-zxy
pushed a commit
that referenced
this pull request
Aug 26, 2022
* feat: Add projected schema * feat: Use projected schema to read sst * feat: Use vector of column to implement Batch * feat: Use projected schema to convert batch to chunk * feat: Add no_projection() to build ProjectedSchema * feat: Memtable supports projection The btree memtable use `is_needed()` to filter unneeded value columns, then use `ProjectedSchema::batch_from_parts()` to construct batch, so it don't need to known the layout of internal columns. * test: Add tests for ProjectedSchema * test: Add tests for ProjectedSchema Also returns error if the `projected_columns` used to build the `ProjectedSchema` is empty. * test: Add test for memtable projection * feat: Table pass projection to storage engine * fix: Use timestamp column name as schema metadata This fix the issue that the metadata refer to the wrong timestamp column if datafusion reorder the fields of the arrow schema. * fix: Fix projected schema not passed to memtable * feat: Add tests for region projection * chore: fix clippy * test: Add test for unordered projection * chore: Move projected_schema to ReadOptions Also fix some typo
paomian
pushed a commit
to paomian/greptimedb
that referenced
this pull request
Oct 19, 2023
* feat: Add projected schema * feat: Use projected schema to read sst * feat: Use vector of column to implement Batch * feat: Use projected schema to convert batch to chunk * feat: Add no_projection() to build ProjectedSchema * feat: Memtable supports projection The btree memtable use `is_needed()` to filter unneeded value columns, then use `ProjectedSchema::batch_from_parts()` to construct batch, so it don't need to known the layout of internal columns. * test: Add tests for ProjectedSchema * test: Add tests for ProjectedSchema Also returns error if the `projected_columns` used to build the `ProjectedSchema` is empty. * test: Add test for memtable projection * feat: Table pass projection to storage engine * fix: Use timestamp column name as schema metadata This fix the issue that the metadata refer to the wrong timestamp column if datafusion reorder the fields of the arrow schema. * fix: Fix projected schema not passed to memtable * feat: Add tests for region projection * chore: fix clippy * test: Add test for unordered projection * chore: Move projected_schema to ReadOptions Also fix some typo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
ProjectedSchemato represent a Schema with projectionProjectedSchemamaintains two schema, one for reading data from memtables and SSTs, which would reads all row key columns and internal columns, and one for the schema that user expects to see after projectionProjectedSchemaas read options to push down projection and adjust outputBatchschema to the projected schemaBatchinternal representation, now it holds all columns in a singleVecRelated Issues
Unresolved Problems
SstSchemais not only used by SST module now, it also represent the schema with internal columns, soInternalSchemaorStoreSchemamay be more proper. I would rename it in another PR