[WIP] Implement BlockManager backend
#104
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
seyuboglu
commented
Jul 28, 2021
|
lgtm! |
krandiash
approved these changes
Jul 29, 2021
Codecov Report
@@ Coverage Diff @@
## dev #104 +/- ##
==========================================
+ Coverage 65.08% 73.74% +8.65%
==========================================
Files 58 51 -7
Lines 3340 3135 -205
Branches 590 532 -58
==========================================
+ Hits 2174 2312 +138
+ Misses 1012 671 -341
+ Partials 154 152 -2
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
|
closes #98 |
This was referenced Jul 30, 2021
seyuboglu
added a commit
that referenced
this pull request
Aug 9, 2021
Closes #109 The BlockManager (see #104) introduces a need for more robust DataPanel testing that tests DataPanels with a diverse set of columns. As we add more columns, we don't want to have to update the DataPanel tests for each new column. Instead, we should specify a TestBed for each column that plugs in to the DataPanel tests. Started this for NumpyArrayColumn with #108. Co-authored-by: Priya <priyamis@cse.iitk.ac.in>
krandiash
pushed a commit
that referenced
this pull request
Aug 14, 2021
Overhaul the internals of the Meerkat `DataPanel`. The changes seek to enable:
1. Vectorized row-wise operations (*e.g.* slicing, reduction)
2. Simplified I/O and improved latency
3. Clarified view vs. copy behavior
- We introduce a [new spec](https://www.notion.so/meerkat-working-doc-40d70d094ac0495684d3fd8ddc809343#2b9460b744a04cbca912c8d017e5887c) detailing when users should expect to get views vs. copies (similar to [this resource](https://numpy.org/doc/stable/reference/arrays.indexing.html) for NumPy) – I'm working on enforcing this spec throughout the codebase.
The new internals are based primarily off the `BlockManager` class, a dict-like object meant to replace the dictionary we were storing the DataPanel's columns in before. The `BlockManager` manages links between a DataPanel's columns and data blocks (`AbstractBlock`, `NumpyBlock`) where the data is actually stored. It implements `consolidate`, which takes columns of similar type in a DataPanel and stores their data together in a block, and `apply` which applies row-wise operations (e.g. __getitem__) to the blocks in a vectorized fashion. Other important classes:
- `BlockRef` objects link a block with the `BlockManager`. These are critical to the functioning of the BlockManager and are the primary type of object passed between the blocks and the block manager. They consists of two things:
1. A reference to the block (`self.block`)
2. A set of columns in the `BlockManager` whose data live in the `Block`
- `BlockableMixin` - a mixin used with `AbstractColumn` that holds references to a column's block and the columns index in the block
- `BlockView` - a simple DataClass holding a block and an index into the block. It is typical for new columns to be created from `BlockView`
Note: I marked this is a WIP because there are still a few more things to be done on this front.
1. Make `concat` BlockManager aware
Other major changes:
- Removed `visible_rows` from `AbstractColumn`,
- Removed `_cloneable_kwargs` in favor of a unified `_clone`, `_copy`, and `_view` module (cloneable.py)
krandiash
pushed a commit
that referenced
this pull request
Aug 14, 2021
Closes #109 The BlockManager (see #104) introduces a need for more robust DataPanel testing that tests DataPanels with a diverse set of columns. As we add more columns, we don't want to have to update the DataPanel tests for each new column. Instead, we should specify a TestBed for each column that plugs in to the DataPanel tests. Started this for NumpyArrayColumn with #108. Co-authored-by: Priya <priyamis@cse.iitk.ac.in>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overhaul the internals of the Meerkat
DataPanel. The changes seek to enable:The new internals are based primarily off the
BlockManagerclass, a dict-like object meant to replace the dictionary we were storing the DataPanel's columns in before. TheBlockManagermanages links between a DataPanel's columns and data blocks (AbstractBlock,NumpyBlock) where the data is actually stored. It implementsconsolidate, which takes columns of similar type in a DataPanel and stores their data together in a block, andapplywhich applies row-wise operations (e.g. getitem) to the blocks in a vectorized fashion. Other important classes:BlockRefobjects link a block with theBlockManager. These are critical to the functioning of the BlockManager and are the primary type of object passed between the blocks and the block manager. They consists of two things:self.block)BlockManagerwhose data live in theBlockBlockableMixin- a mixin used withAbstractColumnthat holds references to a column's block and the columns index in the blockBlockView- a simple DataClass holding a block and an index into the block. It is typical for new columns to be created fromBlockViewNote: I marked this is a WIP because there are still a few more things to be done on this front.
concatBlockManager awareOther major changes:
visible_rowsfromAbstractColumn,_cloneable_kwargsin favor of a unified_clone,_copy, and_viewmodule (cloneable.py)