DM-13876: Implement and register ParquetStorage as a butler storage type #90
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ktlim
reviewed
Apr 19, 2018
|
|
||
| The object returned by this is expected to be a subtype | ||
| of `ParquetTable`, which is a thin wrapper to `pyarrow.ParquetFile` | ||
| that allows for lazy loading of the data. This is notably |
There was a problem hiding this comment.
This comment is incorrect now, isn't it?
| def writeParquetStorage(butlerLocation, obj): | ||
| """Writes pandas dataframe to parquet file | ||
|
|
||
| Note, this takes a `pandas.DataFrame` to write, whereas |
There was a problem hiding this comment.
And this comment; I think it expects a ParquetTable as well.
|
|
||
| Returns | ||
| ------- | ||
| A list of objects as described by the butler location. One item for |
There was a problem hiding this comment.
The original purpose of the list of locations was to allow a single dataset to be assembled from multiple physical serializations. If you're not actually expecting/using the list, I would just use the initial entry (and perhaps even warn if there is more than one).
ktlim
reviewed
Apr 19, 2018
| @@ -288,10 +288,12 @@ def read(self, butlerLocation): | |||
| raise(RuntimeError("No formatter for location:{}".format(butlerLocation))) | |||
|
|
|||
| def butlerLocationExists(self, location): | |||
| """Implementaion of PosixStorage.exists for ButlerLocation objects.""" | |||
| """Implementaion of PosixStorage.exists for ButlerLocation objects. | |||
There was a problem hiding this comment.
Could you fix the typo here while you're in here?
This defines readParquetStorage and writeParquetStorage to define butler get/put operations for ParquetStorage datatypes. They use the wrapper object ParquetTable around pyarrow read/write operations. Read returns a lazy object; meaning no data is loaded, but arbitrary columns can be loaded later using the .to_df method of the ParquetTable.
timothydmorton
force-pushed
the
tickets/DM-13876
branch
from
c6a3e2b
to
bf5ba42
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This defines readParquetStorage and writeParquetStorage to define
butler get/put operations for ParquetStorage datatypes. They use
the wrapper object ParquetTable around pyarrow read/write operations.
Read returns a lazy object; meaning no data is loaded, but arbitrary
columns can be loaded later using the .to_df method of the ParquetTable.