Big refactoring to simplify code and implement more parts of the spec #26
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In particular this PR: * Changes the flow of writing so that input columns are converted to their "arrow" equivalent, most of the time lazily, so that the actual writing is much simpler and just deals with each array type * Introduces an ArrowTypes module that defines the ArrowType trait, which types can overload to signal what kind of arrow type they should be converted to. This isn't super fleshed out quite yet as we need to figure out the acutal requirements for the different arrow types, but it's a start * Support reading/writing compressed arrow buffers automatically (reading) and via keyword arg (writing, `compress=:lz4` or `compress=:zstd` * support nested dict encoding (fixes #15) * fixes #24; not exactly sure what the issue was, but it doesn't happen on this branch * reorganizes tests so that for each kind of "table", we do IPC test, compressed buffers test, and file test
Codecov Report
@@ Coverage Diff @@
## master #26 +/- ##
==========================================
+ Coverage 79.01% 80.64% +1.63%
==========================================
Files 13 14 +1
Lines 2063 2557 +494
==========================================
+ Hits 1630 2062 +432
- Misses 433 495 +62
Continue to review full report at Codecov.
|
This was referenced Oct 1, 2020
quinnj
added a commit
that referenced
this pull request
Oct 3, 2020
…#26) * Big refactoring to simplify code and implement more parts of the spec In particular this PR: * Changes the flow of writing so that input columns are converted to their "arrow" equivalent, most of the time lazily, so that the actual writing is much simpler and just deals with each array type * Introduces an ArrowTypes module that defines the ArrowType trait, which types can overload to signal what kind of arrow type they should be converted to. This isn't super fleshed out quite yet as we need to figure out the acutal requirements for the different arrow types, but it's a start * Support reading/writing compressed arrow buffers automatically (reading) and via keyword arg (writing, `compress=:lz4` or `compress=:zstd` * support nested dict encoding (fixes #15) * fixes #24; not exactly sure what the issue was, but it doesn't happen on this branch * reorganizes tests so that for each kind of "table", we do IPC test, compressed buffers test, and file test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In particular this PR:
their "arrow" equivalent, most of the time lazily, so that the actual
writing is much simpler and just deals with each array type
which types can overload to signal what kind of arrow type they should
be converted to. This isn't super fleshed out quite yet as we need to
figure out the acutal requirements for the different arrow types, but
it's a start
(reading) and via keyword arg (writing,
compress=:lz4orcompress=:zstdhappen on this branch
compressed buffers test, and file test
Char/Symbol; previously they were only allowed as top-level columns, but now they can appear in NamedTuple, vector of vectors, etc.Maptype; I mistakenly thoughtMapmeant that each element of the column was aPair, but it actually means each element is aDict. So it's more similar to theListtype thanStruct; this fixes the reading/writing for this type.unsafe_wraped bytes; these bytes may be from the original arrow blob or an uncompressed buffer