ARROW-17515: [C++][Archery] JSON integration testing with RLE #14179
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
the scalar version visits all types that can exist as a Scalar. Currently this true for all types we have. This will change once we add run-length encoding, which is an array encoding.
- use int32 - calculate physical offset based on buffer size, instead of incorrectly using the physical size
works in archery round trip so far
zeroshade
reviewed
Sep 23, 2022
Comment on lines
+529
to
+535
| Status Visit(const RunLengthEncodedArray& array) { | ||
| --max_recursion_depth_; | ||
| RETURN_NOT_OK(VisitArray(*array.run_ends_array())); | ||
| RETURN_NOT_OK(VisitArray(*array.values_array())); | ||
| ++max_recursion_depth_; | ||
| return Status::OK(); | ||
| } |
There was a problem hiding this comment.
How are we handling the logical offset with the IPC?
There was a problem hiding this comment.
nvm, i see the usage of logical_run_ends and logical_values, but shouldn't this be using those instead?
zeroshade
reviewed
Sep 23, 2022
| run_ends = self.run_ends_field.generate_column(size) | ||
| if name is None: | ||
| name = self.name | ||
| return RunLengthEncodedColumn(name, size, run_ends, values) |
There was a problem hiding this comment.
shouldn't the size here be larger than the size used in generate_column since it would be the "logical" length for the parent?
alamb
reviewed
Dec 16, 2022
| #include "arrow/array/array_base.h" // IWYU pragma: keep | ||
| #include "arrow/array/array_binary.h" // IWYU pragma: keep | ||
| #include "arrow/array/array_decimal.h" // IWYU pragma: keep | ||
| #include "arrow/array/array_dict.h" // IWYU pragma: keep | ||
| #include "arrow/array/array_encoded.h" // IWYU pragma: keep |
There was a problem hiding this comment.
Maybe array_rle or array_ree would be a better name for RunEndEncoded arrays
DictionaryArrays are also a form of encoding so calling this array_encoded seems a bit vague
rtpsw
pushed a commit
to rtpsw/arrow
that referenced
this pull request
Mar 27, 2023
…che#34550) * Closes apache#14340 * Closes apache#32773 * Closes apache#14179 * Closes: apache#32338 Lead-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com> Co-authored-by: Tobias Zagorni <tobias@zagorni.eu> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.