New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-6762: [C++] Support reading JSON files with no newline at end #5564
ARROW-6762: [C++] Support reading JSON files with no newline at end #5564
Conversation
a3498d7
to
71c05d8
Compare
@ursabot build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems fine, just a few nits.
I don't think there is a lifetime issue if ChunkedArrayBuilder::Finish()
is called before the builder is destroyed. It calls TaskGroup::Finish()
, so all tasks in any other threads with a reference to the builder will terminate. This is the case in TableReader::Read
, for example. Is there some use case for running a ChunkedArrayBuilder
in a task group but never calling finish on it?
I had some actual crashes when adding the tests. The main situation is when the reader exits early - for example because chunking fails, but I think it may also happen if readahead fails (think I/O or decompression error). |
Also fix some lifetime issues in parallel mode, and add tests.
71c05d8
to
562783d
Compare
Also fix some lifetime issues in parallel mode, and add tests. Closes #5564 from pitrou/ARROW-6762-json-parser-trailing-newline and squashes the following commits: 562783d <Antoine Pitrou> ARROW-6762: Support reading JSON files with no newline at end Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
Also fix some lifetime issues in parallel mode, and add tests. Closes apache#5564 from pitrou/ARROW-6762-json-parser-trailing-newline and squashes the following commits: 562783d <Antoine Pitrou> ARROW-6762: Support reading JSON files with no newline at end Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
Also fix some lifetime issues in parallel mode, and add tests. Closes apache#5564 from pitrou/ARROW-6762-json-parser-trailing-newline and squashes the following commits: 562783d <Antoine Pitrou> ARROW-6762: Support reading JSON files with no newline at end Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
Also fix some lifetime issues in parallel mode, and add tests. Closes apache#5564 from pitrou/ARROW-6762-json-parser-trailing-newline and squashes the following commits: 562783d <Antoine Pitrou> ARROW-6762: Support reading JSON files with no newline at end Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
Also fix some lifetime issues in parallel mode, and add tests.