-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[C++][JSON] kMaxParserNumRows Value Increase/Removal #28994
Comments
Hello, have any new update for this issue ? I have the same error. |
@bkietz Do you remember why we have |
This was copied from the CSV parser, which was my model when writing the JSON parser. The JSON parser doesn't have a use for this limit, I think. |
OK. Does someone want to work on this? |
Maybe can keep the |
The JSON parser should make this configurable as well. |
Hi guys, has some new information for the issue ? |
Any roadmap? |
I'm not aware of anyone getting ready to work on this issue. If you wanted to open a PR, I could give you a review |
### Rationale for this change Unlimited parse max rows for parse json block. Raise `Row count overflowed int32_t` error when the loop times out of `init_32::max()`. See issue: #28994 ### What changes are included in this PR? Delete const `kMaxParserNumRows`, Minor code( C++ ) modifications and test code(python) ### Are these changes tested? Yes New a test code for parse large (100100) rows json . ### Are there any user-facing changes? No * Closes: #28994 Lead-authored-by: zhipeng <zhipeng.py@gmail.com> Co-authored-by: zhipeng <5310853+Ox0400@users.noreply.github.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
…pache#38582) ### Rationale for this change Unlimited parse max rows for parse json block. Raise `Row count overflowed int32_t` error when the loop times out of `init_32::max()`. See issue: apache#28994 ### What changes are included in this PR? Delete const `kMaxParserNumRows`, Minor code( C++ ) modifications and test code(python) ### Are these changes tested? Yes New a test code for parse large (100100) rows json . ### Are there any user-facing changes? No * Closes: apache#28994 Lead-authored-by: zhipeng <zhipeng.py@gmail.com> Co-authored-by: zhipeng <5310853+Ox0400@users.noreply.github.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
I'm a new pyArrow user and have been investigating occasional errors related to the Python exception: "ArrowInvalid: Exceeded maximum rows" when parsing JSON line files using pyarrow.json.read_json(). In digging in, it looks like the original source of this exception is in cpp/src/arrow/json/parser.cc on line 703, which appears to throw the error when the number of lines processed exceeds kMaxParserNumRows.
This constant appears to be set in arrow/json/parser.h on line 53, and has been set this way since that file's initial commit.
There does not appear to be a comment in the code or in the commit or PR explaining this maximum number of lines.
I'm wondering what the reason for this maximum might be, and if it might be removed, increased, or made overridable in the C++ and the upstream Python. It is common to need to process JSON files of arbitrary length (logs from applications, third-party vendors, etc) where the user of the data does not have control over the size of the file.
Reporter: Ryan Stalets
Note: This issue was originally created as ARROW-13318. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: