-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Data] Add read_json
docs example for setting PyArrow block size when reading large files
#40533
Conversation
Signed-off-by: Scott Lee <sjl@anyscale.com>
python/ray/data/read_api.py
Outdated
When reading large files, the default block size configured in PyArrow can be too small, | ||
resulting in the following error: | ||
``pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries | ||
(try to increase block size?)`` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(try to increase block size?)`` | |
(try to increase block size?)``. |
python/ray/data/read_api.py
Outdated
>>> ray.data.read_json(..., | ||
... read_options=pajson.ReadOptions(block_size=block_size)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
>>> ray.data.read_json(..., | |
... read_options=pajson.ReadOptions(block_size=block_size)) | |
>>> ray.data.read_json( | |
... s3://anonymous@ray-example-data/log.json, | |
... read_options=pajson.ReadOptions(block_size=block_size) | |
... ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you wanted to, you could add a large-log.json
to make this example more realistic, but I don't think that's necessary
python/ray/data/read_api.py
Outdated
>>> import pyarrow.json as pajson | ||
>>> block_size = 10 << 20 # Set block size to 10MB | ||
>>> ray.data.read_json( | ||
... s3://anonymous@ray-example-data/log.json, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missing the ""
- "s3://anonymous@ray-example-data/log.json"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the catch, just fixed
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
…into 1020-json-blocksize
…en reading large files (ray-project#40533) Adds an example for a workaround when reading large files with ray.data.read_json, which involves setting the block size used by PyArrow's JSON loader. Generated docs page --------- Signed-off-by: Scott Lee <sjl@anyscale.com>
Why are these changes needed?
Adds an example for a workaround when reading large files with
ray.data.read_json
, which involves setting the block size used by PyArrow's JSON loader. Generated docs pageRelated issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.