Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
hdrs
raises a critical error when reading/writing a large file to hdfs. This is due to the length limit of a singlehdfsRead
call. I tested with a file of 40G, but I think any file that is larger than 2GB will face the same issue.The length parameter requires an i32, which is at most 2^31-1. Though I have tried that length size and only get an out of memory error. Thus I think it safer to limit the length to 2^30 bytes.
This patch is inspired by the rust underlying implementation of
Write
toFile
.After this patch, it's advisable to use
write_all
instead ofwrite
, andread_to_end
instead ofread
.