Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: read/write large file (>2GB) #83

Merged
merged 2 commits into from
Nov 30, 2022
Merged

fix: read/write large file (>2GB) #83

merged 2 commits into from
Nov 30, 2022

Conversation

fMeow
Copy link
Contributor

@fMeow fMeow commented Nov 30, 2022

hdrs raises a critical error when reading/writing a large file to hdfs. This is due to the length limit of a single hdfsRead call. I tested with a file of 40G, but I think any file that is larger than 2GB will face the same issue.

The length parameter requires an i32, which is at most 2^31-1. Though I have tried that length size and only get an out of memory error. Thus I think it safer to limit the length to 2^30 bytes.

This patch is inspired by the rust underlying implementation of Write to File.

After this patch, it's advisable to use write_all instead of write, and read_to_end instead of read.

src/file.rs Outdated Show resolved Hide resolved
@Xuanwo
Copy link
Owner

Xuanwo commented Nov 30, 2022

Thanks for your PR! Mostly LGTM, I'm glad to merge it after all comments addressed 💯

@Xuanwo Xuanwo merged commit f5b0449 into Xuanwo:main Nov 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants