Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++][Parquet] Allow partial prebuffer in parquet FileReader #43005

Closed
asfimport opened this issue Jun 20, 2023 · 1 comment
Closed

[C++][Parquet] Allow partial prebuffer in parquet FileReader #43005

asfimport opened this issue Jun 20, 2023 · 1 comment

Comments

@asfimport
Copy link
Collaborator

asfimport commented Jun 20, 2023

The current FileReader can only work in  one of the two modes, coalescing (when Prebuffer is called) and non-coalescing (when Prefufer is not called), due to the if statement here

Since Prebuffer is basically caching all specified column chunks, it would raise concerns on memory usage for systems with tight memory budget. In such scenarios, one may want to Prebuffer some small chunks while being able to read the rest chunks using  BufferedInputStream. 

Reporter: Jinpeng Zhou / @jp0317
Assignee: Jinpeng Zhou / @jp0317

PRs and other links:

Note: This issue was originally created as PARQUET-2316. Please see the migration documentation for further details.

@asfimport
Copy link
Collaborator Author

Gang Wu / @wgtmac:
Issue resolved by pull request 36192
#36192

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant