Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimal tests for memory usage contraints in partial-column read APIs #15371

Open
2 of 5 tasks
itamarst opened this issue Mar 28, 2024 · 1 comment
Open
2 of 5 tasks
Assignees
Labels
A-io Area: reading and writing data accepted Ready for implementation internal An internal refactor or improvement test Related to the test suite

Comments

@itamarst
Copy link
Contributor

itamarst commented Mar 28, 2024

Description

As follow-up to #15231, add memory usage tests for other APIs that allow reading a subset of columns from file, to ensure similar bugs to #15098 don't exist now or happen in the future.

I omitted Excel and ODS because those file types won't be that big in practice, and read_json/read_ndjson because you can't currently choose a subset of columns, so you're reading everything. Might still be optimizations one can do but this is more about #15098-style bugs where you load everything even though the user requested a single column.

@itamarst itamarst added the enhancement New feature or an improvement of an existing feature label Mar 28, 2024
@itamarst
Copy link
Contributor Author

Going to start on this.

@stinodego stinodego added test Related to the test suite A-io Area: reading and writing data internal An internal refactor or improvement accepted Ready for implementation and removed enhancement New feature or an improvement of an existing feature labels Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-io Area: reading and writing data accepted Ready for implementation internal An internal refactor or improvement test Related to the test suite
Projects
Status: Ready
Development

No branches or pull requests

2 participants