Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset can read/load only first n rows of csv file. #467

Closed
sandeepsajan0 opened this issue Jun 4, 2020 · 2 comments
Closed

Dataset can read/load only first n rows of csv file. #467

sandeepsajan0 opened this issue Jun 4, 2020 · 2 comments
Labels

Comments

@sandeepsajan0
Copy link

sandeepsajan0 commented Jun 4, 2020

If we have a huge csv file and we can test our code only by first n rows then we shouldn't need to load the whole file. It'll be a great feature. I tried to find it but couldn't if it already exists there, can anyone of you guys tell me how can I use that feature?
Or we should implement that feature?

**Edited:
For now, I am just using the list comprehension with del method on loaded data to pass. But it'll be better if I can load only desired data.
What are your ideas?

@mynameisvinn
Copy link

@sandeepsajan0 the common solution is to read/parse csv file with a generator, so you can iterate over your csv lazily without loading the entire file into memory (and allowing you to peek the first n files).

@hugovk
Copy link
Member

hugovk commented Jan 10, 2022

Closing, as the solution is to iterate lazily using a generator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants