Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage of ExportToFile #7

Merged
merged 5 commits into from Feb 24, 2016

Conversation

Projects
None yet
2 participants
@c-w
Copy link
Contributor

commented Feb 24, 2016

The current implementation of ExportToFile uses a lot of memory for large CSV files because it builds the entire document in memory. This pull request uses a streaming approach to reduce the memory overhead to a constant.

Testing:

  • Run the sample code before the changes and save the results to old.csv
  • Run the sample code after the changes and save the result to new.csv
  • Assert that there is no diff between old.csv and new.csv

c-w added some commits Feb 24, 2016

Reduce memory usage of ExportToFile
Using `File.WriteAllText` and `Export` builds the entire CSV string and
then writes it to disk. This essentially doubles the peak memory usage
of the class and is impractical for larger CSV documents.

Using `File.WriteAllLines` and `ExportToLines` on the other hand only
keeps a single row in memory at a time.

jitbit pushed a commit that referenced this pull request Feb 24, 2016

jitbit
Merge pull request #7 from c-w/master
Reduce memory usage of ExportToFile

@jitbit jitbit merged commit 70216ae into jitbit:master Feb 24, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.