Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage of ExportToFile #7

merged 5 commits into from Feb 24, 2016

Reduce memory usage of ExportToFile #7

merged 5 commits into from Feb 24, 2016


Copy link

@c-w c-w commented Feb 24, 2016

The current implementation of ExportToFile uses a lot of memory for large CSV files because it builds the entire document in memory. This pull request uses a streaming approach to reduce the memory overhead to a constant.


  • Run the sample code before the changes and save the results to old.csv
  • Run the sample code after the changes and save the result to new.csv
  • Assert that there is no diff between old.csv and new.csv
c-w added 5 commits Feb 24, 2016
Using `File.WriteAllText` and `Export` builds the entire CSV string and
then writes it to disk. This essentially doubles the peak memory usage
of the class and is impractical for larger CSV documents.

Using `File.WriteAllLines` and `ExportToLines` on the other hand only
keeps a single row in memory at a time.
jitbit pushed a commit that referenced this issue Feb 24, 2016
Reduce memory usage of ExportToFile
@jitbit jitbit merged commit 70216ae into jitbit:master Feb 24, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants