Skip to content
This repository has been archived by the owner on Nov 27, 2019. It is now read-only.

Do not grow memory without bound #34

Open
drj11 opened this issue Sep 26, 2013 · 3 comments
Open

Do not grow memory without bound #34

drj11 opened this issue Sep 26, 2013 · 3 comments
Labels

Comments

@drj11
Copy link
Contributor

drj11 commented Sep 26, 2013

extract.py seems to grow without bound when doing its stuff (observe with htop).

Not only is this bad form generally, but on free (which is limited to 512MB) it causes the process to be abruptly killed mysteriously. (which leaves lots of files behind: issue #31)

@pwaller
Copy link
Contributor

pwaller commented Sep 26, 2013

Good luck with that.

For a 21mb all_tables.xls it has a maximum resident set of 1.8GB.

@zarino zarino closed this as completed Nov 5, 2013
@pwaller pwaller reopened this Nov 5, 2013
@pwaller
Copy link
Contributor

pwaller commented Nov 5, 2013

This one is still an issue since writes aren't streaming yet.

@pwaller
Copy link
Contributor

pwaller commented Nov 19, 2013

I've just tried xlsxwriter. In the {'constant_memory': True} mode it uses virtually no memory but a lot of CPU time. On my laptop around 14 seconds for ~100krow, compared to "almost nothing" for CSV. I suggest we put in a hard limit of 100krow anyway and use pyexcelerate since it is considerably faster. Currently testing the performance of these in a few scenarios, PR incoming soon.

@drj11 @morty

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants