New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lower memory usage during index file parsing #2396 #2826

Merged
merged 4 commits into from Sep 18, 2015

Conversation

Projects
None yet
3 participants
@sznurek
Copy link
Contributor

sznurek commented Sep 17, 2015

Fixes #2396

As stated in one of the commit messages:

The parsePackageIndex wasn't streaming because resulting list was
hidden in two constructors: Either and tuple. By removing that, error
semantics changed in updatePackageIndexCacheFile.

The updatePackageIndexCacheFile problem was sequence call that evaluated
whole list produced by parsePackageIndex. By using lazySequence, any error
during index file parsing will be raised during writing to cache file.

Also removed readPackageIndexFile, as it wasn't used anywhere as far as
I can tell. If this is a mistake I'll include it again.

sznurek added some commits Jun 17, 2015

Made parsePackageIndex and updatePackageIndexCacheFile streaming.
The parsePackageIndex wasn't streaming because resulting list was
hidden in two constructors: Either and tuple. By removing that error
semantics changed in updatePackageIndexCacheFile.

The updatePackageIndexCacheFile problem was sequence call that evaluated
whole list made by parsePackageIndex. By using lazySequence any error
during index file parsing will be raised during writing to cache file.

Also removed reacPackageIndexFile, as it wasn't used anywhere as far as
I can tell. If this is a mistake I'll include it again.
@ttuegel

This comment has been minimized.

Copy link
Member

ttuegel commented Sep 18, 2015

I can confirm this does improve memory usage significantly!

Before:

ttuegel ~$ rm -fr .cabal/packages/hackage.haskell.org                            
ttuegel ~$ command time cabal update                                             
Downloading the latest package list from hackage.haskell.org
5.59user 0.93system 0:08.73elapsed 74%CPU (0avgtext+0avgdata 514544maxresident)k
376inputs+428352outputs (3major+121981minor)pagefaults 0swaps

After:

ttuegel ~$ rm -fr .cabal/packages/hackage.haskell.org                            
ttuegel ~$ command time /tmp/cabal-dist-cabal-install/build/cabal/cabal update   
Downloading the latest package list from hackage.haskell.org
4.70user 0.56system 0:07.60elapsed 69%CPU (0avgtext+0avgdata 14984maxresident)k
16inputs+428352outputs (0major+1277minor)pagefaults 0swaps

Great work!

ttuegel added a commit that referenced this pull request Sep 18, 2015

Merge pull request #2826 from sznurek/master
Lower memory usage during index file parsing #2396

@ttuegel ttuegel merged commit 74e8ebf into haskell:master Sep 18, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@23Skidoo

This comment has been minimized.

Copy link
Member

23Skidoo commented Sep 18, 2015

I feel a bit uneasy about solving problems caused by lazy I/O with more lazy I/O. Would be nice to move to a more principled scheme at some point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment