Regression in 1.7.1 (compared to 1.6.5) in getPackageFiles RAM usage

[k-bx]

General summary/comments (optional)

If I run "stack build" with 1.7.1 on one of our projects, it grows up to 16GB upon the getPackageFiles step. Doesn't happen on another proj which shares most of deps but not all. Doesn't happen on 1.6.5.

Possibly related: #1235

Bisect results:

acb196788948bab7a01e13cddf6e6cd5b5bf93a0 is the first bad commit
commit acb196788948bab7a01e13cddf6e6cd5b5bf93a0
Author: Emanuel Borsboom <manny@fpcomplete.com>
Date:   Wed Dec 27 15:04:05 2017 -0800

    setup: improvements to selecting bindists on Linux

    These changes are motivated by #3636.

    * `stack setup` looks for GHC bindists and installations by any OS key
      that is compatible (rather than only checking a single one).   This is
      relevant on Linux where different distributions may have different
      combinations of libtinfo 5/6, ncurses 5/6, and gmp 4/5, and will allow
      simpifying the setup-info metadata YAML for future GHC releases.

    * `stack setup` no longer uses different GHC configure options on Linux
      distributions that use GCC with PIE enabled by default.  GHC detects
      this itself since ghc-8.0.2, and Stack's attempted workaround for older
      versions caused more problems than it solved.

:100644 100644 5c2e93fe50a2b6b3a1b61ba3c20d22177000f2c7 1667e4507cf6465eb685127d60402ffd25322066 M      ChangeLog.md
:040000 040000 0a6a8aa14860157aa6c741d147d36924c098b628 14217e82bc4da2a9aac1a3f9f75e12fdbd003d1e M      src

[k-bx]

/cc @borsboom

[k-bx]

Profiling info: stack.prof.gz

Wasn't helpful to me.

[ncaq]

Can you use hp2any-manager: A utility to visualise and compare heap profiles. ?

[k-bx]

It's shown as PINNED

stack-hp.zip

[k-bx]

Huh. For some reason I thought that stack clean --full didn't help previously, but now I've checked on a newly-cloned repo, and it doesn't reproduce. Closing for now, will reopen if I'll get more info.

[k-bx]

Just FYI, one of our .dump-hi files is 11G under some circumstances, probably that's the reason. I'll investigate more and ask somewhere in GHC if it's something that can be fixed.

[k-bx]

Reopening. The circumstances I previously thought are our infrastructure problems (docker scripts etc.) is not actually the case, I can reproduce this problem by just compiling with optimizations (-O). Building without optimizations doesn't create that huge 11G file, but turning on -O does.

[k-bx]

PR merged 😃