Very large growth in memory required to compile with 0.5.0.0

[codebje]

I have a(n externally supplied) protobuf definition with 713 messages and 129 enums. Under 0.4.0.2, this compiled well enough, taking 7.5 minutes with 2.5G maximum residency:

 347,294,340,904 bytes allocated in the heap
  55,614,126,768 bytes copied during GC
   2,576,274,256 bytes maximum residency (45 sample(s))
       9,583,792 bytes maximum slop
            2456 MB total memory in use (1 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0     18003 colls,     0 par   93.486s  102.586s     0.0057s    0.8225s
  Gen  1        45 colls,     0 par   182.046s  199.756s     4.4390s    12.7159s

  TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1)

  SPARKS: 0(0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.001s  (  0.004s elapsed)
  MUT     time  173.485s  (205.604s elapsed)
  GC      time  275.532s  (302.342s elapsed)
  EXIT    time    0.000s  (  0.008s elapsed)
  Total   time  449.017s  (507.957s elapsed)

  Alloc rate    2,001,869,803 bytes per MUT second

  Productivity  38.6% of total user, 40.5% of total elapsed

Under 0.5.0.0, this takes 24 minutes and 8.5G max residency:

 850,873,682,792 bytes allocated in the heap
 140,855,616,328 bytes copied during GC
   8,447,321,488 bytes maximum residency (37 sample(s))
      45,206,320 bytes maximum slop
            8055 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0     36747 colls,     0 par   254.665s  293.696s     0.0080s    6.5932s
  Gen  1        37 colls,     0 par   607.494s  1332.109s     36.0029s    194.0651s

  TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1)

  SPARKS: 0(0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.001s  (  0.003s elapsed)
  MUT     time  561.676s  (939.380s elapsed)
  GC      time  862.159s  (1625.805s elapsed)
  EXIT    time    0.000s  (  0.008s elapsed)
  Total   time  1423.835s  (2565.195s elapsed)

  Alloc rate    1,514,884,418 bytes per MUT second

  Productivity  39.4% of total user, 36.6% of total elapsed

Peak memory usage was a shade over 16Gb for the process. Check out that three minute GC pause! I suspect that may have occurred when GHC walked a large amount of memory that OS X had compressed.

It looks like the primary culprit for the blow-out is buildMessage. By replacing parseMessage and/or buildMessage bodies with Prelude.undefined, I have made the following observations:

Version	parse?	build?	Total memory	Max. GC	GC time	Total time
0.4.0.2	n/a	n/a	2456 MB	12.7s	276s	449s
0.5.0.0	yes	yes	8055 MB	194.1s	862s	1424s
0.5.0.0	no	no	3033 MB	13.7s	443s	729s
0.5.0.0	no	yes	7065 MB	102.8s	620s	1007s
0.5.0.0	yes	no	4690 MB	22.5s	598s	987s

With neither parseMessage nor buildMessage function bodies, the compile complexity is broadly similar to 0.4.0.2 (I suspect perhaps TypeApplications adds some of the additional time, but that's a lot more work to try and isolate.) Testing with one or the other undefined narrows it down to buildMessage as the leading cause of significant compile time and space.

I've tried to work around the problem by splitting the proto up into smaller chunks, which means compilation can actually happen without heap overflows, but Cabal has a fairly low limit on the number of modules it can parse in a Cabal file - I can't just split into ~800 separate files, I have to pull out subgraphs.

Any ideas why buildMessage might be causing compile cost issues?