The problem is with C++ libraries specifically: http://hackage.haskell.org/trac/ghc/ticket/5289
This sucks, obviously. Perhaps there should be a flag?
Herp derp. Totally didn't realize you were the original reporter of the bug. Apologies.
See also double-conversion issue 2.
Since now the bug is resolved, should we update document and use double-conversion by default?