haddock is painfully slow on large library builds (aka agda)

[cartazio]

on the agda code base, haddock runs for quite some time! I've not yet dug into how much is due to ghc as a frontend or not, but notating it here for now. It may very well be that the ghc api is the real culprit, but some serious profiling will be needed to zoom in on the root of the matter.

[bgamari]

Indeed, Haddock performance could use some more eyes. Perhaps you'd like to do some quick profiling and see where the major pain points are?

[mpickering]

There is some profiling info in #462

I also have a local branch where I tried to make some improvements but it didn't seem to make any substantial difference.

[asr]

+1

[andreasabel]

I see you did some work at #621.

[alexbiehl]

Sorry for the hiccup. I meant to say that also in this area @duog has improved the handling of TemplateHaskell which currently causes haddock to build every module in the module graph. With ghc-head it only considers modules transitively depending on TemplateHaskell for compilation.

Bottom line: In general project with TemplateHaskell are much slower than projects without. Because GHC really has to generate code for them first.

[andreasabel]

There is still the problem that an inefficient data structure is used at https://github.com/haskell/haddock/blame/master/haddock-api/src/Haddock/Backends/Hyperlinker/Ast.hs#L46. Given the linear time for each lookup, you have the quadratic factor without needing anymore explanation. Here is the evidence: http://lpaste.net/147305

[alexbiehl]

Yes, we could use a more efficient structure there. @andreasabel could you give it a shot in a PR?

On the other hand: This profile used haddock-2.16.1. If you take a look at the cost centers you will see that most of the time is spent in everything and combine and not in enrichToken though:

edit: never mind, I confused runtime with allocation columns.

        enrich                            Haddock.Backends.Hyperlinker.Ast      5702         211   24.5   18.2    52.0   47.3
         everything                       Haddock.Syb                           5723     7487674    7.0    4.9    27.5   29.1
          combine                         Haddock.Syb                           5724     5299211   20.6   24.2    20.6   24.2 

Please try to try profile with haddock-2.18.1 again it should make the quadric factor more explicit in profiles.

[alexbiehl]

I just found mpickering@65d0bc6

Could you do a profile with this patch applied?

[andreasabel]

I tried to install various versions of haddock (including master), with no success at first. The build instructions I added in PR #670 would have saved me quite some time.

I see that in the meantime, you did the fix already in #669.

[duog]

Are you passing --hyperlinked-source to haddock? IIRC this code path is only used with that flag?