[Hi Haddock] In what format to export Haddocks in .hi files?

[sjakobi]

An important design question of the Hi Haddock project is the choice of format that GHC shall use to include documentation in its .hi interface files. I am currently aware of two competing proposals for that format:

1. Export the fully parsed and renamed AST in haddock-library's MetaDoc/DocH format. This is what I proposed in my GSoC proposal. I'll call this the DocH-variant.

2. Export the raw docstrings, and all information that is necessary to rename identifiers in the documentation or to link to dependencies. Let's call this the raw-docstring-variant.

Here are some aspects of the two variants as I currently see them. Please mention any other aspects in the comments as I'm sure I've missed a few!

Aspects of the DocH-variant

• GHC will depend on haddock-library to parse the documentation.
• haddock, ghci and other consumers of the format don't need to do any further parsing or renaming.
• Unlinkable identifiers are identified and warned about at compile-time.

Aspects of the raw-docstring-variant

• GHC gets a simple lexer that detects identifiers and links and determines what renaming information needs to be included in the interface files.
• The lexer may falsely detect some non-identifiers but that should at worst result in some bloat in the interface files.
• Tools that want to use a different syntax for documentation such as MarkDown or reStructuredText will only have to extend the lexer to detect identifiers in the new syntax.
• This in consequence makes it more likely that we will see other documentation formats and tools in the near future.
• This variant requires less work at compile-time, and is more likely to be cheap enough that the documentation can be output by default.
• haddock, ghci etc. will have to rely on haddock-library or other documentation tools to parse and rename the docs.

(I don't think I'm misrepresenting this idea, but just in case, here is @gbaz's own description.)

Can both variants be combined?

I haven't thought about it much or discussed it, but it currently seems to me, that if I implement the DocH-variant as originally planned, it might not be too hard to add a code path for the the raw-docstring-variant which could be enabled via a ghc haddock-option.
Anyone who has a clearer idea of the things involved here: Please warn me, if you think I'm underestimating the complexity! :)

[harpocrates]

Thanks for opening an issue to discuss this! I strongly support the second variant as I would prefer to not have the Haddock markup format baked into GHC. It seems wrong to force GHC to become aware of one particular markup format, especially a controversial format like Haddock. One other advantage to only packaging raw docstrings is that markup syntax (Haddock, Markdown, etc.) would be able to vary orthogonally to the GHC version.

haddock, ghci etc. will have to rely on haddock-library or other documentation tools to parse and rename the docs.

Why should GHCi need to parse the doc strings at all? It seems to me that displaying just the raw doc strings would be the best thing for the command line... Other than that, it seems morally right that only programs that want to parse the Haddock markup format incur the cost of a library which parses the Haddock format.

I haven't thought about it much or discussed it, but it currently seems to me, that if I implement the DocH-variant as originally planned, it might not be too hard to add a code path for the the raw-docstring-variant which could be enabled via a ghc haddock-option

That solves the problem of giving access to raw docstrings, but not the dependency problem. 😄

[sjakobi]

@harpocrates:

Why should GHCi need to parse the doc strings at all? It seems to me that displaying just the raw doc strings would be the best thing for the command line...

Admittedly this is not entirely clear to me too. (Also, the rendering of the docstrings in ghci is outside of the main focus of the proposal.) But I'm sure that even on the CLI, one can do a lot of nice highlighting with different font styles and colours.

Is that what you had in mind for :doc, @hvr?

[harpocrates]

Admittedly this is not entirely clear to me too. (Also, the rendering of the docstrings in ghci is outside of the main focus of the proposal.) But I'm sure that even on the CLI, one can do a lot of nice highlighting with different font styles and colours.

Copying my reply from the other thread:

• That seems also like something that could be a GHCi option for those who want it (something like the -interactive-print option). One such implementation could then depend on haddock-library, but I wouldn't package that implementation with GHCi.

[sjakobi]

To explain why moving haddock-library into the GHC boot libraries is such a point of contention:

There are currently two proposals underfoot that would increase the dependency footprint of haddock-library:

• Replace 'attoparsec' with 'parsec' #799 adds dependencies on parsec and text (both of which have been introduced as boot libraries in GHC-8.4).
• Support Markdown syntax via commonmark-hs #794 would add a dependency on a new package commonmark-core (which would also depend on parsec and text). That means that if haddock-library becomes a boot library, so does commonmark-core.

@bgamari, could you possibly comment on how these two issues affect haddock-library's chances of becoming a boot library, or know whom to ask?

[hvr]

fwiw, haddock-library was always intended to become a boot library sooner or later anyway; it's always been one in disguise anyway. it's merely been an ugly hack / tech debt it's not already, that I've been too slow to correct sooner (which I had put on hold waiting for Hadrian to get merged and replace the Makefile system), which I'm now terribly regretting, as this would have made this whole discussion moot and consequently safe us the time.

[tomjaguarpaw]

I think it would be rather unfortunate if GHC came to depend too closely on Haddock.

[gbaz]

So.. I think my lexer proposal is sufficiently lightweight and safe that I'm not worried about technical drawbacks (I know herbert may disagree here). I don't see any purpose for combining the two variants, since it doesn't seem to resolve anything.

The fundamental question really is, shall haddock-library become a boot lib.

We should enumerate the tradeoffs properly.

Pro

1. The main benefit I see is for ghci to be able to display formatted as opposed to raw docstrings, with a new :doc command (but with such a command being out of direct scope for this GSoC anyway, as far as I know?).
2. As noted we can warn earlier about unlinked identifiers.
3. The other benefit -- tighter lexing -- doesn't seem significant to me given that a good overapproximate lexer should be easy to produce.

Con

1. making alterations to haddock parsing require changing a bootlib will impede haddock development.
2. ditto for impeding alternate syntaxes or extensions.
3. as a rule of thumb we want to minimize GHC's dependency footprint

Can people help expand either list?

(Btw, with regard to ghci, I also like the suggestion in the other thread and above that ghci could just be given an option for some external formatter for docstrings -- this also lets such things evolve more quickly and e.g. add features for coloring output, etc. The similarity to the existing interactive-print flag means we're not breaking ground, but just extending design principles already in place).

[bgamari]

I also think that GHC shouldn't be too tightly bound to Haddock. GHC is a Haskell compiler and nowhere in the Haskell Report is a documentation syntax described. Consequently, I'm not sure I see a strong reason to include a Haddock parser in GHC. Haddock is already too coupled to GHC and, in my mind, one of the goals of this project is to reduce this coupling, not further entrench it.

Incidentally, I do worry that the community as a whole would only suffer if a competing project to Haddock were to gain traction. We have a hard enough time maintaining a single documentation engine; I really don't think we can afford to maintain two, especially given the various other bits of infrastructure that integrate with Haddock. Regardless, we shouldn't let these social considerations dictate technical decisions.

[sjakobi]

The main benefit I see is for ghci to be able to display formatted as opposed to raw docstrings, with a new :doc command (but with such a command being out of direct scope for this GSoC anyway, as far as I know?).

Rendering the docstrings in ghci is a stretch goal in my proposal.

(Btw, with regard to ghci, I also like the suggestion in the other thread and above that ghci could just be given an option for some external formatter for docstrings -- this also lets such things evolve more quickly and e.g. add features for coloring output, etc. The similarity to the existing interactive-print flag means we're not breaking ground, but just extending design principles already in place).

In case anyone else is unfamiliar with ghci's -interactive-print flag, it's documented here.

And indeed it seems straightforward to add an analogous flag -interactive-render-doc that accepts functions of type LinkEnv -> HsDocString -> IO () or similar. A library that wants to provide a function of this type could then depend on haddock-library.

[wz1000]

Speaking from the tooling perspective, I would like GHC to resolve symbols/identifiers mentioned in comments. It would be great if GHC could at the very least understand the subset of haddock markup that deals with references to symbols(the DocIdentifier constructor).

IIRC, one of the goals of the original proposal was to make it so that that haddock and other tooling don't have to recompile stuff just to generate documentation. If Names are not resolved by GHC on the first go, haddock will be forced to recompile the entire file anyway.

As to if GHC should understand any of the other haddock markup, I'm agnostic. That can always be handled by the ultimate renderer(ghci, haddock html output, haskell-ide-engine tooltips) without an unnecessary and expensive call to GHC to recompile the file.

[gbaz]

Speaking from the tooling perspective, I would like GHC to resolve symbols/identifiers mentioned in comments. It would be great if GHC could at the very least understand the subset of haddock markup that deals with references to symbols(the DocIdentifier constructor).

Right. Both ideas under consideration will provide for this one way or another, and avoid haddock having to do an additional recompilation step.

[gbaz]

By the way, herbert has expressed that he thinks we're sort of fooling ourselves, and that effectively haddock-library is already a bootlib, just wearing glasses and a fake mustache. If this is the case, then it potentially changes the equation here. I understand this has to do with the in-tree flag and how it operates, but I confess I don't understand the details and it would be good to see it spelled out.

[sjakobi]

Speaking from the tooling perspective, I would like GHC to resolve symbols/identifiers mentioned in comments. It would be great if GHC could at the very least understand the subset of haddock markup that deals with references to symbols(the DocIdentifier constructor).

Right. Both ideas under consideration will provide for this one way or another, and avoid haddock having to do an additional recompilation step.

Actually it seemed to me that in the raw-docstring-variant, consumers of .hi-files would still have to re-lex the docstrings in order to know what to rename, link etc.

I think it would be good if this duplicate lexing could be avoided. Maybe the docstring-lexer in GHC could output something like ~[((Int, Int), Name)] to mark the lexed identifiers in the docstring?

[gbaz]

Right -- I was addressing recompilation, not re-lexing of the haddock docstrings. Your suggestion for attaching sourcespans to the name information makes sense to me.