Extract documentation

[kritzcreek]

This PR adds a documentation generator in the form of a mo-doc binary.

Usage

Running mo-doc --source <src_dir> --output <out_dir> --format<html|adoc|plain> will generate documentation for all .mo files within src_dir and output it into out_dir. It mirrors the file structure and outputs documentation in the specified format.

By default mo-doc will generate documentation from src/ into docs/ using the html format.

For example, running these commands inside motoko-base

$ mo-doc
$ firefox docs/List.html

yields

Comment syntax

mo-doc recognizes two forms of documentation comments:

/// starts a line comment (Notice the trailing space!)
/** starts a block comment

Ideally I'd like to not support block comments because dealing with indentation is awkward and hard to explain to users, but for now it's recognized so I can demonstrate it working on the current base library.

Documentation comments are parsed as Markdown for the HTML format and copied verbatim when emitting Asciidoc. I didn't want to write a Markdown -> Asciidoc conversion here, because I think we should special case the base library as little as possible and instead work towards tooling that works on all Motoko libraries.

Recognized declarations

Documentation is generated for all public module level types, values, functions and classes as well as their members.

The tool also recognizes leading documentation comments describing the purpose of the module and serving as an introduction. Those module level documentation comments must be placed before the import section.

When writing documentation for functions one has the option of choosing between a let-binding and a func declaration, which either makes the generated documentation:

public let isNil : <T>(List<T>) -> Bool

Returns true if the list is empty.

or

public func isNil<T>(l : List<T>) : Bool

Returns true if the list is empty.

depending on whether one would like the argument names to show up. In future versions using the func declaration will also allow documenting the individual arguments.

Cross-Referencing

Right now the HTML renderer emits ids for all declarations that can be referenced using Markdown syntax:

[Local reference to List.filter](#value.filter)
[Relative reference to Array.map](Array.html#value.map)

I intend to add an understanding about packages, so we can also reference identifiers across those.
None of the types in type signature are references yet, I plan on making them links in a follow-up PR.

Features not in this PR (that I'd like to implement in the future)

• Parse and typecheck Motoko code blocks in comments
• ToC/Navigation
• A documentation page for the primitive types
• Hyperlinks in all type signatures
• Documentation for record and variant fields in type declarations
• A few markdown extensions (like safe inline HTML)
• Syntax coloring for Motoko code blocks in the markdown

[dfinity-ci]

This PR does not affect the produced WebAssembly code.

[crusso]

suppress scope type parameter & what about the sort?

[kritzcreek]

Suppressed scope parameters but I'll need a refresher on where those obj/func_sort 's go and which are the defaults/which need to be displayed.

[crusso]

Roughly: but look at type pretty printer for inspiration.
T.Local -> default
T.Shared T.Write pat-> shared
T.Shared T.Query pat-> shared query

T.Actor -> 'actor'
T.Module -> 'module'
T.Object -> default

[kritzcreek]

Allright, should be fixed. (My main question was where the obj sort goes for classes, but I went with before the class so we'd get actor class ... (Does module class .... ever make sense?)

[crusso]

module class makes some sense but don't know how useful it is since we don't have open type definitions.

[crusso]

Some minor things to fix.

You probably don't need to fix the scope parameter issue to produce doc for the Motoko-base, so feel free to do that in a separate PR if easier (or bored with this)

[kritzcreek]

Allright, I think that should be all the comments adressed, PTAL. (And thank you for the thorough review!)