Add api-markdown-documenter package to replace our external api-documenter fork

[Josmithr]

Adds new package: @fluid-tools/api-markdown-documenter - a wrapper around API-Documenter which offers a programmatic API surface for generating API documentation from an API report generated by API-Extractor.

Goal

The purpose of this library is to process the reports generated by API-Extractor and generate a suite Markdown-formatted API documentation from them. Generally, this is accomplished by using API-Documenter directly, but that package has a number of serious limitations:

• It prescribes a specific documentation suite layout, and one that was crafted for use with DocFX specifically.
  • We are not using DocFX, and the DocFX format does not work well with our website generation system (Hugo).
• It generates a separate file for each API item.
  • This ends up being a massive list of files, and requires a lot of needless cross-page navigation when reading the documentation.
• It exposes only a single, CLI entrypoint, which takes in a list of API reports and generates a set of files.
  • This does not offer much in the way of flexibility, and means we are limited with regards to how we use the generated content.

This library aims to solve these problems by exposing an API surface with the following primary goals:

• Support programmatic usage
  • Allows consumers to do more with the generated content other than simply writing it unmodified to disk. By offering an API that returns a description of documentation content, consumers may choose to embed the content in other document files, transform the contents to inject system-specific (e.g. Hugo) front-matter before writing content to disk, etc.
• Highly extensible / configurable
  • Offers flexibility around file / directory layout, content-type-wise rendering policy, etc. (with a suite of default policies and rendering helpers).

Current Status

V1 adds the general logic for generating markdown documentation with some level of configurability. It does not yet generate documentation in visual parity with our existing API-Documenter fork (though it is pretty close). Such parity (aside from some visual and hierarchical improvements over our existing fork) will be added in future PRs.

The main entry-points to the library are in MarkdownDocumenter.ts.

Extensibility options are exposed via config types declared in Policies.ts and RenderingPolicy.ts.

For sample rendering output, see the contents of src/test/snapshots. It is the test output for a sample test-suite copied over from our API-Documenter fork.

Note that the package currently does not have a CLI entry-point. I'm not sure we really need one. I see this being invoked programmatically so we can inject Hugo frontmatter, etc. dynamically. This can always be re-evaluated in the future though, if we think one would be valuable.

Open Questions

1. Where does this live? This really isn't Fluid-related, it's just a tool we intend to maintain / use. Should this be in its own repo? In a repo with other library packages we maintain and use, but aren't actually Fluid-specific?

Next steps

This PR will merge the package in as a "private" package. Further work will need to be done before making it public and consuming it in the website code (/docs). My intention is to tackle these as subsequent PRs

• Verify Hugo compatibility in this website code
• Update rendering to have style-wise parity with the current implementation

And, of course, eventually replace our use of the existing API-Documenter fork with the new package.

For reference, I have a draft PR up with the changes (relative to this branch) to start using the new package in our docs build: Josmithr#1

Long-Term Goals

Longer term, we wish to expand on this library to be even more flexible / configurable and to rely on API-Documenter-specific concepts less. The intention is to model this library in terms of Abstract Syntax Trees (ASTs), using unist and mdast to offer consumers a more granular surface for generating / modifying generated API docs contents.

[Josmithr]

Partially reviewed. I got through the MarkdownDocumenterConfiguration file. I'll pick it up again this afternoon.

One high level question: what's the distinction between "policy" and "configuration?"

The terminology was conjured a bit arbitrarily, but I was viewing "config" as the overall config object, and "policies" as the individual configuration options for file structure, rendering, etc.

Definitely open to suggestions, especially if we have an existing terminology convention :)

[tylerbutler]

Very exciting work! Looks good to me overall. I quibble with the distinction between config and policy, but not worth blocking this initial change.

[tylerbutler]

Do you want to run these in CI?

[Josmithr]

Interesting. Yes. I have updated this, but I copied this from another template. Apparently we aren't running these consistently?

[github-actions]

This commit is queued for merging with the next branch! Please ignore this PR for now. Contact @microsoft/fluid-cr-infra for help.