[mdoc][enhancement] Detach output pipeline from the processing one

[dend]

What

Today, the XML generation pipeline is tightly coupled with the core tooling. This makes it less flexible and doesn't allow for much extensibility in terms of producing outputs that can be re-used in other places, e.g. Schema-Based Document Processor in DocFX.

Why

As we evolve the documentation pipeline on docs.microsoft.com, we are making sure that we standardize on a single format for all reference content, and that is YAML combined with Markdown (structured + human-edited). With that in mind, as we start looking at moving the .NET API documentation to the new pipeline, we would like to make sure that we can generate the right documentation through mdoc directly instead of having to convert XML to YAML, the way we do today.

[joelmartinez]

/fyi @conceptdev and @lobrien

I'll reproduce some relevant plans that I'd previously submitted as a candidate for a GSoC project:

Implement Programmatic API for Documentation Repo

mdoc is used to scan .NET and UWP assemblies and create a repository of EcmaXML which can be source controlled. This repository is then used as the source to publish documentation to end users (via web and otherwise). Many engineering teams write scripts that read this source in order to create statistics and analysis of their APIs, or even automatically document their library based on business heuristics. This task would build a programmatic interface to create, update, and query these repositories; all of these scenarios and others would benefit from the presence of this API.

Deliverables: An API that lets you create, read, and update an EcmaXML repository. Updates will use a batching system, so you can collect changes to a file and write all at once. Querying would be implemented with a custom IQueryable provider, so that you can easily search repositories with LINQ without running out of memory. Additionally, Unit tests will be required.

---

The underlying idea with that project, is that building this API would require the same refactoring and decoupling of logic from XML processing that this gh issue is describing. With this API built, and producing the same verifiable output as exists currently, we can then provide an alternate implementation that would support SDP as well.