Injecting external content into a doc comment ("@include" tag?)

[octogonz]

In RFC: Core set of tags for TSDoc, @EisenbergEffect brought up the topic of a TSDoc tag that would inject content from an external source. I've already seen a couple of examples of this internally. Sometimes the Markdown engine itself is used for this purpose.

Is there a way that this could be done generically using a TSDoc core tag (e.g. @include)? Or will this always be proprietary to the particular documentation pipeline, in which case we should treat it as a custom tag?

@dend

[dend]

If we are building a generic solution, I see how this can be proprietary to particular documentation pipelines. As an example, for DocFX, the import/include syntax is defined here: File Inclusion. I don't see the value in constraining this to a specific syntax given that it can be a custom tag.

Thoughts @pgonzal @EisenbergEffect?

[EisenbergEffect]

Perhaps all that is needed is to "reserve" a tag for this particular purpose and then extract that information into a known format so that arbitrary external processors can work with it. Then, in the canonical implementation, have an API that you can register an include handler with to do the actual processing.

[octogonz]

As an example, for DocFX, the import/include syntax is defined here: File Inclusion. I don't see the value in constraining this to a specific syntax given that it can be a custom tag.

The example on that page implies a syntax like this:

...Other inline contents... [!include["example title"]("some/path")]

@dend is DocFX intending to be CommonMark compatible? As far as I know CommonMark supports ![foo](/url "title") for images but not an exclamation mark inside square brackets. When I paste the above snippet into https://markdown-it.github.io/ in CommonMark mode, the output gets garbled:

<p>...Other inline contents... [!include<a href="%22some/path%22">&quot;example title&quot;</a>]</p>

The current aim is for TSDoc syntax to be able to coexist with (a reasonable subset of) the CommonMark syntax. So, if the DocFX content contained an at-sign like this...

...Other inline contents... [!include["see @realdonaldtrump"]("some/path")]

...then TSDoc would incorrectly interpret @realdonaldtrump as a custom doc comment tag, because the surrounding syntax looks like a stream of meaningless symbols. (Apologies for the example content heheh -- it was the first thing that popped into my head!)

Most Markdown engines do not stick to CommonMark, but instead mix in some highly unpredictable custom notations. That's fine in *.md files, but I think we'd want to minimize that practice in *.d.ts files if the goal is interoperability between tooling.

If you're looking for a way to embed custom extensions inside CommonMark, I would suggest instead using HTML tags which are completely supported by the standard. Compared to proprietary notations, HTML is a familiar and rich notation that makes it easy for other tooling to ignore unrecognized/unsupported constructs.

[octogonz]

If <title> and <filepath> are generally the only important parameters, the generic solution would be pretty straightforward. It can just follow {@link} syntax:

/**
 * Some external content gets injected below this line.
 * {@include some/path | example title}
 */

[tenry92]

Should this be usable for non-markdown includes, such as code snippets? If this is the case, we can't TSDoc simply load the referenced file's contents in place, as the consuming software might not be aware to possibly add code highlighting etc. to it. I think, the consumer should be aware of the file to be included (file path, file name, file extension).

[octogonz]

Should this be usable for non-markdown includes, such as code snippets?

I'm thinking this might be up to the discretion of the particular documentation tool. At least, the original goal of TSDoc was to ensure that different tools can agree on the parsing of TypeScript doc comments. As far as the handling of an external file, its file extension might mean different things to different documentation tools, and the handling of that file might governed by some other standard such as CommonMark or HTML.

[dend]

This is where I think we just need to ensure that TSDoc has flexibility in the Markdown markup. We are working on adding Markdown extensions to docs.microsoft.com and therefore it would be good to ensure that TSDoc does not break because it encounters a piece of Markdown it doesn't understand.

[octogonz]

This is where I think we just need to ensure that TSDoc has flexibility in the Markdown markup. We are working on adding Markdown extensions to docs.microsoft.com and therefore it would be good to ensure that TSDoc does not break because it encounters a piece of Markdown it doesn't understand.

The way to accomplish that is for the custom syntaxes to be built from a standard extensibility mechanism. For example, <author>Bob</author> is fine to stick in the middle of a doc comment, since a parser can easily skip over an unrecognized HTML tag. Similarly, {@author Bob} would also be okay because TSDoc supports custom inline tags. Whereas an ad hoc syntax like [!author name="Bob"] is not going to be handled correctly unless the [! ] syntax can be standardized and made extensible.

In many cases the ad hoc syntax can pass through and get rendered by the back end without any trouble. Causal users may be fine with this, even if there are occasional glitches. TSDoc will support it in lax mode. But for large scale authoring, these ad hoc syntaxes are going to run into all the troublesome Markdown grammar ambiguities. (For example, when someone needs to put punctuation characters in the "Bob" part, it's likely to get misinterpreted as some other construct, leading to a very confusing authoring experience.) This is unavoidable, since there's simply no way for one tool to introduce arbitrary punctuation characters into the input and expect other tools to handle that correctly.

So, to the extent that DocFX's syntax extensions are intended to be used in source code comments, we should try to design them with the idea that other tools will need to process them. (That said, I suspect that many of the extensions will not really be needed inside code comments, since this authoring scenario has relatively conservative needs.)

[bukowa]

It's been a while, I landed here while googling for a way to include example into my docs from examples folder that will be later parsed by typedoc. Are there any standards in that regard yet?

[Gerrit0]

For TypeDoc specifically, there is typedoc-plugin-include-example which can include files.

It's also easy to implement a plugin which adds support for {@include ./test.md} and {@includeCode ./test.ts}

$ typedoc --plugin ./include-plugin.js

Plugin code

// CC0
// TypeDoc 0.26
// @ts-check
import td from "typedoc";
import path from "path";
import fs from "fs";

/** @param {td.Application} app */
export function load(app) {
    app.on(td.Application.EVENT_BOOTSTRAP_END, () => {
        const tags = app.options.getValue("inlineTags").slice();
        if (!tags.includes("@include")) {
            tags.push("@include");
        }
        if (!tags.includes("@includeCode")) {
            tags.push("@includeCode");
        }
        app.options.setValue("inlineTags", tags);
    });

    app.converter.on(td.Converter.EVENT_CREATE_DECLARATION, checkIncludeTags);
    app.converter.on(td.Converter.EVENT_CREATE_PARAMETER, checkIncludeTags);
    app.converter.on(td.Converter.EVENT_CREATE_SIGNATURE, checkIncludeTags);
    app.converter.on(td.Converter.EVENT_CREATE_TYPE_PARAMETER, checkIncludeTags);
}

/**
 * @param {td.Context} context
 * @param {td.Reflection} refl
 */
function checkIncludeTags(context, refl) {
    if (!refl.comment?.sourcePath) return;

    const relative = path.dirname(refl.comment.sourcePath);
    checkIncludeTagsParts(context, refl, relative, refl.comment.summary);
    for (const tag of refl.comment.blockTags) {
        checkIncludeTagsParts(context, refl, relative, tag.content);
    }
}

/**
 * @param {td.Context} context
 * @param {td.Reflection} refl
 * @param {string} relative
 * @param {td.CommentDisplayPart[]} parts
 * @param {string[]} included
 */
function checkIncludeTagsParts(context, refl, relative, parts, included = []) {
    for (let i = 0; i < parts.length; ++i) {
        const part = parts[i];
        if (part.kind === "inline-tag" && ["@include", "@includeCode"].includes(part.tag)) {
            const file = path.resolve(relative, part.text.trim());
            if (included.includes(file) && part.tag === "@include") {
                context.logger.error(
                    `${part.tag} tag in comment for ${refl.getFriendlyFullName()} specified "${part.text}" to include, which resulted in a circular include:\n\t${included.join("\n\t")}`,
                );
            } else if (fs.existsSync(file)) {
                const text = fs.readFileSync(file, "utf-8");
                if (part.tag === "@include") {
                    const sf = new td.MinimalSourceFile(text, file);
                    const { content } = context.converter.parseRawComment(sf, context.project.files);
                    checkIncludeTagsParts(context, refl, path.dirname(file), content, [...included, file]);
                    parts.splice(i, 1, ...content);
                } else {
                    parts[i] = {
                        kind: "code",
                        text: makeCodeBlock(path.extname(file).substring(1), text),
                    };
                }
            } else {
                context.logger.warn(
                    `${part.tag} tag in comment for ${refl.getFriendlyFullName()} specified "${part.text}" to include, which was resolved to "${file}" and does not exist.`,
                );
            }
        }
    }
}

/**
 * @param {string} lang
 * @param {string} code
 */
function makeCodeBlock(lang, code) {
    //
    const escaped = code.replace(/`(?=`)/g, "`\u200B");
    return "\n\n```" + lang + "\n" + escaped.trimEnd() + "\n```";
}

[bukowa]

For TypeDoc specifically, there is typedoc-plugin-include-example which can include files.

Thank you - ill try this approach next time (I were looking for a quickstart of how to create such plugin in typedocs documentation but couldn't find one), meanwhile this works for me with typedoc also:

My script

examples/example.js:

/*
---
title: "2. Quickstart Guide"
group: Documents
category: README
---
 */
import assert from "node:assert";
// more code

src/index.ts:

/**
 * Entry point for my package
 *
 * @document ../examples/quick-start.md
*/

scripts/examples-to-md.js:

/**
 * Converts javascript examples to Markdown format.
 *
 * @example
 * ```bash
 * node example-to-markdown.js /path/to/examples
 * ```
 *
 */
import {readdirSync, readFileSync, writeFileSync} from 'node:fs';
import {resolve, join, basename} from 'node:path';

// regular expression to match frontmatter
const reFrontmatter = /\/\*\s*([\s\S]*?)\s*\*\//;

// function to convert example file to markdown
function exampleToMarkdown(file) {
    let content = readFileSync(file, 'utf-8');
    let frontmatter = reFrontmatter.exec(content)[1].trim();
    content = content.replace(reFrontmatter, '').trim();
    let newContent = `${frontmatter}
\`\`\`\`javascript
${content}
\`\`\`\`\``
    let newFile = join(resolve(file, '..'), basename(file, '.js') + '.md');
    console.log(`Writing markdown to new file '${newFile}'`)
    writeFileSync(newFile, newContent);
}

// read first argument
const dir = process.argv[2]
if (!dir) {
    console.error('Please provide a directory');
    process.exit(1);
}

// resolve the directory
const fullDir = resolve(dir);
console.log(`Reading examples from '${fullDir}'`);

// read all files in the directory
const files = readdirSync(fullDir);

// convert each file ending with `.js`
files.filter(file => file.endsWith('.js')).forEach(
    file => exampleToMarkdown(resolve(fullDir, file))
)

And then:

node ./scripts/examples-to-md.js ./examples && typedoc --tsconfig tsconfig.json