Support proper JSDoc block AST

[brettz9]

💻

• Would you like to work on this feature?

What problem are you trying to solve?

Besides being a virtual standard for documentation, JSDoc has been blessed further by TypeScript in its using it to encode type information (in JavaScript mode).

(JSDoc could even be used in theory to be used as a basis for conversion to TypeScript which could in turn be converted into WebAssembly through AssemblyScript, providing a role beyond documentation and type checking into actual running code.)

However, until now there has been no standard way to represent JSDoc blocks, beyond treating them just as other comments, in AST.

In eslint-plugin-jsdoc, in conjunction with via jsdoccomment , we use comment AST to allow rules to be defined which target particular contexts based on their AST structure. We do this so that users have full control of contexts to which rules apply, and so that they can require or prevent certain JSDoc structures from existing. A user could therefore require certain tags but only where a particular tag was present, or report that a particular type was used or shouldn't be used as the child of another type. These types include, via jsdoc-type-pratt-parser, support for any TypeScript types that are used within JSDoc. In other words, any aspect of a JSDoc block can be targeted and targeted intelligently.

In a proof of concept ESLint parser, at https://github.com/es-joy/jsdoc-eslint-parser , I use @babel/eslint-parser (or optionally @typescript-eslint/parser) to get jsdoc and jsdocBlocks properties added which point to JSDoc AST (these are analogous to leadingComments/trailingComments and comments respectively, but for JSDoc). Using a full-blown parser makes possible AST queries which, unlike our eslint-plugin-jsdoc hack, can combine JSDoc as well as regular JavaScript AST into a single expression, and be targeted by ESLint rules such as no-restricted-syntax. This AST could in theory be targeted by parser-specific ESLint rules as well.

What this doesn't get us is the ability to manipulate AST, given that there apparently are not ESTree-capable tools which support such AST-aware manipulation as with @babel/traverse.

A specific use case we have with ESLint (and it is an approach I'd like to use in my own projects as well) is that we'd like to have certain directives in our JSDoc which can be stripped or modified before being supplied to TypeScript, so that we can keep our source code in plain JavaScript + JSDoc, while getting a declaration file built and one that doesn't need editing separately from the JavaScript, with its type information derived solely from inline JSDoc. (TypeScript's awareness of JSDoc + plain JavaScript, though good, doesn't currently provide quite as much robustness as regular TypeScript when in plain JSDoc+JavaScript mode, so we want to work around some of those limitations in our build process, for example, by stripping or expanding typedefs which TypeScript always exports, though we just want to use them as internal aliases.)

I imagine there may be other use cases for transformation. Some might wish to convert JSDoc-enhanced JavaScript into asm.js, or as mentioned, into proper TypeScript for use by AssemblyScript, causing a pipeline from regular JavaScript to WebAssembly code. With ESLint's awareness of such comments, rules could be made to require or prevent certain syntax or structures for typing purposes (e.g., reporting against using the "number" type instead of a particular float/integer type).

Describe the solution you'd like

So besides my discussion question asking whether you might align more with ESTree (so your types and generator could work out of the box), I'm especially wondering if you would be open to supporting optional parsing and support of special JSDoc block AST properties which could be understood in code traversal, manipulation and generation (if not builders as well). We have an algorithm, originally used in ESLint, for determining the attachment points of a JSDoc node if that may help.

While many will no doubt continue to favor the more succinct form that TypeScript proper provides, it is undoubtedly compelling for projects to maintain accessibility to a greater developer base by allowing plain JavaScript which can be progressively enhanced in a standard, simple way, whereby novices can more easily set aside concern of the JSDoc, while learning to make use of it or allowing others on the type to do so.

If such a pipeline existed, existing code might already be ready for type awareness as well.

Describe alternatives you've considered

One alternative is to fork Babel types and Babel generator to allow us to manipulate the JSDoc AST we have with the experimental @es-joy/jsdoc-eslint-parser and then reserialize it. This wouldn't as easily allow others to benefit from the work, or ourselves to benefit as easily from future improvements. It also requires some extra work to get familiar with Babel, to make a fork, deal with the TypeScript source which is less familiar, etc.

Another alternative is to build our own ESTree-aware manipulating tools for use with our parser, but this would require even more work.

Yet another alternative is just using Babel as it is, and applying regular expressions against the stringified comments. Such manipulation can be unpleasant and prohibitive.

Yet another alternative is using Babel as it is, but manipulating the comments and then reserializing them back as regular comments (assuming Babel types' manipulation supports modification and then reserialization of comments too). This is probably the route I would take if there is no interest by your project, but besides this being less elegant than solely manipulating and serializing the JSDoc AST , it seems that it may be compelling for other use cases for JSDoc blocks to be treated as (optionally) first class citizens by the parser, manipulators and generator, especially given TypeScript and AssemblyScript applications.

Documentation, Adoption, Migration Strategy

I can only think of adding an option like structuredJsdoc: true to the parser which triggers attachment of such structured comment nodes.

@babel/types might also expose builders like t.jsdocBlock(), t.jsdocTag(), t.jsdocType(), t.jsdocDescriptionLine, etc., but your traversal, manipulation methods, generator, etc. should require no public changes requiring any special documentation.

I realize this may be rather ambitious, but I thought I'd spell about some of the potential appeal in case you may be willing to adopt. Thanks!

[babel-bot]

Hey @brettz9! We really appreciate you taking the time to report an issue. The collaborators on this project attempt to help as many people as possible, but we're a limited number of volunteers, so it's possible this won't be addressed swiftly.

If you need any help, or just have general Babel or JavaScript questions, we have a vibrant Slack community that typically always has someone willing to help. You can sign-up here for an invite.

[JLHwung]

The goal of @babel/parser to parse ECMAScript and its proposal, as well as some popular extension such as TypeScript / Flow. However, JSDoc does not fall into this catalogue: In essence, JSDoc is an independent markup language, although it has been widely used in JavaScript projects, it can be, theoretically, used in any other programming language, as long as the language supports comments as whitespace.

Because comment is whitespace, @babel/parser only attaches them to closest AST node. I don't think the parsed jsdoc AST should be attached to the JavaScript AST because they are whitespaces in JS world. Users can put assembly / machine code / html or anything in the comments and have a tool recognizing them to perform certain tasks. Therefore, adding jsdoc to the JS AST will be open-ended.

In you case, I think it is okay to "manipulating the comments and then reserializing them back as regular comments". Babel does support such usage:

traverse(ast, {
  enter(path) {
    const jsdocAST = JSDocParse(path.node.leadingComments);
    transformJSDoc(jsdocAST);
    path.node.leadingComments = JSDocGenerate(jsdocAST);
  }
}

[brettz9]

The goal of @babel/parser to parse ECMAScript and its proposal, as well as some popular extension such as TypeScript / Flow. However, JSDoc does not fall into this catalogue:

But in a sense I think it does fall under at least the TypeScript category. TypeScript has its own formal handling of JSDoc, blessing, extending, and formalizing it, especially through tsdoc. When building a declaration file from plain JavaScript, the JSDoc types will be used in the resulting declaration file, so it is not purely for documentation, but for type-awareness.

Because comment is whitespace, @babel/parser only attaches them to closest AST node. I don't think the parsed jsdoc AST should be attached to the JavaScript AST because they are whitespaces in JS world.

In the scheme I've used in @es-joy/jsdoc-eslint-parser, they are also attached to the closest AST node; there is no need for AST specifiers to figure out the semantics of where it should belong. However, our @es-joy/jsdoccomment project has code to find out, for given ES AST, which JSDoc block is likely to satisfy (e.g., for a function expression, JSDoc on the preceding variable declaration should likely satisfy).

Users can put assembly / machine code / html or anything in the comments and have a tool recognizing them to perform certain tasks. Therefore, adding jsdoc to the JS AST will be open-ended.

The semantics at least are not too open-ended, as they codify types based on TypeScript. But performing tasks based on them is, I think, a feature. This is how TypeScript is able to do type checking on plain JavaScript+JSDoc files. Tools can already opt to attach comments, so I don't imagine it is opening a Pandora's Box to formalize the extension.

In you case, I think it is okay to "manipulating the comments and then reserializing them back as regular comments". Babel does support such usage:

traverse(ast, {
  enter(path) {
    const jsdocAST = JSDocParse(path.node.leadingComments);
    transformJSDoc(jsdocAST);
    path.node.leadingComments = JSDocGenerate(jsdocAST);
  }
}

The problem is that this does not allow querying the JSDoc (including querying the JSDoc within the same query expression as the ES).

By using the JSDoc-within-AST approach, you can see how light it was for us to use esquery at https://github.com/es-joy/js2ts-assistant for our use case, e.g., to grab @typedef's with @local and then use that info to replace TypeScript-flavored JSDoc AST (the JsdocTypeName query which targets AST produced by jsdoc-type-pratt-parser) so as to be able to allow use of @typedef for type aliases within plain JavaScript, without getting all the @typedef aliases exported into the TypeScript declaration file.