So, I've written an AST spec: what to do next?

[vicentereig]

Hello!

First off, I wanted to thank you all for the work you are putting on all this unified and unist enchilada. I am really impressed by the discipline, level of organization, and the quality of the work you folks put out there. Thank you!

Over the last quarter, we've been diving deep in the unified ecosystem. Given the extent of the surface area, building features of our CMS on top of it has been a smooth and productive experience. We've been waist-deep in processing mdx for a large documentation site. Oh boy, how many times did I wish that node provided all these tools as part of the standard library.

What I haven't been able to grasp yet is what happens once you are done documenting your AST. I've seen this pattern where all AST specs are nicely documented in md like https://github.com/phuctm97/ntast.
Is it a manual process to translate it into the type/nodes you use to parse/compile the the source code? I wouldn't be surprised if you unified folks had some tooling to turn those specs into workable items, so I can write my parser and build the AST representation.

We are delighted to work with these tools you make available tot he community and excited to contribute back!

unist has proven to be incredibly useful toolbox, turning into core piece of our CMS. We use to process mdx and conveniently transform it into many formats:

• indexing our contents in Algolia
• keeping the site navigation in sync with as the content tree changes
• proactively keeping the site from generating broken links
• rendering the docs in the browser obviously
• or nudging our content writers in the right direction as they write articles to support our customers.
• ...

It's been fun. :)

Cheers!

[wooorm]

Thanks for the kind words @vicentereig <3
If some of the things y’all made are OSS, I’d love to take a peek at them.

Is it a manual process to translate it into the type/nodes you use to parse/compile the the source code? I wouldn't be surprised if you unified folks had some tooling to turn those specs into workable items, so I can write my parser and build the AST representation.

It is in fact a manual process! All that stuff is manual.
I’m not really a fan of TS, although I do see some value in JSDoc-based TS, especially because that makes it possible to generate docs from code. I haven’t built something like that yet though (and it seems daunting to figure out a one-size-fits-all for the hundreds of projects)

[ChristianMurphy]

I've written an AST spec: what to do next?

@wooorm tends to create custom/manual parsers, which can be fast and highly flexible.

Though that isn't the only starting point, and likely not the easiest starting point.
Parser generators, like Peg.js, Lezer, and Nearly.js (just to name a few), offer a grammar focused approach.
Writting grammars and parsers in general can be challenging, having a tool chain automatically take care of backtracking, look ahead, branching, etc, which can make it a bit more approachable than doing so manually.

[ChristianMurphy]

Redot for example uses a Peg grammar https://github.com/redotjs/redot/blob/42d246a3351fda6e22b8a3f644f958ae1017216b/packages/redot-parse/dot.pegjs

Though it may be dropped in the future, since graphviz now generates a JSON format which could be made unist compatible.
Which is another tip, if there is an existing parser, consider using it.
Doing a tree walk to turn an existing non-unist AST into one that is unist compatible, is often easier and more maintainable than writing a parser.

[vicentereig]

If some of the things y’all made are OSS, I’d love to take a peek at them.

Not sure whether our tools per se are worth sharing the community, since they are pretty much tailored to our content schema and the way our CMS has grown organically over the years. On a positive note, I'm prepping —at the ludicrous speed of a 🐢— to publish some notes around the versatility of unist for content-management-related problems. My faves so far are modelling the Content Tree of a website, and minimizing broken links as it changes due to re-structures. Happy to share that when I've got something drafted out!

It is in fact a manual process! All that stuff is manual.

Awesome, thanks for confirming.

I’m not really a fan of TS, although I do see some value in JSDoc-based TS, especially because that makes it possible to generate docs from code. I haven’t built something like that yet though (and it seems daunting to figure out a one-size-fits-all for the hundreds of projects)

I hear you, the kind of thing that looks great on paper, but reminds me of the experience I had generating code with Apache Thrift.

Doing a tree walk to turn an existing non-unist AST into one that is unist compatible, is often easier and more maintainable than writing a parser.

@ChristianMurphy I am going off here on a complete super tangent: the article by the folks at pganalyze reminded me of what an integration with unist would look like: https://pganalyze.com/blog/pg-query-2-0-postgres-query-parser

Thank you two for checking this out over the weekend!

Talk soon.