JSON or YAML syntax trees

[marbemac]

Have you all considered this use case? Would be really useful to be able to plug JSON into the unified ecosystem. I was beginning to look through https://github.com/vtrushin/json-to-ast until I thought, why not unified!?

[wooorm]

I think JSON (and potentially JS) is super interesting!
YAML is very advanced, though, so I think that’ll be a lot of work!
There’s also the question if it should be an AST or a CST (the latter contains info on existing white-space, so that’s useful if you’d like to keep the same style).

What do you plan on doing with the syntax tree?

[marbemac]

True, worst case YAML could be converted to JSON first using js-yaml or similar.

Good question, for our purposes we probably would not need a CST, but I could totally see the benefits.

I work over @stoplightio. We're building the next version of our platform, and it includes a full blown IDE for specifications - basically a visual designer + coding environment for specs like, Swagger, Markdown, JSON Schema, protobuf files, etc. I'm evaluating the current landscape of text processing tools and I gotta say, the unified ecosystem looks fantastic - great work!

In particular, we need a unified AST for all of the specifications we might want to handle, so that the rest of our tooling has a single object shape to work with for a few reasons:

1. The specifications we are starting with (Swagger, JSON Schema, and a few others) are all JSON based, and I think it would be pretty cool to build unified-powered linters for them.
2. In addition, we need easy access to the line/column position of a specific JSON path so that we can show the results of said linter alongside the right lines in the code.
3. When we start working with files that are not already in JSON (protobuf files, for example), we'll need a good text processing solution, and a structured JSON ast to transform it into.

There are a few other reasons, but I'll stop there since I've already written a wall of text :). Pretty excited about what you've built, and thought I'd include a bit more information to see if you think the use case is a decent fit for unified.

Incidentally, the json-to-ast ast is remarkably (no pun intended) close to unist. Looks like just a change of "loc" to "position", removal of the "loc.source" property, and lowercasing the "type" property. Am I missing anything there?

https://astexplorer.net/#/gist/a86d9887a38ec82dac032b30f0588aa1/d07774ad33f9dc404560db2ada38c6cbc082b96f

[marbemac]

Thinking about it a bit more, we'd probably have to create the below. For fast+refile, I'm not sure if work has already been done that overlaps in unified-engine, or the vfile ecosystem? This is all a bit of a different use case for unified then what's been done so far, but it feels like the possibilities could be awesome. OR I could be totally off-base here, and perhaps this sort of thing is better farmed off to individual validators and linters that are already built (like jsonlint, or the various swagger validation libraries that are really not very good unfortunately). You've got much more experience in this space than I, I'm all ears :).

• jast+rejson

  • pluggable, to add typings and extra info if needed to resulting jast tree. A couple of example flavors:
    • rejson-spec-oas2 (swagger 2)
    • rejson-spec-oas3
    • rejson-spec-jschema (json schema)
  • linters
    • rejson-lint-oas2 (swagger 2)
    • rejson-lint-oas3
    • rejson-lint-jschema

• fast+refile

  • helpers to create unist trees from common file system constructs

[j-f1]

Prettier would probably be interested in a YAML parser that provides access to comments (prettier/prettier#3017) so that they would be able to add full support for YAML.

[wooorm]

JSON

Incidentally, the json-to-ast ast is remarkably (no pun intended) close to unist. Looks like just a change of "loc" to "position", removal of the "loc.source" property, and lowercasing the "type" property. Am I missing anything there?

Probably not! I suggest creating something like rehype-parse for JSON, wrapping json-to-ast. You’d also need a compiler, but I think that wouldn’t take a lot of work?

Thinking about it a bit more, we'd probably have to create the below.

We could use reson as a name for the JSON processor, it’s still free, and more pronounceable than rejson

Files

For fast+refile, [...]

Could you expand a bit on what you’d envision this to be?

There’s one plugin, unified-diff, which works on a file, regardless of tree.
There’s also some rules in remark-lint that check files (search for file- and no-file-).

YAML

Prettier would probably be interested in a YAML parser that provides access to comments (prettier/prettier#3017) so that they would be able to add full support for YAML.

I know that the folks behind js-yaml would also like a markdown CST for markdown-it (that potentially remark could use as well), so maybe their interest in YAML and comments changed since 2015?!

YAML is super interesting, but I personally don’t have time to create a YAML parser, so we’d need someone to lead that development, or some company to sponsor it!

[wooorm]

If someone is interested in doing this, I can help out / mentor!

/cc @marbemac, @j-f1, someone else?

[wtcross]

I see huge potential for creation of utilities to provide deep static analysis of files for tools like Kubernetes and Ansible. Having a (hypothetical) re-yaml project would enable creating transformers to domain-aware ASTs. This would enable not only static analysis, but also robust documentation creation automation.

If somebody is interested in working on this and priority is the issue I might be able to help find a sponsor for funding.

[Murderlon]

Some of the hard work to support YAML in unified has already been done, the parser: https://github.com/ikatyang/yaml-unist-parser.

As far as I know, you still need:

• a compiler to stringify the YAML AST (such as remark-stringify).

And optionally:

• utilities to operate on the YAML specific AST.
• a wrapper to bring it together such as remark (remark-parse + remark-stringify).
• plugins

[wooorm]

@wtcross As I see you work on Ansible, could you expand more on what you’d be interested in?

[ChristianMurphy]

Thanks for starting the discussion @marbemac!
We're in the process unifying ideas in with discussions unifiedjs/collective#44
If you'd like to continue this thread, or start a new one https://github.com/syntax-tree/unist/discussions/categories/ideas will be the home for ideas going forward.