Universal Syntax Tree used by @unifiedjs
Clone or download
Latest commit 49032b9 Nov 18, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore Add markdown formatting Aug 21, 2018
code-of-conduct.md Add `code-of-conduct.md` Nov 17, 2017
contributing.md Move chat to spectrum Nov 3, 2018
logo.svg Update logo Nov 3, 2018
package.json Add formatting to test script Aug 31, 2018
readme.md Add definition of character Nov 18, 2018

readme.md

unist

Universal Syntax Tree.


unist is a specification for syntax trees. It has a big ecosystem of utilities in JavaScript for working with these trees. It’s implemented by several other specifications.

This document may not be released. See releases for released documents. The latest released version is 2.0.0.

Table of Contents

Introduction

This document defines a general-purpose format for syntax trees. Development of unist started in July 2015. This specification is written in a Web IDL-like grammar.

Syntax tree

Syntax trees are representations of source code or even natural language. These trees are abstractions that make it possible to analyse, transform, and generate code.

Syntax trees come in two flavours:

  • concrete syntax trees: structures that represent every detail (such as white-space in white-space insensitive languages)
  • abstract syntax trees: structures that only represent details relating to the syntactic structure of code (such as ignoring whether a double or single quote was used in languages that support both, such as JavaScript).

This specification can express both abstract and concrete syntax trees.

Where this specification fits

unist is not intended to be self-sufficient. Instead, it is expected that other specifications implement unist and extend it to express language specific nodes. For example, see projects such as mdast (for markdown), hast (for HTML), and nlcst (for natural language).

unist relates to JSON in that compliant syntax trees can be expressed completely in JSON. However, unist is not limited to JSON and can be expressed in other data formats, such as XML.

unist relates to JavaScript in that it has a rich ecosystem of utilities for working with compliant syntax trees in JavaScript. The five most used utilities combined are downloaded ten million times each month. However, unist is not limited to JavaScript and can be used in other programming languages.

unist relates to the unified, remark, rehype, and retext projects in that unist syntax trees are used throughout their ecosystems.

unist relates to the vfile project in that it accepts unist nodes for its message store, and that vfile can be a source file of a syntax tree.

Nodes

Syntactic units in unist syntax trees are called nodes, and implement the Node interface.

Node

interface Node {
  type: string
  data: Data?
  position: Position?
}

The type field is a non-empty string representing the variant of a node. This field can be used to determine the type a node implements.

The data field represents information from the ecosystem. The value of the data field implements the Data interface.

The position field represents the location of a node in a source document. The value of the position field implements the Position interface. The position field must not be present if a node is generated.

Specifications implementing unist are encouraged to define more fields. Ecosystems can define fields on Data.

Any value in unist must be expressible in JSON values: string, number, object, array, true, false, or null. This means that the syntax tree should be able to be converted to and from JSON and produce the same tree. For example, in JavaScript, a tree can be passed through JSON.parse(JSON.stringify(tree)) and result in the same tree.

Position

interface Position {
  start: Point
  end: Point
  indent: [number >= 1]?
}

Position represents the location of a node in a source file.

The start field of Position represents the place of the first character of the parsed source region. The end field of Position represents the place of the first character after the parsed source region, whether it exists or not. The value of the start and end fields implement the Point interface.

The indent field of Position represents the start column at each index (plus start line) in the source region, for elements that span multiple lines.

If the syntactic unit represented by a node is not present in the source file at the time of parsing, the node is said to be generated and it must not have positional information.

For example, if the following value was represented as unist:

alpha
bravo

…the first word (alpha) would start at line 1, column 1, offset 0, and end at line 1, column 6, offset 5. The line feed would start at line 1, column 6, offset 5, and end at line 2, column 1, offset 6. The last word (bravo) would start at line 2, column 1, offset 6, and end at line 2, column 6, offset 11.

Point

interface Point {
  line: number >= 1
  column: number >= 1
  offset: number >= 0?
}

Point represents one place in a source file.

The line field (1-indexed integer) represents a line in a source file. The column field (1-indexed integer) represents a column in a source file. The offset field (0-indexed integer) represents a character in a source file.

The term character means a (UTF-16) code unit which is defined in the Web IDL specification.

Data

interface Data { }

Data represents information associated by the ecosystem with the node.

This space is guaranteed to never be specified by unist or specifications implementing unist.

Parent

interface Parent <: Node {
  children: [Node]
}

Nodes containing other nodes (said to be children) extend the abstract interface Parent (Node).

The children field is a list representing the children of a node.

Literal

interface Literal <: Node {
  value: any
}

Nodes containing a value extend the abstract interface Literal (Node).

The value field can contain any value.

Glossary

Tree

A tree is a node and all of its descendants (if any).

Child

Node X is child of node Y, if Y’s children include X.

Parent

Node X is parent of node Y, if Y is a child of X.

Index

The index of a child is its number of preceding siblings, or 0 if it has none.

Sibling

Node X is a sibling of node Y, if X and Y have the same parent (if any).

The previous sibling of a child is its sibling at its index minus 1.

The next sibling of a child is its sibling at its index plus 1.

Root

The root of a node is itself, if without parent, or the root of its parent.

The root of a tree is any node in that tree without parent.

Descendant

Node X is descendant of node Y, if X is a child of Y, or if X is a child of node Z that is a descendant of Y.

An inclusive descendant is a node or one of its descendants.

Ancestor

Node X is an ancestor of node Y, if Y is a descendant of X.

An inclusive ancestor is a node or one of its ancestors.

Head

The head of a node is its first child (if any).

Tail

The tail of a node is its last child (if any).

Leaf

A leaf is a node with no children.

Branch

A branch is a node with one or more children.

Generated

A node is generated if it does not have positional information.

Type

The type of a node is the value of its type field.

Positional information

The positional information of a node is the value of its position field.

File

A file is a source document that represents the original file that was parsed to produce the syntax tree. Positional information represents the place of a node in this file. Files are provided by the host environment and not defined by unist.

For example, see projects such as vfile.

Utilities

Utilities are functions that work with nodes.

There are several projects that deal with nodes from specifications implementing unist:

List of Utilities

References

Contribute

unist is built by people just like you! Check out contributing.md for ways to get started.

This project has a Code of Conduct. By interacting with this repository, organisation, or community you agree to abide by its terms.

Want to chat with the community and contributors? Join us in spectrum!

Have an idea for a cool new utility or tool? That’s great! If you want feedback, help, or just to share it with the world you can do so by creating an issue in the syntax-tree/ideas repository!

Acknowledgments

The initial release of this project was authored by @wooorm.

Special thanks to @eush77 for their work, ideas, and incredibly valuable feedback!

Thanks to @anandthakker, @anko, @arobase-che, @azu, @BarryThePenguin, @ben-eb, @blahah, @ChristianMurphy, @derhuerst, @dozoisch, @eush77, @fazouane-marouane, @gibson042, @ikatyang, @izumin5210, @jasonLaster, @JDvorak, @jlevy, @justjake, @kmck, @kt3k, @KyleAMathews, @muraken720, @mrzmmr, @nwtn, @rhysd, @Rokt33r, @Sarah-Seo, @sethvincent, @simov, @staltz, @tmcw, and @vhf, for contributing to unist and related projects!

License

Copyright © Titus Wormer. This work is licensed under a Creative Commons Attribution 4.0 International License.