Permalink
Switch branches/tags
v7.4.0 v7.3.0 v7.2.2 v3.6.0 v3.5.0 v3.3.1 v3.3.0 v3.1.2 v3.1.1 txt-to-ast@3.0.3 txt-to-ast@3.0.2 txt-to-ast@3.0.1 txt-to-ast@3.0.0 txt-to-ast@3.0.0-next.0 txt-to-ast@2.0.0 txt-to-ast@2.0.0-beta.0 txt-ast-traverse@2.0.4 txt-ast-traverse@2.0.3 txt-ast-traverse@2.0.2 txt-ast-traverse@2.0.1 txt-ast-traverse@2.0.0 txt-ast-traverse@2.0.0-next.0 textlint@11.0.1 textlint@11.0.0 textlint@10.2.1 textlint@10.2.0 textlint@10.1.5 textlint@10.1.4 textlint@10.1.3 textlint@10.1.2 textlint@10.1.1 textlint@10.1.0 textlint@10.0.1 textlint@10.0.0 textlint@10.0.0-next.2 textlint@10.0.0-next.1 textlint@10.0.0-next.0 textlint@9.1.1 textlint@9.1.0 textlint@9.0.1 textlint@9.0.0 textlint@9.0.0-beta.0 textlint@8.2.1 textlint@8.2.0 textlint@8.1.0 textlint@8.0.1 textlint@8.0.0 @textlint/textlint-plugin-text@4.0.1 @textlint/textlint-plugin-text@4.0.0 @textlint/textlint-plugin-text@3.0.10 @textlint/textlint-plugin-text@3.0.9 @textlint/textlint-plugin-text@3.0.8 @textlint/textlint-plugin-text@3.0.7 @textlint/textlint-plugin-markdown@5.0.1 @textlint/textlint-plugin-markdown@5.0.0 @textlint/textlint-plugin-markdown@4.0.10 @textlint/textlint-plugin-markdown@4.0.9 @textlint/textlint-plugin-markdown@4.0.8 @textlint/textlint-plugin-markdown@4.0.7 @textlint/text-to-ast@3.0.9 @textlint/text-to-ast@3.0.8 @textlint/text-to-ast@3.0.7 @textlint/text-to-ast@3.0.6 @textlint/text-to-ast@3.0.5 @textlint/text-to-ast@3.0.4 @textlint/monorepo-scripts@2.0.0 @textlint/monorepo-scripts@2.0.0-beta.0 @textlint/markdown-to-ast@6.0.9 @textlint/markdown-to-ast@6.0.8 @textlint/markdown-to-ast@6.0.7 @textlint/markdown-to-ast@6.0.6 @textlint/markdown-to-ast@6.0.5 @textlint/markdown-to-ast@6.0.4 @textlint/linter-formatter@3.0.8 @textlint/linter-formatter@3.0.7 @textlint/linter-formatter@3.0.6 @textlint/linter-formatter@3.0.5 @textlint/linter-formatter@3.0.4 @textlint/linter-formatter@3.0.3 @textlint/kernel@3.0.0 @textlint/kernel@2.0.9 @textlint/kernel@2.0.8 @textlint/kernel@2.0.7 @textlint/kernel@2.0.6 @textlint/kernel@2.0.5 @textlint/kernel@2.0.4 @textlint/kernel@2.0.3 @textlint/kernel@2.0.2 @textlint/kernel@2.0.1 @textlint/kernel@2.0.0 @textlint/kernel@2.0.0-next.2 @textlint/kernel@2.0.0-next.1 @textlint/kernel@2.0.0-next.0 @textlint/kernel@1.0.3 @textlint/kernel@1.0.2 @textlint/kernel@1.0.1 @textlint/kernel@1.0.0 @textlint/kernel@1.0.0-beta.0 @textlint/kernel@0.2.1 @textlint/kernel@0.2.0
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
391 lines (303 sloc) 10.2 KB
id title
txtnode
TxtAST Interface

TxtAST define AST(Abstract Syntax Tree) for processing in textlint.

What is AST?

Abstract syntax tree is a tree representation of the abstract syntactic structure of text.

Each node of the tree has same interface, is called TxtNode.

ast-explorer fork

AST explorer for textlint is useful for understanding AST.

TxtNode

TxtNode is a abstract node.

/**
 * Basic TxtNode
 * Probably, Real TxtNode implementation has more properties.
 */
interface TxtNode {
    type: string;
    raw: string;
    range: TextNodeRange;
    loc: TxtNodeLineLocation;
    // parent is runtime information
    // Not need in AST
    // For example, top Root Node like `Document` has not parent.
    parent?: TxtNode;

    [index: string]: any;
}


/**
 * Location
 */
interface TxtNodeLineLocation {
    start: TxtNodePosition;
    end: TxtNodePosition;
}

/**
 * Position's line start with 1.
 * Position's column start with 0.
 * This is for compatibility with JavaScript AST.
 * https://gist.github.com/azu/8866b2cb9b7a933e01fe
 */
interface TxtNodePosition {
    line: number; // start with 1
    column: number; // start with 0
}

/**
 * Range start with 0
 */
type TextNodeRange = [number, number];

TxtNode must have these properties.

  • type: type of Node
  • raw: raw value of Node
    • if you want to get raw value, please use getSource(<node>) instead of it..
  • loc: location object
  • range: location info array like [startIndex, endIndex]
  • parent: (optional) parent node of this node.
    • It is attached in runtime
    • Parser user ignore this property

TxtTextNode

TxtTextNode is inherit the TxtNode abstract interface.

/**
 * Text Node.
 * Text Node has inline value.
 * For example, `Str` Node is an TxtTextNode.
 */
interface TxtTextNode extends TxtNode {
    value: string;
}

TxtTextNode must have these properties.

  • value: the value of inline node.

Example: Str node is a TxtTextNode.

TxtParentNode

TxtParentNode is inherit the TxtNode abstract interface.

/**
 * Parent Node.
 * Parent Node has children that are consist of TxtNode or TxtTextNode
 */
interface TxtParentNode extends TxtNode {
    children: Array<TxtNode | TxtTextNode>;
}

TxtParentNode must have these properties.

  • children: child nodes of this node.

Example: Paragraph node is a TxtParentNode.

type

type is TxtNode type.

All Types are defined in @textlint/ast-node-types. You can use this ASTNodeTypes value via following way:

import { ASTNodeTypes } from "@textlint/ast-node-types";
console.log(ASTNodeTypes.Str); // "Str"

See packages/ast-node-types for more details.

These types are be available at all times:

{
    // TxtParentNode
    "Document": "Document",
    "Paragraph": "Paragraph",
    "BlockQuote": "BlockQuote",
    "ListItem": "ListItem",
    "List": "List",
    "Header": "Header",
    "CodeBlock": "CodeBlock",
    "HtmlBlock": "HtmlBlock",
    "ReferenceDef": "ReferenceDef",
    "HorizontalRule": "HorizontalRule",
    "Comment": "Comment",
    // TxtTextNode
    "Str": "Str",
    "Break": "Break",
    "Emphasis": "Emphasis",
    "Strong": "Strong",
    "Html": "Html",
    "Link": "Link",
    "Image": "Image",
    "Code": "Code"
}

The type is based on HTML tag.

Minimal node property

TxtAST allow to extend node property. But, Following node should have some properties.

Header

  • depth: level of header
    • Example: <h1> is depth:1, <h2> is depth:2...

Link

  • url: link url

Image

  • url: image url

Built-in Parser

textlint has built-in parsers.

Package Version Description
@textlint/markdown-to-ast-to-ast npm markdown parser
@textlint/text-to-ast npm plain text parser

If you want to get other type, please create new issue.

Package

That TxtNode interface is defined in packages/ast-node-types.

If you want to use this interface from TypeScript, packages/ast-node-types is useful.

Online Parsing Demo

ast-explorer fork

AST explorer for textlint is useful for understanding AST.

Minimum(recommended) rules is following code:

/**
 * @param {RuleContext} context
 */
export default function(context) {
    const { Syntax } = context;
    // root object
    return {
        [Syntax.Document](node) {},
        [Syntax.Paragraph](node) {},
        [Syntax.Str](node) {}
    };
}

loc

loc is location info object.

"loc": {
    "start": {
        "line": 2,
        "column": 4
    },
    "end": {
        "line": 2,
        "column": 10
    }
}
  • line of location start with 1 (1-indexed).
  • column of location start with 0 (0-indexed).

This is for compatibility with JavaScript AST.

Important Note:

Text -> AST TxtNode(0-based columns here) -> textlint -> TextLintMessage(1-based columns)

TxtNode has 0-based columns, but the result of linting named TextLintMessage has 1-based columns.

In other word, textlint's rule handle TxtNode, but formatter handle TextLintMessage.

Example

Input: *text*

Output: The AST by AST explorer for textlint + Markdown

{
    "type": "Document",
    "children": [
        {
            "type": "Paragraph",
            "children": [
                {
                    "type": "Emphasis",
                    "children": [
                        {
                            "type": "Str",
                            "value": "text",
                            "loc": {
                                "start": {
                                    "line": 1,
                                    "column": 1
                                },
                                "end": {
                                    "line": 1,
                                    "column": 5
                                }
                            },
                            "range": [
                                1,
                                5
                            ],
                            "raw": "text"
                        }
                    ],
                    "loc": {
                        "start": {
                            "line": 1,
                            "column": 0
                        },
                        "end": {
                            "line": 1,
                            "column": 6
                        }
                    },
                    "range": [
                        0,
                        6
                    ],
                    "raw": "*text*"
                }
            ],
            "loc": {
                "start": {
                    "line": 1,
                    "column": 0
                },
                "end": {
                    "line": 1,
                    "column": 6
                }
            },
            "range": [
                0,
                6
            ],
            "raw": "*text*"
        }
    ],
    "loc": {
        "start": {
            "line": 1,
            "column": 0
        },
        "end": {
            "line": 1,
            "column": 6
        }
    },
    "range": [
        0,
        6
    ],
    "raw": "*text*"
}

Illustration

          *   text   *
          |   |__|   |
          |   value  |
          |__________|
               raw
  • Document is a TxtParentNode and type is Document
    • have children, but not have value
  • Paragraph is a TxtParentNode and type is Paragraph
    • have children, but not have value
  • Emphasis is a TxtTextNode and type is Emphasis
    • have value
  • "text" is a TxtTextNode and type is Str
    • have value

Unist

TxtAST have a minimum of compatibility for unist: Universal Syntax Tree.

We discuss about Unist in Compliances tests for TxtNode #141.

For testing Processor plugin

You can use @textlint/ast-tester for testing your processor plugin's parser.

import { test, isTextlintAST } from "@textlint/ast-tester";
// your implement
import yourParse from "your-parser";
// recommenced: test much pattern test
const AST = yourParse("This is text");

// Validate AST
test(AST); // if the AST is invalid, then throw Error

isTextlintAST(AST); // true or false

Warning

Other properties is not assured.

For example, markdown's Header node has level property, but other format has not it.