TinyAst Guide

TinyAst Guide

TinyAst provides a syntax tree API for efficient AST manipulation with fluent queries and undo/redo support.

Parsing

Create a syntax tree from source text:

using TinyTokenizer.Ast;

// Simple parse
var tree = SyntaxTree.Parse("function foo() { return 1; }");

// Parse with schema (enables syntax node matching)
var tree = SyntaxTree.Parse(source, Schema.Default);

// Parse with custom options
var schema = Schema.Create()
    .WithOperators(CommonOperators.JavaScript)
    .Build();
var tree = SyntaxTree.Parse(source, schema);

Tree Structure

SyntaxNode Hierarchy

The public API uses "red" nodes — navigable wrappers with parent references:

SyntaxNode (abstract base)
├── SyntaxBlock — Contains children ({ }, [ ], ( ))
├── SyntaxToken — Terminal tokens (identifiers, operators, etc.)
└── SyntaxList — Root token list

Accessing the Root

var tree = SyntaxTree.Parse("foo(x)");
SyntaxNode root = tree.Root;

// Root is always a SyntaxList
if (root is SyntaxList list)
{
    Console.WriteLine($"Root has {list.Children.Count()} children");
}

Navigating Children

var tree = SyntaxTree.Parse("a + b");

foreach (var child in tree.Root.Children)
{
    Console.WriteLine($"{child.Kind} at {child.Position}");
}
// Output:
// Ident at 0
// Whitespace at 1
// Operator at 2
// Whitespace at 3
// Ident at 4

Parent Navigation

var tree = SyntaxTree.Parse("foo(bar)");

// Find a nested node
var bar = tree.Leaves.First(l => l.Text == "bar");

// Navigate up
SyntaxNode? parent = bar.Parent;      // The paren block
SyntaxNode? grandparent = parent?.Parent;  // The root

NodeKind

Every node has a Kind indicating its type:

Kind	Node Type	Description
Ident	SyntaxToken	Identifiers
Numeric	SyntaxToken	Number literals
String	SyntaxToken	Quoted strings
Operator	SyntaxToken	Multi-char operators (=>, ==, etc.)
Symbol	SyntaxToken	Single characters (., ,, ;)
TaggedIdent	SyntaxToken	Prefixed identifiers (#define, @attr)
Comment	SyntaxToken	Comments
Error	SyntaxToken	Parse errors
BraceBlock	SyntaxBlock	{ }
BracketBlock	SyntaxBlock	[ ]
ParenBlock	SyntaxBlock	( )
500-1999	SyntaxToken	Schema-defined keywords
2000+	varies	Schema-defined syntax nodes

Keywords receive unique NodeKind values when defined via Schema.DefineKeywords(). Use kind.IsKeyword() to check.

foreach (var node in tree.Root.Children)
{
    switch (node.Kind)
    {
        case NodeKind.Ident:
            Console.WriteLine($"Identifier: {((SyntaxToken)node).Text}");
            break;
        case NodeKind.BraceBlock:
            Console.WriteLine($"Block with {((SyntaxBlock)node).Children.Count()} children");
            break;
        default:
            if (node.Kind.IsKeyword())
                Console.WriteLine($"Keyword: {((SyntaxToken)node).Text}");
            break;
    }
}

Accessing Leaves

Get all leaf (terminal) nodes:

var tree = SyntaxTree.Parse("foo(x, y)");

// All leaves (including whitespace)
foreach (var leaf in tree.Leaves)
{
    Console.WriteLine($"{leaf.Kind}: '{leaf.Text}'");
}

// Filter to specific kinds
var idents = tree.Leaves.Where(l => l.Kind == NodeKind.Ident);  // [Leaf("foo"), Leaf("x"), Leaf("y")]

Querying Nodes

Use the Query API for powerful CSS-like selectors:

var tree = SyntaxTree.Parse("foo(x) + bar(y)");

// Find specific identifiers
var foos = tree.Select(Query.Ident("foo"));  // [Ident("foo")]

// Find all identifiers
var allIdents = tree.Select(Query.AnyIdent);  // [Ident("foo"), Ident("x"), Ident("bar"), Ident("y")]

// Find function calls (ident followed by parens)
var funcCalls = tree.Select(Query.AnyIdent.FollowedBy(Query.ParenBlock));  // [Ident("foo"), Ident("bar")]

Editing the Tree

Use SyntaxEditor for batched mutations:

tree.CreateEditor()
    .Replace(Query.Ident("foo"), "bar")
    .InsertAfter(Query.BraceBlock.First().Start(), "// entry\n")
    .Commit();

Undo/Redo

All edits support undo/redo:

var tree = SyntaxTree.Parse("hello");

tree.CreateEditor()
    .Replace(Query.Ident("hello"), "goodbye")
    .Commit();

Console.WriteLine(tree.Root.ToFullString()); // "goodbye"

tree.Undo();
Console.WriteLine(tree.Root.ToFullString()); // "hello"

tree.Redo();
Console.WriteLine(tree.Root.ToFullString()); // "goodbye"

Position and Width

Every node tracks its position in the source:

var tree = SyntaxTree.Parse("foo bar");
var bar = tree.Leaves.First(l => l.Text == "bar");

Console.WriteLine($"Position: {bar.Position}"); // 4
Console.WriteLine($"Width: {bar.Width}");       // 3
Console.WriteLine($"End: {bar.End}");           // 7

Converting to String

var tree = SyntaxTree.Parse("foo(x)");

// Full text including all whitespace
string full = tree.Root.ToFullString();

// Individual leaf text
foreach (var leaf in tree.Leaves)
{
    Console.WriteLine(leaf.Text);
}

Red-Green Architecture

TinyAst uses red-green trees (similar to Roslyn):

• Green nodes (internal) — Immutable, shared, position-agnostic
• Red nodes (public) — Navigable wrappers with parent references

This design enables:

• Memory efficiency — Shared subtrees
• Fast edits — Only rebuild affected paths
• Undo/redo — Swap root green node

See Architecture for details.

See Also

• Query API — CSS-like node selectors
• SyntaxEditor — Batch mutations
• TreeWalker — DOM-style traversal
• Trivia — Whitespace and comments
• Schema — Configuring parsing