Getting Started

Getting Started

Installation

Install TinyTokenizer via NuGet:

dotnet add package TinyAst

Or add a reference to the TinyTokenizer project in your solution.

Parsing with TinyAst

The primary API is SyntaxTree.Parse() which gives you a queryable, editable syntax tree:

using TinyTokenizer.Ast;

// Parse source into a syntax tree
var tree = SyntaxTree.Parse("function foo() { return 1; }");

// Access root and children
var root = tree.Root;
Console.WriteLine($"Root has {root.Children.Count()} children");

// Find all identifiers
var idents = tree.Leaves.Where(l => l.Kind == NodeKind.Ident);
foreach (var ident in idents)
{
    Console.WriteLine(ident.Text);
}

Querying Nodes

Use the Query static class for CSS-like selectors:

var tree = SyntaxTree.Parse("foo(x) + bar(y)");

// Find specific identifiers
var fooNodes = tree.Select(Query.Ident("foo"));  // [Ident("foo")]

// Find all identifiers
var allIdents = tree.Select(Query.AnyIdent);  // [Ident("foo"), Ident("x"), Ident("bar"), Ident("y")]

// Find function calls (ident followed by parens)
var funcCalls = tree.Select(Query.AnyIdent.FollowedBy(Query.ParenBlock));  // [Ident("foo"), Ident("bar")]

Editing the Tree

Make batched mutations with undo/redo support:

var tree = SyntaxTree.Parse("hello world");

tree.CreateEditor()
    .Replace(Query.Ident("hello"), "goodbye")
    .Replace(Query.Ident("world"), "universe")
    .Commit();

Console.WriteLine(tree.Root.ToFullString()); // "goodbye universe"

tree.Undo();
Console.WriteLine(tree.Root.ToFullString()); // "hello world"

With Schema (Pattern Matching)

Use a Schema to enable syntax node matching:

var tree = SyntaxTree.Parse("obj.method(x)", Schema.Default);

// Match structural patterns
var methodCalls = tree.Match<MethodCallSyntax>().ToList();  // [MethodCallSyntax { Object="obj", Method="method" }]

Low-Level Tokenization

For scenarios where you don't need a syntax tree, use direct tokenization:

using TinyTokenizer;

var tokens = "func(a, b)".TokenizeToTokens();

// tokens contains:
// - IdentToken("func")
// - BlockToken("(a, b)") with children:
//   - IdentToken("a")
//   - SymbolToken(",")
//   - WhitespaceToken(" ")
//   - IdentToken("b")

With Options

Configure tokenization behavior using TokenizerOptions:

var options = TokenizerOptions.Default
    .WithCommentStyles(CommentStyle.CStyleSingleLine, CommentStyle.CStyleMultiLine)
    .WithOperators(CommonOperators.CFamily);

var tokens = "x = 42; // comment".TokenizeToTokens(options);

Next Steps

• TinyAst Guide — Deep dive into the syntax tree API
• Query API — Master the query combinator system
• SyntaxEditor — Learn batch editing and undo/redo
• Schema — Configure parsing and syntax definitions