API Reference

API Reference

Quick reference for core types, enums, and configuration options.

Core Types

Tokenization

Type	Description
Lexer	Level 1 — Stateless character classifier
TokenParser	Level 2 — Combines simple tokens into semantic tokens
Tokenizer	Combined Level 1 + 2 (legacy, prefer two-level API)
TokenizerOptions	Configuration for tokenization

Tokens

Type	Description
SimpleToken	Level 1 struct — atomic character classifications
Token	Level 2 abstract base — semantic tokens
IdentToken	Identifiers
WhitespaceToken	Spaces, tabs, newlines
SymbolToken	Single-character symbols
OperatorToken	Multi-character operators
NumericToken	Number literals
StringToken	Quoted strings
TaggedIdentToken	Prefixed identifiers (#define, @attr)
CommentToken	Comments
BlockToken	Delimited blocks with children
ErrorToken	Parse errors

TinyAst

Type	Description
SyntaxTree	Main entry point — parse, query, edit
RedNode	Abstract navigable node
RedLeaf	Terminal token node
RedBlock	Block node with children
SyntaxEditor	Batch mutations
TreeWalker	DOM-style traversal
Schema	Unified configuration
Query	Static factory for node queries
SyntaxNode	Base for pattern-matched nodes
Trivia	Whitespace/comment info

TokenType Enum

public enum TokenType
{
    Ident,
    Whitespace,
    Symbol,
    Operator,
    Numeric,
    String,
    TaggedIdent,
    Comment,
    Block,
    Error
}

NodeKind Enum

public enum NodeKind
{
    Ident,
    Whitespace,
    Symbol,
    Operator,
    Numeric,
    String,
    TaggedIdent,
    Comment,
    BraceBlock,
    BracketBlock,
    ParenBlock,
    Error
}

NumericType Enum

public enum NumericType
{
    Integer,       // 123
    FloatingPoint  // 3.14, .5
}

BlockTokenType Enum

public enum BlockTokenType
{
    Brace,    // { }
    Bracket,  // [ ]
    Paren     // ( )
}

TriviaKind Enum

public enum TriviaKind
{
    Whitespace,
    Newline,
    SingleLineComment,
    MultiLineComment
}

FilterResult Enum

public enum FilterResult
{
    Accept,  // Include node in results
    Reject,  // Exclude node and entire subtree
    Skip     // Exclude node, still visit children
}

NodeFilter Flags

[Flags]
public enum NodeFilter
{
    All = Leaves | Blocks,
    Leaves = 1,
    Blocks = 2
}

CommonOperators

Pre-defined operator sets:

Set	Contents
Universal	==, !=, &&, ||, <=, >=, <<, >>
CFamily	Universal + ++, --, ->, ::, ...
JavaScript	CFamily + ===, !==, =>, ?., ??, **
Rust	CFamily + =>, .., ..=, ::
Python	Universal + **, //, :=, @

var options = TokenizerOptions.Default
    .WithOperators(CommonOperators.JavaScript);

CommentStyle

Pre-defined comment styles:

Style	Start	End
CStyleSingleLine	//	newline
CStyleMultiLine	/*	*/
HashSingleLine	#	newline

// Built-in
var options = TokenizerOptions.Default
    .WithCommentStyles(CommentStyle.CStyleSingleLine);

// Custom single-line
var lua = new CommentStyle("--");

// Custom multi-line
var html = new CommentStyle("<!--", "-->");

Token Format Specifiers

Specifier	Output	Example
G	Content (default)	foo
T	Type name	IdentToken
P	Position	42
R	Range	42..45
D	Debug	Ident[42..45]

Console.WriteLine($"{token:D}");  // "Ident[0..3]"

Extension Methods

String Extensions

// Basic tokenization
IEnumerable<Token> tokens = "source".TokenizeToTokens();
IEnumerable<Token> tokens = "source".TokenizeToTokens(options);

Stream Extensions

// Async tokenization
Task<ImmutableArray<Token>> tokens = stream.TokenizeAsync();
Task<ImmutableArray<Token>> tokens = stream.TokenizeAsync(options);

// Streaming
IAsyncEnumerable<Token> tokens = stream.TokenizeStreamingAsync();
IAsyncEnumerable<Token> tokens = stream.TokenizeStreamingAsync(options);

PipeReader Extensions

IAsyncEnumerable<Token> tokens = pipeReader.TokenizeStreamingAsync();
IAsyncEnumerable<Token> tokens = pipeReader.TokenizeStreamingAsync(options, encoding);

SyntaxTree Methods

// Parsing
var tree = SyntaxTree.Parse(source);
var tree = SyntaxTree.Parse(source, schema);

// Properties
RedBlock root = tree.Root;
IEnumerable<RedLeaf> leaves = tree.Leaves;
bool hasSchema = tree.HasSchema;

// Querying
IEnumerable<RedNode> nodes = tree.Select(query);
IEnumerable<T> matches = tree.Match<T>();  // Requires schema

// Editing
SyntaxEditor editor = tree.CreateEditor();
tree.Undo();
tree.Redo();

// Schema
SyntaxTree bound = tree.WithSchema(schema);

// Traversal
TreeWalker walker = tree.CreateTreeWalker();

Query Methods

See Query API for complete reference.

// Named queries
Query.Ident("main")
Query.Symbol(".")
Query.Operator("=>")

// Any-kind queries
Query.AnyIdent
Query.AnyBlock
Query.ParenBlock

// Combinators
query.First()
query.Where(predicate)
query.FollowedBy(other)
query.Then(other)
Query.Sequence(a, b, c)

// Positions
query.Before()
query.After()
query.InnerStart()
query.InnerEnd()

SyntaxEditor Methods

editor.Insert(position, text)
editor.InsertBefore(node, text)
editor.InsertAfter(node, text)
editor.Replace(target, text)
editor.Replace(target, transformer)
editor.Edit(target, contentTransformer)
editor.Remove(target)
editor.Commit()

Requirements

• .NET 8.0 or later
• CommunityToolkit.HighPerformance (automatically included)

Limitations

• Maximum file size: ~2GB (positions stored as int)

See Also

• Getting Started — Quick introduction
• Architecture — Design overview
• Configuration — TokenizerOptions details