v0.11.3

• Fixed bug where SyntaxBinder would duplicate surrounding data for nested nodes which contained children.

v0.11.2

Release Notes (TinyTokenizer / TinyAst)

• Query.Between now defaults to exclusive

  • Query.Between(start, end) matches only the content between the start/end patterns (delimiters excluded).
  • To include delimiters, use Query.Between(start, end, inclusive: true).

• More reliable BetweenQuery behavior in compositions

  • BetweenQuery now consumes a stable start→end span for matching (safer in Sequence/composed queries), while editor operations still respect exclusive vs inclusive region boundaries.

• Improved region selection performance for offset-based ranges

  • Region traversal gained support for “offset regions” (regions that start after the matched node), enabling efficient selection for queries like BetweenQuery without falling back to heavier traversal.

• New convenience APIs for working from existing nodes

  • Query.Wrap(SyntaxBlock block) returns a BlockNodeQuery, enabling .Inner() / .Start() / .End() on an existing block node.
  • Query.Wrap(SyntaxNode node) returns a block query for blocks or an exact-node query otherwise.
  • Query.Inner(SyntaxBlock block) is shorthand for Query.Wrap(block).Inner().

Upgrade Note

• If you previously relied on Query.Between(start, end) including delimiters, change calls to Query.Between(start, end, inclusive: true).

v0.11.1

Release Notes — Newline Query + Query Engine Optimization

• Query.Newline is now explicitly defined as “node occurs after a newline” where a match is true if the node has leading newline trivia OR the previous sibling has trailing newline trivia.
• Newline detection is faster and allocation-free in green matching by using token-centric boundary flags (HasLeadingNewlineTrivia / HasTrailingNewlineTrivia) rather than scanning trivia.
• Green-node flag semantics are clarified and enforced: boundary trivia flags are token-owned; container/list nodes do not inherit boundary flags from their first/last child (they only aggregate Contains* subtree flags).
• Query engine hot paths were tightened by removing LINQ from selection modes and commonly used combinators/Select(...) implementations, reducing iterator/closure allocations while preserving document order/semantics.
• SyntaxEditor region resolution was optimized to materialize NodePath only on match (less allocation during region-heavy edits).
• Test coverage expanded for newline semantics and for green-flag correctness across editor mutations + undo/redo.
• Added a targeted benchmark suite for SyntaxTree.Select(Query.Newline); benchmark runner is back to discovering/running all benchmarks by default (filters still supported).

Performance (NewlineQueryBenchmarks, current vs master, fixed warmup/iterations/outlier removal)

• Select(Query.Newline) - Count: ~5–10% faster, allocations essentially unchanged
• Select(Query.Newline.First()) - First match: ~14–15% faster, ~6–7% less allocated

v0.11

Release Notes: TinyAst rename

Highlights

• Introduces TinyAst as the new NuGet package name for this project.
• This is a packaging/branding transition only: the public API remains in the existing namespaces (TinyTokenizer / TinyTokenizer.Ast) for compatibility.

Install / Upgrade

• New installs should use:
  • dotnet add package TinyAst
• Existing users:
  • Replace TinyTokenizer package references with TinyAst.
  • No code changes are required for this release (namespaces stay the same).

Compatibility

• Assembly name remains TinyTokenizer and namespaces remain TinyTokenizer.* to minimize runtime and source breakage.
• A future major release will migrate namespaces to TinyAst.* (planned).

Deprecation Notice

• The TinyTokenizer NuGet package will be marked deprecated and will stop receiving updates. Use TinyAst going forward.

Validation

• Build and tests were run in Release configuration; all tests passed.

v0.10.1

Release Notes — v0.11

🐛 Bug Fixes

Keyword Query Now Matches Correctly

Query.Keyword("if") now correctly matches only the "if" keyword, not all keywords in the same category. Previously, using Query.Keyword() in syntax definitions would incorrectly match any keyword from the same category due to an internal matching bug.

// Now works as expected
var schema = Schema.Create()
    .DefineKeywordCategory("ControlFlow", "if", "else", "while", "for")
    .DefineSyntax<IfStatement>("IfStatement", b => b
        .Pattern(p => p.Keyword("if").ParenBlock().BraceBlock()))
    .Build();

// Only matches "if", not "else", "while", or "for"
var ifStatements = tree.Match<IfStatement>();

Block Trivia Handling Improved

Fixed an issue where indentation after a block opener's newline was incorrectly dropped instead of being attached to the first child element. This improves round-trip fidelity when editing code with SyntaxEditor.

---

⚠️ Behavioral Change

If your code was unintentionally relying on Query.Keyword("x") matching multiple keywords, you'll need to update:

Old Behavior	New Behavior	Migration
Query.Keyword("int") matched all type keywords	Matches only "int"	Use Query.KeywordCategory("Types") for category matching
Worked without schema	Returns empty without schema	Ensure tree has schema via SyntaxTree.Parse(source, schema)

---

📝 Notes

• Query.Keyword() now requires a schema to be attached to the tree
• Schemaless trees return empty results (no exceptions thrown)
• No changes to public API signatures

v0.10

Release Notes: v0.10.0

Release Date: January 6, 2026
Previous Version: v0.9 (January 4, 2026)

Overview

This release contains significant improvements to the SyntaxEditor and Query system, fixing fundamental issues with multi-node replacements and range queries. The internal architecture has been refactored to properly handle queries that match multiple consecutive siblings.

🚀 New Features

Query Region System

A new internal QueryRegion abstraction enables queries to correctly resolve to contiguous ranges of slots within parent containers. This powers correct behavior for:

• Query.Between(start, end) - Now correctly replaces/removes/inserts around entire matched ranges
• Block boundary queries (.Start(), .End(), .Inner()) - Properly resolve to slot positions for accurate editing
• Sequence queries - Multi-node matches are handled atomically

Block Boundary API Improvements

The IBlockContainerNode interface now uses a cleaner API:

// Old API (removed)
Query.Syntax<FunctionSyntax>().InnerStart("body")
Query.Syntax<FunctionSyntax>().InnerEnd("body")

// New API
Query.Syntax<FunctionSyntax>().Block("body").Start()
Query.Syntax<FunctionSyntax>().Block("body").End()
Query.Syntax<FunctionSyntax>().Block("body").Inner()

Green Node Trivia Access

New methods on GreenNode for efficient trivia access:

• GetLeadingTrivia() / GetTrailingTrivia() - Access trivia via first/last leaf descendant
• GetLeadingTriviaWidth() / GetTrailingTriviaWidth() - Precomputed width access
• GetFirstLeaf() / GetLastLeaf() - Navigate to boundary leaves in O(depth)

⚡ Performance Improvements

PathTrackingWalker

A new internal PathTrackingWalker tracks paths incrementally during tree traversal:

• O(1) per step instead of O(depth) for NodePath.FromNode()
• Pushes sibling index on descent, pops on ascent
• Used by all SelectRegions implementations

Lazy Path Resolution

QueryRegion.ParentPath is computed lazily on first access, avoiding tree-walking overhead when regions are enumerated but not all are used for editing.

Optimized Trivia Extraction

• SyntaxEditor.GetTrivia() now delegates to green node methods (2 lines vs 30+ lines)
• Uses precomputed LeadingTriviaWidth/TrailingTriviaWidth instead of runtime summation
• GetContentWithoutTrivia() uses span-based slicing

Inlined Query Matching

• KindNodeQuery.SelectRegions inlines Kind check and predicate evaluation
• BlockNodeQuery.SelectRegions inlines block type check
• Avoids virtual TryMatch dispatch in hot loops

🐛 Bug Fixes

Fixed: Query.Between with SyntaxEditor.Replace()

Previously, Replace() with range queries like Query.Between(start, end) would only replace the first node. Now the entire matched range is replaced atomically.

Fixed: Block Inner Content Replacement

Replacing content inside empty blocks (e.g., { }) now works correctly. The zero-width inner region is properly resolved.

Fixed: Multi-Position Insertions

InsertAfter and InsertBefore with queries matching multiple positions now correctly insert at all matched positions.

📊 New Benchmarks

Added SyntaxEditorBenchmarks.cs with comprehensive benchmarks:

Category	Benchmarks
Replace	Replace_SingleToken, Replace_MultipleTokens, Replace_AllIdents
BlockInner	Replace_BlockInner_Empty, Replace_BlockInner_WithContent, Replace_BlockInner_Larger
Insert	InsertAfter_SinglePosition, InsertAfter_ManyPositions, InsertBefore_ManyPositions
SelectRegions	SelectRegions_DeepTree (20/50 levels), SelectRegions_DeepTree_Blocks, SelectRegions_DeepTree_First
Transform	Edit_Transform_SingleToken, Replace_WithNodeTransformer
UndoRedo	Edit_ThenUndo, MultipleEdits_ThenUndoAll

🔧 Breaking Changes

None for public API. All changes are internal implementation details.

Internal API Changes (for library contributors)

• Removed InsertionQuery and InsertionPoint enum - replaced by BoundaryQuery + IRegionQuery
• IBlockContainerNode.InnerStart()/InnerEnd() replaced by .Block().Start()/.End()
• Query types now implement IRegionQuery interface

📁 Files Changed

+2,751 / -891 lines across 21 files

New Files:
  QueryRegion.cs (245 lines)
  SyntaxEditorBenchmarks.cs (375 lines)

Major Changes:
  SyntaxEditor.cs (+530 lines refactored)
  NodeQueryTypes.cs (+386 lines)
  QueryCombinators.cs (+181 lines)
  GreenBlock.cs (+203 lines)
  IBlockContainerNode.cs (+171 lines)
  GreenNode.cs (+88 lines)

🧪 Test Coverage

• 15+ new tests for Query.Between with all editor operations
• Block inner content replacement tests (empty and non-empty)
• Deep tree traversal tests (20-50 levels)
• Multi-position insertion tests

Upgrade Guide

No action required for most users. If you were using internal APIs:

// If you implemented IBlockContainerNode, update usage:
// Before:
editor.InsertAfter(query.InnerStart("body"), "code")

// After:
editor.InsertAfter(query.Block("body").Start(), "code")

v0.9

Release Notes: TinyTokenizer v0.9

🐛 Bug Fixes

Fixed Double-Binding Bug in SyntaxBinder

Fixed a bug where GreenSyntaxNode instances were incorrectly double-wrapped during rebinding after tree mutations. This caused "Expected child at slot 0 to be X, but got SyntaxNode" errors when re-querying syntax nodes after multiple edit/commit cycles.

Affected scenario: Parsing with a schema, making an edit, committing, then re-querying for previously found syntax nodes would fail because the binding system would re-wrap already-bound nodes.

Fix: SyntaxBinder.BindNode() now detects already-bound GreenSyntaxNode instances and skips pattern matching while still recursively binding their children.

🆕 Keyword Categorization System

TinyTokenizer now supports schema-defined keywords that are recognized during lexing and assigned unique NodeKind values. This enables language-aware parsing with proper keyword identification.

Defining Keywords

var schema = Schema.Create()
    .DefineKeywords(CommonKeywords.CTypes)        // int, float, void...
    .DefineKeywords(CommonKeywords.ControlFlow)   // if, else, while, return...
    .DefineKeywordCategory("MyKeywords", "foo", "bar", "baz")
    .Build();

var tree = SyntaxTree.Parse("int main() { return 0; }", schema);

Querying Keywords

// All keywords
var keywords = tree.Select(Query.AnyKeyword);

// Specific keyword
var ifKeywords = tree.Select(Query.Keyword("if"));

// By category
var types = tree.Select(Query.KeywordCategory("CTypes"));

Case-Insensitive Keywords

Perfect for SQL and other case-insensitive languages:

var schema = Schema.Create()
    .DefineKeywordCategory("Sql", caseSensitive: false, "SELECT", "FROM", "WHERE")
    .Build();

// Matches SELECT, select, Select, etc.
var tree = SyntaxTree.Parse("select * from users", schema);

CommonKeywords Presets

Preset	Keywords
CTypes	int, float, double, void, char, long, short
ControlFlow	if, else, while, for, return, break, continue
CppModifiers	public, private, static, const, virtual, override
CppClassifiers	class, struct, enum, union, namespace, template
CppExceptions	try, catch, throw, noexcept
CppMemory	new, delete, nullptr, this, sizeof, auto

Checking Keyword Info

if (leaf.Kind.IsKeyword())
{
    var info = schema.GetKeywordInfo(leaf.Kind);
    Console.WriteLine($"{info.Keyword} is in category {info.Category}");
}

⚡ Performance

• O(1) dictionary-based keyword lookup during lexing
• No delegate allocation or invocation overhead
• Separate dictionaries for case-sensitive and case-insensitive keywords

📚 Documentation

• Schema — Keyword Definitions
• Query API — Keyword Queries

⚠️ Breaking Changes

Red Node Naming Refactored to Roslyn Conventions

The public "red" node types have been renamed to match Roslyn's naming conventions:

Before	After
RedNode	SyntaxNode
RedLeaf	SyntaxToken
RedBlock	SyntaxBlock
RedList	SyntaxList

Migration: Find-and-replace the type names. The APIs are otherwise unchanged.

Red Nodes No Longer Cached

Red nodes are now fully ephemeral — each GetChild() call returns a fresh instance. However, SyntaxNode now overrides Equals/GetHashCode to compare by underlying green node identity and position, so equality comparisons work correctly:

var child1 = parent.GetChild(0);
var child2 = parent.GetChild(0);

// Different instances, but equal (same green node + same position)
child1.Equals(child2)  // true
child1 == child2       // true

SyntaxNode.Tree Property Added

All red nodes now have a Tree property providing access to their containing SyntaxTree. This follows Roslyn's pattern and enables accessing Schema via node.Tree.Schema.

✅ Compatibility

• NodeKind remains a ushort enum — keywords use range 500-1999

v0.8

TinyTokenizer v0.8.0 Release Notes

What's New

🎯 Zero-Allocation Parsing

All internal parsing buffers now use pooled memory via ArrayPoolBufferWriter<char>, dramatically reducing GC pressure during tokenization.

⚡ O(k) Operator Matching

New trie-based operator matching replaces linear scanning. Operator lookup is now O(k) where k is the operator length, regardless of how many operators are configured.

📝 Token Formatting

Token now implements IFormattable with format specifiers:

var token = tokens.First();
$"{token:G}"  // Content (default)
$"{token:T}"  // Type name
$"{token:P}"  // Position
$"{token:R}"  // Range (start..end)
$"{token:D}"  // Debug (Type[start..end])

🔍 Public Trivia Access

Trivia (whitespace, comments) is now publicly accessible:

foreach (var trivia in leaf.LeadingTrivia)
{
    Console.WriteLine($"{trivia.Kind}: '{trivia.Text}'");
}

📋 Schema Helpers

New convenience members on SyntaxTree:

if (tree.HasSchema)
{
    // Safe to use semantic operations
}

var boundTree = tree.WithSchema(mySchema);

---

📊 Performance Benchmarks

All benchmarks run on Windows 11, Intel Core i9-12900K, .NET 8.0.

Sibling Navigation (🏆 Biggest Win)

Operation	Before	After	Improvement
SiblingIndex access	5,600 ns	190 ns	29x faster
NextSibling()	5,620 ns	315 ns	18x faster
PreviousSibling()	5,872 ns	553 ns	10x faster

Token Parsing

Input Size	Before	After	Speed	Memory
Small (1KB)	3.35 μs / 10.78 KB	2.58 μs / 8.97 KB	23% faster	17% less
Medium (100KB)	51.6 μs / 151 KB	42.1 μs / 116 KB	18% faster	23% less
Large (1MB)	14.7 ms / 16.4 MB	10.2 ms / 12.6 MB	30% faster	23% less

Parser-Only (Level 2)

Input Size	Before	After	Speed	Memory
Small	2.88 μs / 10.73 KB	2.33 μs / 8.93 KB	19% faster	17% less
Medium	48.7 μs / 151 KB	33.4 μs / 116 KB	31% faster	23% less
Large	15.7 ms / 16.4 MB	9.7 ms / 12.6 MB	38% faster	23% less

Lexer (Level 1)

Input Size	Before	After	Speed	Memory
Small	263 ns	231 ns	12% faster	—
Medium	3,561 ns	3,101 ns	13% faster	—
LexToArray (Large)	3.4 ms / 7.2 MB	2.9 ms / 5.4 MB	14% faster	25% less

Operator Matching

Operator Count	Before	After	Speed	Memory
10 operators	271 μs / 724 KB	206 μs / 615 KB	24% faster	15% less
50 operators	296 μs / 733 KB	223 μs / 620 KB	25% faster	15% less
100 operators	316 μs / 736 KB	229 μs / 621 KB	28% faster	16% less

Content-Specific Parsing

Content Type	Before	After	Speed	Memory
Blocks	2,984 μs / 3.95 MB	2,845 μs / 3.22 MB	5% faster	19% less
Comments	2,640 μs / 2.25 MB	2,349 μs / 1.74 MB	11% faster	23% less
Strings	5,200 μs / 3.86 MB	4,914 μs / 3.07 MB	5% faster	20% less

SyntaxTree Operations

Operation	Before	After	Speed	Memory
Parse Medium (~5KB)	1.13 ms / 2.46 MB	889 μs / 2.19 MB	21% faster	11% less
Parse Deeply Nested (50 levels)	92.3 μs	81.4 μs	12% faster	—
Query OfType<RedLeaf>	94.4 μs	81.4 μs	14% faster	—
Find Brace Blocks	80.7 μs	70.3 μs	13% faster	—
TreeWalker Traverse (Large)	623 μs	583 μs	6% faster	—

---

Summary

Metric	Improvement
Sibling navigation	Up to 29x faster
Parsing throughput	18-38% faster
Memory allocations	15-25% reduction
Operator matching	24-28% faster (scales with operator count)

---

Breaking Changes

Position Type Changed

SimpleToken.Position and Token.Position changed from long to int.

Migration: No action needed for most code. Files larger than 2GB are not supported.

Removed API

SyntaxTree.ToFullString() has been removed.

Migration: Use SyntaxTree.ToString() instead.

New Dependencies

• CommunityToolkit.HighPerformance 8.2.2

Test Coverage Additions

• Error Recovery: Unclosed blocks, mismatched delimiters, string termination
• Unicode: Multi-script identifiers, emoji, RTL text, invisible characters
• Concurrency: Thread-safe tree access validation
• Numeric Parsing: Edge cases for integers and floating-point numbers

---

Full Changelog: 47 files changed, +3,725 / -615 lines

v0.7.0

Release: v1.x.x - Standardized Serialization & Trivia Preservation

🎯 Highlights

This release introduces a unified text serialization API and fixes a critical bug affecting whitespace preservation in nested code structures.

---

✨ New Features

Unified ITextSerializable Interface

All parsed structures now implement a consistent interface for serializing back to text:

// High-performance buffer-based serialization
token.WriteTo(bufferWriter);

// Get text as string
string text = token.ToText();

// Write to StringBuilder or TextWriter
token.WriteTo(stringBuilder);
token.WriteTo(streamWriter);

// Zero-allocation span-based writing
if (token.TryWriteTo(stackBuffer, out int written))
{
    // Use stackBuffer[..written]
}

// Pre-allocate with known length
int length = token.TextLength;

Implemented by: Token, SimpleToken, GreenNode, RedNode, SyntaxTree

---

🐛 Bug Fixes

Fixed: Whitespace Lost in Nested Blocks

Issue: When parsing code with nested blocks (e.g., nested if statements, loops), indentation before closing braces was being lost during serialization.

Before:

if (a) {
    if (b) {
        x = 1;
    }
} else {      // ❌ Indentation lost: "} else {" instead of "    } else {"
    y = 0;
}

After: All whitespace and trivia is now correctly preserved through parse → edit → serialize cycles.

---

⚠️ Breaking Changes

ToString() Behavior Changed

ToString() now returns a debug representation instead of serialized text:

// Old behavior (removed)
token.ToString()  // returned "actual token content"

// New behavior
token.ToString()  // returns "IdentToken@42" (debug info)
token.ToText()    // returns "actual token content" ✅

Migration: Replace all .ToString() calls with .ToText() when you need the serialized content.

Deprecated Methods

Deprecated	Replacement
SyntaxTree.Serialize()	SyntaxTree.ToText()
SyntaxTree.ToFullString()	SyntaxTree.ToText()

GreenBlock Constructor (Internal API)

If you're using the internal GreenBlock constructor directly, use the new factory method:

// Old (no longer works)
new GreenBlock('{', children, leadingTrivia, trailingTrivia)

// New
GreenBlock.Create('{', children, openerLeadingTrivia, openerTrailingTrivia, 
                  closerLeadingTrivia, closerTrailingTrivia)

---

📊 Testing

• 70+ new tests for block serialization roundtrips
• Coverage for deeply nested structures (up to 100 levels)
• Real-world GLSL shader parsing scenarios
• Mixed line endings and whitespace preservation
• Undo/redo correctly restores exact original text

---

🔧 Internal Improvements

• GreenBlock redesigned with explicit opener/closer GreenLeaf nodes for proper trivia attachment
• GreenNodeCache.CreateDelimiter() for efficient delimiter node creation
• All serialization now routes through IBufferWriter<char> for consistent, high-performance output

v0.6.8

v0.6.8 - Improved Debugging Experience

What's New

Enhanced debugger display for all AST nodes

Debugging TinyTokenizer syntax trees is now significantly faster and more informative. Instead of serializing entire sub-trees when you hover over a node in the debugger, you'll see concise summaries like:

RedLeaf:      Ident[0..5] "myVar"
RedBlock:     BraceBlock[0..50] '{' (3 children) "{ foo..."
FunctionCall: FunctionCall[0..15] "foo" (2 args)

Before & After

Before: Expanding a RedBlock in the watch window would trigger full serialization, showing something like { int x = 1; int y = 2; int z = 3; ... } (potentially thousands of characters, slow to compute).

After: Instant display of BraceBlock[0..127] '{' (12 children) "{ int x = 1; int..." — kind, position range, child count, and a 20-char preview.

Details

• All Green nodes (GreenNode, GreenLeaf, GreenBlock, GreenList, GreenSyntaxNode, GreenTrivia)
• All Red nodes (RedNode, RedLeaf, RedBlock, RedList, SyntaxNode and subclasses)
• ToString() behavior unchanged — serialization still works as expected