v0.8

TinyTokenizer v0.8.0 Release Notes

What's New

🎯 Zero-Allocation Parsing

All internal parsing buffers now use pooled memory via ArrayPoolBufferWriter<char>, dramatically reducing GC pressure during tokenization.

⚡ O(k) Operator Matching

New trie-based operator matching replaces linear scanning. Operator lookup is now O(k) where k is the operator length, regardless of how many operators are configured.

📝 Token Formatting

Token now implements IFormattable with format specifiers:

var token = tokens.First();
$"{token:G}"  // Content (default)
$"{token:T}"  // Type name
$"{token:P}"  // Position
$"{token:R}"  // Range (start..end)
$"{token:D}"  // Debug (Type[start..end])

🔍 Public Trivia Access

Trivia (whitespace, comments) is now publicly accessible:

foreach (var trivia in leaf.LeadingTrivia)
{
    Console.WriteLine($"{trivia.Kind}: '{trivia.Text}'");
}

📋 Schema Helpers

New convenience members on SyntaxTree:

if (tree.HasSchema)
{
    // Safe to use semantic operations
}

var boundTree = tree.WithSchema(mySchema);

---

📊 Performance Benchmarks

All benchmarks run on Windows 11, Intel Core i9-12900K, .NET 8.0.

Sibling Navigation (🏆 Biggest Win)

Operation	Before	After	Improvement
SiblingIndex access	5,600 ns	190 ns	29x faster
NextSibling()	5,620 ns	315 ns	18x faster
PreviousSibling()	5,872 ns	553 ns	10x faster

Token Parsing

Input Size	Before	After	Speed	Memory
Small (1KB)	3.35 μs / 10.78 KB	2.58 μs / 8.97 KB	23% faster	17% less
Medium (100KB)	51.6 μs / 151 KB	42.1 μs / 116 KB	18% faster	23% less
Large (1MB)	14.7 ms / 16.4 MB	10.2 ms / 12.6 MB	30% faster	23% less

Parser-Only (Level 2)

Input Size	Before	After	Speed	Memory
Small	2.88 μs / 10.73 KB	2.33 μs / 8.93 KB	19% faster	17% less
Medium	48.7 μs / 151 KB	33.4 μs / 116 KB	31% faster	23% less
Large	15.7 ms / 16.4 MB	9.7 ms / 12.6 MB	38% faster	23% less

Lexer (Level 1)

Input Size	Before	After	Speed	Memory
Small	263 ns	231 ns	12% faster	—
Medium	3,561 ns	3,101 ns	13% faster	—
LexToArray (Large)	3.4 ms / 7.2 MB	2.9 ms / 5.4 MB	14% faster	25% less

Operator Matching

Operator Count	Before	After	Speed	Memory
10 operators	271 μs / 724 KB	206 μs / 615 KB	24% faster	15% less
50 operators	296 μs / 733 KB	223 μs / 620 KB	25% faster	15% less
100 operators	316 μs / 736 KB	229 μs / 621 KB	28% faster	16% less

Content-Specific Parsing

Content Type	Before	After	Speed	Memory
Blocks	2,984 μs / 3.95 MB	2,845 μs / 3.22 MB	5% faster	19% less
Comments	2,640 μs / 2.25 MB	2,349 μs / 1.74 MB	11% faster	23% less
Strings	5,200 μs / 3.86 MB	4,914 μs / 3.07 MB	5% faster	20% less

SyntaxTree Operations

Operation	Before	After	Speed	Memory
Parse Medium (~5KB)	1.13 ms / 2.46 MB	889 μs / 2.19 MB	21% faster	11% less
Parse Deeply Nested (50 levels)	92.3 μs	81.4 μs	12% faster	—
Query OfType<RedLeaf>	94.4 μs	81.4 μs	14% faster	—
Find Brace Blocks	80.7 μs	70.3 μs	13% faster	—
TreeWalker Traverse (Large)	623 μs	583 μs	6% faster	—

---

Summary

Metric	Improvement
Sibling navigation	Up to 29x faster
Parsing throughput	18-38% faster
Memory allocations	15-25% reduction
Operator matching	24-28% faster (scales with operator count)

---

Breaking Changes

Position Type Changed

SimpleToken.Position and Token.Position changed from long to int.

Migration: No action needed for most code. Files larger than 2GB are not supported.

Removed API

SyntaxTree.ToFullString() has been removed.

Migration: Use SyntaxTree.ToString() instead.

New Dependencies

• CommunityToolkit.HighPerformance 8.2.2

Test Coverage Additions

• Error Recovery: Unclosed blocks, mismatched delimiters, string termination
• Unicode: Multi-script identifiers, emoji, RTL text, invisible characters
• Concurrency: Thread-safe tree access validation
• Numeric Parsing: Edge cases for integers and floating-point numbers

---

Full Changelog: 47 files changed, +3,725 / -615 lines