v0.9

Release Notes: TinyTokenizer v0.9

🐛 Bug Fixes

Fixed Double-Binding Bug in SyntaxBinder

Fixed a bug where GreenSyntaxNode instances were incorrectly double-wrapped during rebinding after tree mutations. This caused "Expected child at slot 0 to be X, but got SyntaxNode" errors when re-querying syntax nodes after multiple edit/commit cycles.

Affected scenario: Parsing with a schema, making an edit, committing, then re-querying for previously found syntax nodes would fail because the binding system would re-wrap already-bound nodes.

Fix: SyntaxBinder.BindNode() now detects already-bound GreenSyntaxNode instances and skips pattern matching while still recursively binding their children.

🆕 Keyword Categorization System

TinyTokenizer now supports schema-defined keywords that are recognized during lexing and assigned unique NodeKind values. This enables language-aware parsing with proper keyword identification.

Defining Keywords

var schema = Schema.Create()
    .DefineKeywords(CommonKeywords.CTypes)        // int, float, void...
    .DefineKeywords(CommonKeywords.ControlFlow)   // if, else, while, return...
    .DefineKeywordCategory("MyKeywords", "foo", "bar", "baz")
    .Build();

var tree = SyntaxTree.Parse("int main() { return 0; }", schema);

Querying Keywords

// All keywords
var keywords = tree.Select(Query.AnyKeyword);

// Specific keyword
var ifKeywords = tree.Select(Query.Keyword("if"));

// By category
var types = tree.Select(Query.KeywordCategory("CTypes"));

Case-Insensitive Keywords

Perfect for SQL and other case-insensitive languages:

var schema = Schema.Create()
    .DefineKeywordCategory("Sql", caseSensitive: false, "SELECT", "FROM", "WHERE")
    .Build();

// Matches SELECT, select, Select, etc.
var tree = SyntaxTree.Parse("select * from users", schema);

CommonKeywords Presets

Preset	Keywords
CTypes	int, float, double, void, char, long, short
ControlFlow	if, else, while, for, return, break, continue
CppModifiers	public, private, static, const, virtual, override
CppClassifiers	class, struct, enum, union, namespace, template
CppExceptions	try, catch, throw, noexcept
CppMemory	new, delete, nullptr, this, sizeof, auto

Checking Keyword Info

if (leaf.Kind.IsKeyword())
{
    var info = schema.GetKeywordInfo(leaf.Kind);
    Console.WriteLine($"{info.Keyword} is in category {info.Category}");
}

⚡ Performance

• O(1) dictionary-based keyword lookup during lexing
• No delegate allocation or invocation overhead
• Separate dictionaries for case-sensitive and case-insensitive keywords

📚 Documentation

• Schema — Keyword Definitions
• Query API — Keyword Queries

⚠️ Breaking Changes

Red Node Naming Refactored to Roslyn Conventions

The public "red" node types have been renamed to match Roslyn's naming conventions:

Before	After
RedNode	SyntaxNode
RedLeaf	SyntaxToken
RedBlock	SyntaxBlock
RedList	SyntaxList

Migration: Find-and-replace the type names. The APIs are otherwise unchanged.

Red Nodes No Longer Cached

Red nodes are now fully ephemeral — each GetChild() call returns a fresh instance. However, SyntaxNode now overrides Equals/GetHashCode to compare by underlying green node identity and position, so equality comparisons work correctly:

var child1 = parent.GetChild(0);
var child2 = parent.GetChild(0);

// Different instances, but equal (same green node + same position)
child1.Equals(child2)  // true
child1 == child2       // true

SyntaxNode.Tree Property Added

All red nodes now have a Tree property providing access to their containing SyntaxTree. This follows Roslyn's pattern and enables accessing Schema via node.Tree.Schema.

✅ Compatibility

• NodeKind remains a ushort enum — keywords use range 500-1999