Schema

Schema

The Schema class provides unified configuration for tokenization settings, keyword definitions, and syntax node definitions.

Why Schema?

Schema combines three concerns:

1. Tokenization settings — Operators, comments, symbols, tag prefixes (same as Configuration)
2. Keyword definitions — Language keywords recognized during lexing (e.g., if, while, int)
3. Syntax definitions — Patterns for matching higher-level constructs (function calls, etc.)

This allows a single configuration object for parsing and pattern matching.

Quick Start

using TinyTokenizer.Ast;

// Create a schema with tokenization + keywords + syntax definitions
var schema = Schema.Create()
    .WithOperators(CommonOperators.JavaScript)
    .WithCommentStyles(CommentStyle.CStyleSingleLine, CommentStyle.CStyleMultiLine)
    .DefineKeywords(CommonKeywords.ControlFlow)  // if, else, while, for, return...
    .DefineKeywords(CommonKeywords.CTypes)       // int, float, double, void...
    .Define(BuiltInDefinitions.FunctionName)
    .Define(BuiltInDefinitions.MethodCall)
    .Build();

// Parse with schema — keywords are recognized during lexing
var tree = SyntaxTree.Parse("int main() { return 0; }", schema);

// Query keywords
var keywords = tree.Select(Query.AnyKeyword);           // all keywords
var types = tree.Select(Query.KeywordCategory("Types")); // just type keywords

Schema.Default

A pre-configured schema with common settings:

var tree = SyntaxTree.Parse(source, Schema.Default);

// Schema.Default includes:
// - CommonOperators.Universal
// - C-style single and multi-line comments
// - Built-in definitions: FunctionName, ArrayAccess, PropertyAccess, MethodCall
// - No keywords (add via .DefineKeywords() for your language)

Building a Schema

Tokenization Settings

Same fluent API as Configuration:

var schema = Schema.Create()
    .WithOperators(CommonOperators.CFamily)
    .WithOperators("=>", "?.", "??")
    .WithCommentStyles(CommentStyle.CStyleSingleLine)
    .WithTagPrefixes('#', '@')
    .WithSymbols(',', ';')
    .Build();

Keyword Definitions

Keywords are identifiers recognized during lexing with a unique NodeKind.

var schema = Schema.Create()
    // Use presets
    .DefineKeywords(CommonKeywords.CTypes)       // int, float, double, void...
    .DefineKeywords(CommonKeywords.ControlFlow)  // if, else, while, for, return...
    
    // Or define custom categories
    .DefineKeywordCategory("SqlKeywords", "SELECT", "FROM", "WHERE")
    .DefineKeywordCategory("SqlKeywords", caseSensitive: false, "SELECT", "FROM")  // Case-insensitive
    .Build();

// Query keywords
var tree = SyntaxTree.Parse("int x = 0; if (x > 0) return x;", schema);
var allKeywords = tree.Select(Query.AnyKeyword);
var types = tree.Select(Query.KeywordCategory("Types"));

// Check keyword info
if (leaf.Kind.IsKeyword())
{
    var info = schema.GetKeywordInfo(leaf.Kind);  // { Category, Keyword, Kind }
}

Adding Syntax Definitions

Register pattern matchers for structural constructs:

var schema = Schema.Create()
    .WithOperators(CommonOperators.JavaScript)
    // Built-in definitions
    .Define(BuiltInDefinitions.FunctionName)
    .Define(BuiltInDefinitions.ArrayAccess)
    .Define(BuiltInDefinitions.PropertyAccess)
    .Define(BuiltInDefinitions.MethodCall)
    // Custom definitions (see Syntax Nodes)
    .Define(myCustomDefinition)
    .Build();

Built-in Definitions

Definition	Pattern	Matches
FunctionName	Ident followed by ParenBlock	foo(x)
ArrayAccess	Ident followed by BracketBlock	arr[0]
PropertyAccess	Ident, ., Ident	obj.prop
MethodCall	Ident, ., Ident, ParenBlock	obj.method(x)

Converting from TokenizerOptions

If you have existing TokenizerOptions, convert to Schema:

var options = TokenizerOptions.Default
    .WithOperators(CommonOperators.CFamily)
    .WithCommentStyles(CommentStyle.CStyleSingleLine);

// Create schema from options
var schema = Schema.FromOptions(options);

// Or add definitions to the converted schema
var schemaWithDefs = Schema.FromOptions(options)
    .Define(BuiltInDefinitions.FunctionName)
    .Build();

Checking Schema Attachment

var tree = SyntaxTree.Parse(source, schema);

if (tree.HasSchema)
{
    // Safe to use Match<T>() and other schema-dependent operations
    var funcs = tree.Match<FunctionCallSyntax>();
}

Attaching Schema to Existing Tree

// Parse without schema
var tree = SyntaxTree.Parse(source);

// Later, attach a schema
var boundTree = tree.WithSchema(schema);

// Now can use schema-dependent operations
var matches = boundTree.Match<FunctionCallSyntax>();

Custom Syntax Definitions

See Syntax Nodes for creating custom SyntaxNode types and definitions:

// 1. Define a node class
public sealed class LambdaSyntax : SyntaxNode { ... }

// 2. Create a definition
var lambdaDef = Syntax.Define<LambdaSyntax>("Lambda")
    .Match(Query.Sequence(Query.ParenBlock, Query.Operator("=>"), Query.BraceBlock))
    .Build();

// 3. Add to schema
var schema = Schema.Create()
    .Define(lambdaDef)
    .Build();

See Also

• Configuration — TokenizerOptions details
• Syntax Nodes — Creating custom syntax patterns
• TinyAst Guide — Using SyntaxTree with Schema