Error Handling

Error Handling

TinyTokenizer uses error recovery to continue parsing malformed input. Errors are represented as ErrorToken instances rather than throwing exceptions.

ErrorToken

When the tokenizer encounters malformed input, it produces an ErrorToken:

public sealed record ErrorToken : Token
{
    public override TokenType Type => TokenType.Error;
    public required string ErrorMessage { get; init; }
}

Common Error Conditions

Condition	Example	Error Message
Unmatched closing delimiter	} without {	"Unexpected closing delimiter '}'"
Unclosed block	{ without }	"Unclosed block starting with '{'"
Unterminated string	"hello	(Emits opening quote as SymbolToken)
Unterminated multi-line comment	/* comment	"Unterminated multi-line comment"

Detecting Errors in Tokens

var tokens = "hello } world".TokenizeToTokens();

foreach (var token in tokens)
{
    if (token is ErrorToken error)
    {
        Console.WriteLine($"Error at position {error.Position}: {error.ErrorMessage}");
        Console.WriteLine($"  Content: '{error.Content}'");
    }
}

Detecting Errors in Syntax Trees

var tree = SyntaxTree.Parse("}hello{");

// Query for error nodes
var errors = tree.Root.Children
    .Where(n => n.Kind == NodeKind.Error)
    .Cast<SyntaxToken>();

foreach (var error in errors)
{
    Console.WriteLine($"Error at {error.Position}: {error.Text}");
}

Using Query API

var tree = SyntaxTree.Parse("}hello{");
var errors = tree.Select(Query.Any.Where(n => n.Kind == NodeKind.Error));  // [Error("}"), Error("{")]

if (errors.Any())
{
    Console.WriteLine($"Found {errors.Count()} errors");  // "Found 2 errors"
}

Error Recovery Behavior

The tokenizer continues parsing after errors:

// Input with multiple errors
var tokens = "{ valid } } more { valid".TokenizeToTokens().ToList();

// tokens contains:
// - BlockToken("{ valid }")
// - WhitespaceToken(" ")
// - ErrorToken("}") - unexpected closer
// - WhitespaceToken(" ")
// - IdentToken("more")
// - WhitespaceToken(" ")
// - ErrorToken("{") - unclosed block at end

This allows processing partially valid input and reporting all errors at once.

Checking for Parse Success

var tokens = source.TokenizeToTokens().ToList();

bool hasErrors = tokens.Any(t => t is ErrorToken);

if (hasErrors)
{
    var errors = tokens.OfType<ErrorToken>().ToList();
    Console.WriteLine($"Parse failed with {errors.Count} errors:");
    foreach (var error in errors)
    {
        Console.WriteLine($"  Line position {error.Position}: {error.ErrorMessage}");
    }
}
else
{
    Console.WriteLine("Parse succeeded");
}

Error Handling Patterns

Fail Fast

Stop processing on first error:

foreach (var token in source.TokenizeToTokens())
{
    if (token is ErrorToken error)
    {
        throw new ParseException($"Error at {error.Position}: {error.ErrorMessage}");
    }

    // Process valid token
}

Collect All Errors

Gather all errors before reporting:

var tokens = source.TokenizeToTokens().ToList();
var errors = tokens.OfType<ErrorToken>().ToList();

if (errors.Count > 0)
{
    var report = string.Join("\n", errors.Select(e =>
        $"  Position {e.Position}: {e.ErrorMessage}"));

    throw new ParseException($"Found {errors.Count} errors:\n{report}");
}

Best Effort Processing

Process what you can, skip errors:

foreach (var token in source.TokenizeToTokens())
{
    switch (token)
    {
        case ErrorToken:
            // Log and continue
            continue;

        case IdentToken ident:
            ProcessIdentifier(ident);
            break;

        case BlockToken block:
            ProcessBlock(block);
            break;
    }
}

Streaming with Error Detection

var errors = new List<ErrorToken>();

await foreach (var token in stream.TokenizeStreamingAsync())
{
    if (token is ErrorToken error)
    {
        errors.Add(error);
        continue;
    }

    // Process valid token
}

if (errors.Count > 0)
{
    Console.WriteLine($"Completed with {errors.Count} errors");
}

Error Position Information

Use Position to locate errors in source:

var source = "hello } world";
var tokens = source.TokenizeToTokens().ToList();

foreach (var error in tokens.OfType<ErrorToken>())
{
    // Get surrounding context
    var start = Math.Max(0, error.Position - 10);
    var end = Math.Min(source.Length, error.Position + 10);
    var context = source[start..end];

    Console.WriteLine($"Error: {error.ErrorMessage}");
    Console.WriteLine($"Near: ...{context}...");
    Console.WriteLine($"      {new string(' ', error.Position - start)}^");
}

Unterminated Strings

Unterminated strings are handled specially — the opening quote becomes a SymbolToken:

var tokens = "hello \"unterminated".TokenizeToTokens().ToList();

// tokens contains:
// - IdentToken("hello")
// - WhitespaceToken(" ")
// - SymbolToken("\"")  ← Opening quote as symbol
// - IdentToken("unterminated")

This allows processing the rest of the file even when a string isn't closed.

See Also

• Token Types — ErrorToken details
• Async Streaming — Error handling in streams
• TinyAst Guide — Working with error nodes in trees