Trivia

Trivia

Trivia represents whitespace and comments attached to tokens. TinyAst preserves trivia to enable round-tripping (parsing and re-serializing without losing formatting).

What is Trivia?

Trivia is non-semantic content surrounding tokens:

// Given: "  x = 1; // comment\n"
//        ^^       ^^^^^^^^^^^^
//        |        trailing trivia (comment + newline)
//        leading trivia (spaces)

Each SyntaxToken node can have:

• Leading trivia — Whitespace/comments before the token
• Trailing trivia — Whitespace/comments after the token (may include a line ending)

Accessing Trivia

var tree = SyntaxTree.Parse("  x = 1; // comment\n");

foreach (var leaf in tree.Leaves)
{
    Console.WriteLine($"Token: '{leaf.Text}'");

    // Leading trivia
    foreach (var trivia in leaf.LeadingTrivia)
    {
        Console.WriteLine($"  Leading: {trivia.Kind} = '{trivia.Text}'");
    }

    // Trailing trivia
    foreach (var trivia in leaf.TrailingTrivia)
    {
        Console.WriteLine($"  Trailing: {trivia.Kind} = '{trivia.Text}'");
    }
}

TriviaKind Values

Kind	Description	Example
Whitespace	Spaces and tabs	" ", "\t"
Newline	Line endings	"\n", "\r\n"
SingleLineComment	Single-line comments	"// comment"
MultiLineComment	Multi-line comments	"/* comment */"

Trivia Struct

public readonly struct Trivia
{
    public TriviaKind Kind { get; }
    public string Text { get; }
    public int Position { get; }
    public int Width { get; }
}

Trivia Assignment Rules

Trivia is assigned to the "closest" token:

1. Leading trivia — Whitespace/comments at line start, before a token
2. Trailing trivia — Whitespace/comments after a token. If the line ends, the newline belongs to the trailing trivia of the last token on that line.

// Input:
//   foo   // comment
//   bar

// Token "foo":
//   Leading: "   " (leading spaces)
//   Trailing: "   // comment\n" (spaces + comment + newline)

// Token "bar":
//   Leading: "   " (indent on new line)
//   Trailing: (none)

## Newlines and `Query.Newline`

Newlines are represented as trivia (`TriviaKind.Newline`). `Query.Newline` matches a node when either:

- The node owns leading newline trivia, OR
- The previous sibling owns trailing newline trivia.

This is token-centric boundary ownership: container nodes don't "own" newline boundaries; the boundary lives on the nearest token's trivia.

Trivia-Preserving Edits

The [[SyntaxEditor|Edit]] method preserves trivia automatically:

var tree = SyntaxTree.Parse("  hello   world  ");

tree.CreateEditor()
    .Edit(Query.AnyIdent, content => content.ToUpper())
    .Commit();

// Result: "  HELLO   WORLD  "
// Whitespace preserved!

Edit vs Replace

Method	Trivia Handling
Edit(query, content => ...)	Automatically preserved
Replace(query, text)	Replaces entire node including trivia
Replace(query, node => ...)	You control trivia in transformer

// Edit preserves trivia
tree.CreateEditor()
    .Edit(Query.Ident("foo"), s => s.ToUpper())
    .Commit();
// "  foo  " → "  FOO  "

// Replace removes trivia
tree.CreateEditor()
    .Replace(Query.Ident("foo"), "BAR")
    .Commit();
// "  foo  " → "  BAR  " (trivia still there because it's on surrounding nodes)

Checking for Trivia

var leaf = tree.Leaves.First();

// Check if has any trivia
bool hasLeading = leaf.LeadingTrivia.Any();
bool hasTrailing = leaf.TrailingTrivia.Any();

// Check for specific kind
bool hasComment = leaf.TrailingTrivia.Any(t => t.Kind == TriviaKind.SingleLineComment);

Full String vs Text

var tree = SyntaxTree.Parse("  foo  ");
var foo = tree.Leaves.First(l => l.Kind == NodeKind.Ident);

// Text = just the token content
Console.WriteLine(foo.Text);           // "foo"

// ToFullString() = includes trivia
Console.WriteLine(foo.ToFullString()); // "  foo  "

Practical Examples

Extract Comments

var tree = SyntaxTree.Parse("x = 1; // important\ny = 2; // also important");

var comments = tree.Leaves
    .SelectMany(l => l.TrailingTrivia)
    .Where(t => t.Kind == TriviaKind.SingleLineComment)
    .Select(t => t.Text);

foreach (var comment in comments)
{
    Console.WriteLine(comment);
}
// Output:
// // important
// // also important

Check Indentation

var tree = SyntaxTree.Parse("    indented");
var firstToken = tree.Leaves.First();

var indent = firstToken.LeadingTrivia
    .Where(t => t.Kind == TriviaKind.Whitespace)
    .Sum(t => t.Width);

Console.WriteLine($"Indentation: {indent} characters");

Normalize Whitespace

// This removes trivia by using Replace instead of Edit
var tree = SyntaxTree.Parse("  foo   bar  ");

var normalized = string.Join(" ",
    tree.Leaves
        .Where(l => l.Kind != NodeKind.Whitespace)
        .Select(l => l.Text));

Console.WriteLine(normalized); // "foo bar"

See Also

• SyntaxEditor — Edit vs Replace for trivia handling
• TinyAst Guide — Working with leaf nodes
• Token Types — WhitespaceToken and CommentToken