Token Types

Token Types

TinyTokenizer has two levels of token types corresponding to its Architecture.

Level 1 — SimpleToken

SimpleToken is a struct produced by the Lexer. It represents atomic character classifications without semantic meaning.

public readonly record struct SimpleToken(
    SimpleTokenType Type,
    ReadOnlyMemory<char> Content,
    int Position);

SimpleTokenType Values

Type	Description	Example
Ident	Identifier characters	foo, myVar123
Digits	Consecutive digits	123, 456
Whitespace	Spaces, tabs (not newlines)	, \t
Newline	Line endings	\n, \r\n
Symbol	Configured symbol characters	From TokenizerOptions.Symbols
OpenBrace	Opening curly brace	{
CloseBrace	Closing curly brace	}
OpenBracket	Opening square bracket	[
CloseBracket	Closing square bracket	]
OpenParen	Opening parenthesis	(
CloseParen	Closing parenthesis	)
SingleQuote	Single quote	'
DoubleQuote	Double quote	"
Backslash	Escape character	\
Slash	Forward slash	/
Asterisk	Asterisk	*
Dot	Period	.
Hash	Hash symbol	#
At	At symbol	@
Equals	Equals sign	=
Plus	Plus sign	+
Minus	Minus/hyphen	-
LessThan	Less than	<
GreaterThan	Greater than	>
Pipe	Vertical bar	|
Ampersand	Ampersand	&
Percent	Percent	%
Caret	Caret	^
Tilde	Tilde	~
Question	Question mark	?
Exclamation	Exclamation	!
Colon	Colon	:
Comma	Comma	,
Semicolon	Semicolon	;

Level 2 — Token

Token is an abstract record produced by the TokenParser. It represents semantic tokens with meaning.

public abstract record Token
{
    public required ReadOnlyMemory<char> Content { get; init; }
    public required int Position { get; init; }
    public abstract TokenType Type { get; }

    public ReadOnlySpan<char> ContentSpan => Content.Span;
}

Concrete Token Types

IdentToken

Identifiers — variable names, keywords, etc.

public sealed record IdentToken : Token
{
    public override TokenType Type => TokenType.Ident;
}

// Example: "myVariable" → IdentToken { Content = "myVariable" }

WhitespaceToken

Whitespace including spaces, tabs, and newlines.

public sealed record WhitespaceToken : Token
{
    public override TokenType Type => TokenType.Whitespace;
}

// Example: "  \n" → WhitespaceToken { Content = "  \n" }

SymbolToken

Single-character symbols from the configured symbol set.

public sealed record SymbolToken : Token
{
    public override TokenType Type => TokenType.Symbol;
}

// Example: "," → SymbolToken { Content = "," }

OperatorToken

Multi-character operators matched from the configured operator set.

public sealed record OperatorToken : Token
{
    public override TokenType Type => TokenType.Operator;
}

// Example: "===" → OperatorToken { Content = "===" }

NumericToken

Numeric literals with type classification.

public sealed record NumericToken : Token
{
    public override TokenType Type => TokenType.Numeric;
    public required NumericType NumericType { get; init; }
}

public enum NumericType
{
    Integer,       // 123
    FloatingPoint  // 3.14, .5
}

// Example: "3.14" → NumericToken { Content = "3.14", NumericType = FloatingPoint }

StringToken

Quoted string literals with escape handling.

public sealed record StringToken : Token
{
    public override TokenType Type => TokenType.String;
    public required char Quote { get; init; }
}

// Example: "\"hello\"" → StringToken { Content = "\"hello\"", Quote = '"' }

TaggedIdentToken

Identifiers with a prefix tag character.

public sealed record TaggedIdentToken : Token
{
    public override TokenType Type => TokenType.TaggedIdent;
    public required char Tag { get; init; }
    public required ReadOnlyMemory<char> Name { get; init; }
}

// Example: "#define" → TaggedIdentToken { Content = "#define", Tag = '#', Name = "define" }
// Example: "@attribute" → TaggedIdentToken { Content = "@attribute", Tag = '@', Name = "attribute" }

CommentToken

Single-line or multi-line comments.

public sealed record CommentToken : Token
{
    public override TokenType Type => TokenType.Comment;
    public required bool IsMultiLine { get; init; }
}

// Example: "// comment" → CommentToken { Content = "// comment", IsMultiLine = false }
// Example: "/* block */" → CommentToken { Content = "/* block */", IsMultiLine = true }

BlockToken

Delimited blocks with nested children.

public sealed record BlockToken : Token
{
    public override TokenType Type => TokenType.Block;
    public required BlockTokenType BlockType { get; init; }
    public required ImmutableArray<Token> Children { get; init; }
    public required ReadOnlyMemory<char> InnerContent { get; init; }
}

public enum BlockTokenType
{
    Brace,    // { }
    Bracket,  // [ ]
    Paren     // ( )
}

// Example: "(a, b)" → BlockToken {
//     BlockType = Paren,
//     Children = [IdentToken("a"), SymbolToken(","), WhitespaceToken(" "), IdentToken("b")]
// }

ErrorToken

Produced for malformed input — enables error recovery.

public sealed record ErrorToken : Token
{
    public override TokenType Type => TokenType.Error;
    public required string ErrorMessage { get; init; }
}

// Example: "}" (unmatched) → ErrorToken { Content = "}", ErrorMessage = "Unexpected closing delimiter '}'" }

Token Formatting (IFormattable)

All tokens implement IFormattable with format specifiers:

Specifier	Output	Example
G (default)	Content	foo
T	Type name	IdentToken
P	Position	42
R	Range	42..45
D	Debug format	Ident[42..45]

var token = tokens.First();
Console.WriteLine($"{token:G}");  // "foo"
Console.WriteLine($"{token:T}");  // "IdentToken"
Console.WriteLine($"{token:P}");  // "0"
Console.WriteLine($"{token:R}");  // "0..3"
Console.WriteLine($"{token:D}");  // "Ident[0..3]"

See Also

• Architecture — Two-level tokenization design
• Configuration — Configure symbols, operators, comments
• Error Handling — Working with ErrorToken