Tokens

Tokens in CDTk

Tokens are the foundation of every compiler. They represent patterns in the input text, such as numbers, identifiers, or operators. In CDTk, TokenSet is defined as a class where each token is a field, serving as the first step in the compilation pipeline.

---

What Are Tokens?

A token consists of:

• A field reference: The variable itself serves as the unique identity of the token.
• A regex pattern: Determines which text the token matches.

For example:

• Identifier matches myVariable.
• Number matches 42 or 3.14.
• Plus matches +.

CDTk processes the input text by converting it into a sequence of tokens. For instance:

class Tokens : TokenSet
{
    public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";  // Variable names
    public Token Number = @"\d+";                        // Integers
    public Token Plus = @"\+";                           // +
    public Token Equals = @"=";                          // Assignment operator
}

// Input: x = 42
// Output Tokens: Identifier("x"), Equals("="), Number("42")

Key v8 improvements:

• Field-Based Identity: Token identity is determined by the field reference, not a name or type.
• Regex-Only Matching: Tokens match strictly based on their regex patterns.

---

Defining Tokens with TokenSet

In CDTk, tokens are grouped into a TokenSet class. Each token is defined as a field within the class:

class Tokens : TokenSet
{
    public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";  // Matches variable names
    public Token Number = @"\d+";                        // Matches integers
    public Token Plus = @"\+";                           // Matches +
    public Token Equals = @"=";                          // Matches =
    public Token Whitespace = new Token(@"\s+").Ignore(); // Skips whitespace
}

Key Features

• Field Reference: Token fields are automatically discovered and linked.
• Regex Pattern: Patterns define the text matched by each token.
• Order: Token matching resolves in the order fields are defined.

---

Common Token Patterns

Here are some examples of token patterns in different contexts:

Identifiers and Keywords

public Token Keyword = @"if|while|return";  // Matches keywords
public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";  // Matches variable names

Numbers

public Token Integer = @"\d+";                  // Whole numbers
public Token Float = @"\d+\.\d+([eE][+-]?\d+)?"; // Floating-point numbers
public Token Hexadecimal = @"0[xX][0-9A-Fa-f]+"; // Hex numbers

Strings

public Token String = @"""([^""\\]|\\.)*""";  // Double-quoted strings
public Token Char = @"'([^'\\]|\\.)'";         // Single-quoted characters

Operators and Delimiters

public Token Equals = @"==";     // Equality operator
public Token Assign = @"=";      // Assignment operator
public Token Plus = @"\+";       // Plus operator
public Token LParen = @"\(";     // Left parenthesis
public Token Semicolon = @";";   // Semicolon

Comments and Whitespace

public Token Whitespace = new Token(@"\s+").Ignore();     // Ignored spaces
public Token LineComment = new Token(@"//.*").Ignore();   // Ignored // comments
public Token BlockComment = new Token(@"/\*.*?\*/").Ignore(); // Skips /*...*/ comments

---

Ignoring Tokens

Sometimes, tokens like whitespace or comments must be excluded from the output token stream. Use .Ignore() to skip such tokens:

public Token Whitespace = new Token(@"\s+").Ignore();
public Token LineComment = new Token(@"//.*").Ignore();
public Token BlockComment = new Token(@"/\*.*?\*/").Ignore();

Ignored tokens simplify parsing by removing unnecessary elements from the token stream.

---

Token Ordering

Order of tokens in a TokenSet matters, especially when patterns overlap. Always define specific tokens before generic ones.

Example: Overlapping Patterns

// Incorrect
public Token Identifier = @"\w+";   // Matches everything, including "if"
public Token Keyword = @"if";       // Will never match because "Identifier" captures it

// Correct
public Token Keyword = @"if";       
public Token Identifier = @"\w+";   // Matches non-keywords

---

Validation with TokenSet

CDTk provides a .Validate() method for pre-compilation grammar checks. Validate your tokens along with other grammar definitions to catch issues early:

var compiler = new Compiler()
    .WithTokens(new Tokens())
    .Build();

var diagnostics = compiler.Validate();
if (diagnostics.HasErrors)
{
    foreach (var diagnostic in diagnostics.Items)
    {
        Console.WriteLine($"{diagnostic.Level}: {diagnostic.Message}");
    }
}

Validation ensures:

• Regex patterns are valid.
• No token name collisions or overlaps occur.

---

Best Practices

1. Use Descriptive Names: Token fields should clearly describe their purpose.

   public Token LeftParenthesis = @"\(";  // Clear
   public Token LParen = @"\(";           // Ambiguous

2. Test Regular Expressions: Use tools like Regex101 or Regexr to validate patterns.

3. Avoid Overlap Conflicts: Define specific tokens before generic ones.

4. Use .Ignore() Intelligently: Only ignore tokens that simplify parsing.

5. Call .Validate() Regularly: Leverage validation to identify issues early in the development process.

---

Next Steps

Tokens are only the first step. Continue building your compiler with:

• Rules: Define grammars for tokenized input.
• Mapping: Transform parsed data into output.
• Models: Perform advanced semantic transformations.
• Validation: Ensure your grammar definitions are error-free.