Debugging

Debugging in CDTk

Debugging is an essential part of building compilers and transpilers in CDTk. Whether you're investigating unexpected behavior in tokenization, rule processing, or code generation, CDTk provides robust tools to help you identify and resolve issues.

This guide covers debugging strategies for each part of the CDTk pipeline—tokenization, parsing, mapping, and compilation.

---

Table of Contents

1. Overview of Debugging in CDTk
2. Using Diagnostics
3. Debugging Tokenization
4. Debugging Grammar and Rules
5. Debugging Code Generation
6. Accessing the AST
7. Common Issues and Resolutions
8. Best Practices for Debugging

---

Overview of Debugging in CDTk

CDTk makes it easy to debug each stage of the compilation pipeline. The pipeline is broken into three major stages:

1. Tokenization – Converting input text into tokens using the TokenSet.
2. Parsing – Organizing tokens into an Abstract Syntax Tree (AST) based on RuleSet.
3. Code Generation – Transforming the AST into output text using MapSet.

When debugging, focus on the stage where the issue occurs:

• If the input text isn't tokenized correctly, debug tokenization.
• If parsing fails or results in an incorrect AST, debug grammar and rules.
• If the generated output is incorrect or missing, debug mapping.

---

Using Diagnostics

The Diagnostics system in CDTk tracks issues across all pipeline stages and organizes them into three levels:

• Info: Non-critical messages for additional context.
• Warning: Recoverable issues that may lead to unexpected behavior.
• Error: Critical issues that halt the compilation process.

Adding Diagnostics

You can add diagnostics during lexical analysis, rule validation, or semantic processing:

var diagnostics = new Diagnostics();

diagnostics.Add(
    Stage.LexicalAnalysis,
    DiagnosticLevel.Error,
    "Unexpected token identified.",
    new SourceSpan(10, 1, 1, 10)
);

Viewing Diagnostics

Post-compilation, review diagnostic messages for insights:

var result = compiler.Compile(inputCode);

if (!result.Success)
{
    foreach (var diag in result.Diagnostics.Items)
    {
        Console.WriteLine($"{diag.Stage} {diag.Level}: {diag.Message}");
        if (diag.Span != SourceSpan.Unknown)
        {
            Console.WriteLine($"  Location: Line {diag.Span.Line}, Column {diag.Span.Column}");
        }
    }
}

---

Debugging Tokenization

Tokenization breaks input text into tokens. If your input doesn't produce the expected tokens:

1. Verify the regex patterns in your TokenSet.
2. Check the order of tokens, as CDTk uses the first matching token.

Steps to Debug Tokenization

1. Test Token Patterns

Manually validate token patterns using the Regex.Match method:

var pattern = @"[A-Za-z_][A-Za-z0-9_]*";
var match = Regex.Match("myVariable", pattern);

if (match.Success)
{
    Console.WriteLine($"Matched: {match.Value}");
}

2. Print Tokens Generated

Log the generated tokens after tokenization:

var tokens = lexer.Tokenize(inputCode, diagnostics);

foreach (var token in tokens)
{
    Console.WriteLine($"Token: {token.Type}, Lexeme: {token.Lexeme}");
}

3. Reorder Token Definitions

Place more specific token patterns before general ones:

var tokens = new TokenSet
{
    new Token("Keyword", @"if|else|while"),   // Specific tokens come first
    new Token("Identifier", @"[A-Za-z_][A-Za-z0-9_]*")  // General tokens last
};

---

Debugging Grammar and Rules

Approach

If parsing fails or produces an incorrect AST:

1. Test individual rules:
   • Simplify complex grammar to isolate the issue.
2. Check token references:
   • Ensure you use @TokenName for token references.
3. Validate alternation and precedence:
   • Use grouping and ordering to eliminate ambiguity.

---

Debugging Rule Validation

The Validate() method allows you to add custom checks for semantic correctness:

new Rule<Function>("'function' name:@Identifier args:Arguments body:Block")
    .Returns<FuncNode>("name", "args", "body")
    .Validate((ctx, astNode, _) => 
    {
        if (!ctx.CurrentScope.Declare(astNode.GetString("name")))
        {
            ctx.Report(
                DiagnosticLevel.Error,
                $"Duplicate function declaration: {astNode.GetString("name")}",
                astNode.Span
            );
        }
    });

---

Logging the AST

Inspect the produced AST for incorrect node types or field data:

void PrintAst(AstNode node, int depth = 0)
{
    Console.WriteLine(new string(' ', depth * 2) + node.Type);
    foreach (var field in node.Fields)
    {
        if (field.Value is AstNode childNode)
        {
            PrintAst(childNode, depth + 1);
        }
        else
        {
            Console.WriteLine(new string(' ', (depth + 1) * 2) + $"{field.Key}: {field.Value}");
        }
    }
}

Call PrintAst(result.Ast); to log the AST structure.

---

Debugging Code Generation

Code generation issues arise from mappings in the MapSet. If Compile succeeds but the output is incorrect:

1. Ensure the MapSet handles all node types in your AST.
2. Check for placeholder mismatches between the mapping template and the .Returns() field names.

---

Debugging Fallback Mappings

Add fallback mappings to track unprocessed nodes:

new Map("UnknownNode", "// Unhandled node: {type}");

---

Debugging Output Templates

Check templates for compatibility with AST fields:

new Map("BinaryOpNode", "({left} {op} {right})");

// Example of debugging fields:
Console.WriteLine(node.GetString("left"));
Console.WriteLine(node.GetString("op"));
Console.WriteLine(node.GetString("right"));

---

Accessing the AST

Debugging often involves inspecting the AST for completeness and correctness. Steps:

1. Verify all expected node types exist.
2. Confirm field mappings from .Returns() in grammar rules.

Example:

if (result.Ast != null)
{
    PrintAst(result.Ast);
}

---

Common Issues and Their Fixes

Issue	Resolution
Token doesn't match input	Verify regex patterns and test manually. Ensure correct token order.
Unexpected token during parsing	Ensure tokens match rule definitions in the RuleSet.
Parsing infinite loop / recursion	Refactor left-recursive rules to iterative forms.
Node missing from map output	Ensure your MapSet includes a mapping for each AST node type.
Errors in scope validation	Use ctx.CurrentScope.Declare() and ctx.PushScope() to debug scope issues.

---

Best Practices for Debugging

1. Use Diagnostics Early: Add .Validate() and .Error() to track issues in real time.
2. Log Outputs Incrementally: Print tokens, rules, and AST structures during pipeline stages.
3. Design Fallbacks: Protect parsing and mapping stages with fallback rules and mappings.
4. Modular Debugging: Test tokenization, parsing, and mapping independently before combining.

---

Next Steps

Now that you know how to debug compilers in CDTk:

• Learn about tokens: Tokens
• Explore grammar rules: Rules
• Dive into mapping: Mapping
• Begin with example compilers: Examples