Learn

Learn CDTk: Getting Started

Welcome to CDTk! This guide introduces the basics of building a compiler in 3 steps:

1. TokenSet: Recognize lexical patterns (tokens).
2. RuleSet: Structure grammar rules.
3. MapSet: Define how parsed input transforms into output.

---

Start Here: The Basics

CDTk simplifies compiler design with its unified and fully-typed pipeline:

• TokenSet: Identify and categorize input patterns (e.g., numbers, keywords, operators).
• RuleSet: Combine tokens into meaningful structures using grammar rules.
• MapSet: Convert parsed structures into your desired output (e.g., code, JSON, config).

using CDTk;

class Tokens : TokenSet
{
    public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";  // Variables
    public Token Number = @"\d+";                        // Numbers
    public Token Equals = @"=";                          // Assignment operator
    public Token Whitespace = new Token(@"\s+").Ignore(); // Ignored whitespace
}

class Rules : RuleSet
{
    public Rule AssignmentNode = new Rule("variable:@Identifier '=' value:@Number")
        .Returns("variable", "value");
}

class Maps : MapSet
{
    public Map AssignmentNode = "int {variable} = {value};";  // Outputs C syntax
}

var compiler = new Compiler()
    .WithTokens(new Tokens())
    .WithRules(new Rules())
    .WithTarget(new Maps())
    .Build();

var result = compiler.Compile("x = 42;");
Console.WriteLine(result.Output);  // Outputs: int x = 42;

---

The Pipeline

When you call compiler.Compile(input), CDTk processes the input systematically:

1. Lexing (TokenSet): Converts input text into tokens (e.g., Identifier("x"), Equals("="), Number("42")).
2. Parsing (RuleSet): Organizes tokens into an AST (Abstract Syntax Tree).
3. Code Generation (MapSet): Transforms the AST into output (e.g., int x = 42;).

Learn More: Check out the Pipeline Overview.

---

Defining Your Compiler

TokenSet

Tokens define basic patterns in your language. For example:

class Tokens : TokenSet
{
    public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";  // Variables
    public Token Number = @"\d+";                        // Numbers
    public Token Equals = @"=";                          // Assignment operator
    public Token Whitespace = new Token(@"\s+").Ignore(); // Ignored whitespace
}

Learn More: See Tokens.

RuleSet

Rules describe your grammar in a declarative syntax:

class Rules : RuleSet
{
    public Rule AssignmentNode = new Rule("variable:@Identifier '=' value:@Number")
        .Returns("variable", "value");
}

Learn More: See Rules.

MapSet

Define templates for code generation or output transformation:

class Maps : MapSet
{
    public Map AssignmentNode = "int {variable} = {value};";  // Outputs C syntax
}

Learn More: See Mapping.

---

Compiling Your Input

Combine the TokenSet, RuleSet, and MapSet into Compiler:

var compiler = new Compiler()
    .WithTokens(new Tokens())
    .WithRules(new Rules())
    .WithTarget(new Maps())
    .Build();

var result = compiler.Compile("x = 42;");
Console.WriteLine(result.Output);  // Outputs: int x = 42;

---

Debugging and Next Steps

Debugging

If you encounter issues:

1. Unexpected Tokens: Adjust your TokenSet definitions.
2. Grammar Errors: Refine your RuleSet for clarity.
3. Missing Outputs: Confirm MapSet covers all expected node types.

Validation

CDTk v8 introduces .Validate() for pre-compilation checks:

var diagnostics = compiler.Validate();
if (diagnostics.HasErrors)
{
    foreach (var diagnostic in diagnostics.Items)
    {
        Console.WriteLine($"{diagnostic.Level}: {diagnostic.Message}");
    }
}

Learn More: See the Debugging Guide.

---

Explore Further

CDTk offers advanced features and infinite customization:

• Tokens: Dive deeper into token field definitions.
• Rules: Learn about recursive structures and reference equality.
• Mapping: Explore multi-target generation with field-based mappings.
• Models: Understand how to leverage models for advanced semantic transformations in your pipeline.
• Validation: Learn how to validate your grammar definitions and attach additional metadata to rules.
• Examples: Study real-world compiler setups.