Rules

Rules In CDTk

In CDTk, grammar rules are the core of parsing logic. They define how tokens combine into higher-level structures (like expressions or statements) and ultimately form an Abstract Syntax Tree (AST). Rules are grouped into a RuleSet class, leveraging the new field-based identity system introduced in v8.

---

What Are Grammar Rules?

A grammar rule specifies:

1. Input Structure: Defines the sequence or pattern of tokens and sub-rules that the rule matches.
2. Action: Details the construction of structured objects, such as AST nodes, from the matched tokens and outputs of sub-rules.

For example:

class Rules : RuleSet
{
    public Rule AssignmentNode = new Rule("variable:@Identifier '=' value:@Number")
        .Returns("variable", "value");
}

Components:

• Pattern: Describes the structure of tokens or nested rules (e.g., @Identifier '=' @Number).
• Returns: Maps token or rule outputs to fields in the constructed AST node.

---

Defining Rules

In CDTk, rules are defined as fields within a RuleSet class. The field reference serves as the rule's identity, and patterns are declared declaratively:

class Rules : RuleSet
{
    public Rule Expression = new Rule("left:@Identifier '+' right:@Number")
        .Returns("left", "right");
}

• Field Reference: The Expression field is the unique identity of this rule.
• Declarative Pattern: Describes the grammar structure.
• Returns: Specifies which fields populate the resulting AST node.

---

Rule Grammar

CDTk uses a declarative grammar syntax for rules. Here's what each part means:

Rule Syntax

Symbol	Description
@TokenName	Matches a token defined in the TokenSet.
RuleName	Matches and applies another rule.
'='	Matches a literal string or operator.
`	`
*	Matches zero or more repetitions.
+	Matches one or more repetitions.
?	Matches zero or one occurrence.

Example Patterns

1. Basic Rule:

   public Rule BinaryExpression = new Rule("left:@Identifier '+' right:@Number")
       .Returns("left", "right");

2. Repetition:

   public Rule RepeatedTerms = new Rule("terms:@Term+").Returns("terms");

3. Optional Components:

   public Rule OptionalSemicolon = new Rule("expr:@Expression ';'?").Returns("expr");

4. Alternation (OR):

   public Rule Factor = new Rule("@Number | '(' expr:@Expression ')'").Returns("value");

---

Validation with RuleSet

CDTk v8 introduces a .Validate() system to check the correctness of grammar rules before compilation.

Example: Validating Rules

var rules = new Rules();
var diagnostics = rules.Validate();

if (diagnostics.HasErrors)
{
    foreach (var diagnostic in diagnostics.Items)
    {
        Console.WriteLine($"{diagnostic.Level}: {diagnostic.Message}");
    }
}

What Validation Checks:

• Rule patterns reference valid tokens and other rules.
• No undefined or ambiguous references exist.
• Patterns adhere to expected sequences and alternations.

Validation helps catch errors early, ensuring smooth compilation and fewer runtime issues.

---

AST Node Construction

Rules define how tokens and nested rule outputs populate AST node fields.

Example: Field Mapping

public Rule AssignmentNode = new Rule("variable:@Identifier '=' value:@Number")
    .Returns("variable", "value");

• variable: Maps the @Identifier token to this field.
• value: Maps the @Number token to this field.

Nested AST

Rules can construct hierarchical AST nodes using nested rules:

public Rule Expression = new Rule("left:@Term '+' right:@Term")
    .Returns("left", "right");

---

Adding Semantic Models

CDTk v8 introduces complex semantic transformations via models. You can add models to enrich output processing and add deeper analysis capabilities to your compiler:

Example: Using Models in Rules

class Rules : RuleSet
{
    public Rule AssignmentNode = new Rule("variable:@Identifier '=' value:@Number")
        .Returns("variable", "value");

    public MyModel Model => new MyModel(__AllRules, __Ast);
}

Models:

• Process parsed AST nodes.
• Enhance grammar capabilities with explicit, type-safe logic.
• Require an explicit constructor — no magic injection.

Learn more in the Advanced Models Guide.

---

Order and Priority

Rule evaluation follows the order in which rules are defined. Place more specific patterns before generic ones.

Example

// Correct: Keyword before Identifier
public Rule Keyword = new Rule("keyword:'if'");
public Rule Identifier = new Rule("id:@Identifier");

---

Rule Best Practices

1. Use Clear Field Names: Rule field names should clearly describe their intent.
2. Validate Often: Use the .Validate() method to ensure correctness of your rule patterns.
3. Order Matters: Place specific rules before generic ones to avoid unintended matches.
4. Leverage Models: Extend rule behavior with semantic transformations using models.

---

Debugging Rules

If a rule doesn’t behave as expected:

• Verify Tokens: Ensure matched tokens are properly defined in your TokenSet.
• Inspect Rule Order: Ensure alternate patterns don’t conflict.
• Use .Validate(): Check diagnostics for structural or logical errors.

---

Next Steps

Now that you understand rules, explore:

• Code Generation / Mapping: Mapping
• Implementing Models: Models
• Grammar Validation: Validation