-
Notifications
You must be signed in to change notification settings - Fork 0
Pipeline
The CDTk pipeline is a unified, strongly typed sequence that transforms raw text into structured output. Every stage is declarative, memory‑safe, and designed to integrate seamlessly with the next. This page provides a clear overview of how CDTk processes input from start to finish.
CDTk compilers follow a deterministic pipeline:
Input Text → TokenSet → RuleSet → Models (optional) → Mapping → Output
Each stage is fully typed, allocation‑conscious, and implemented entirely in safe C#. The design keeps language definitions compact while maintaining extremely high performance across all phases of compilation.
The sections below break down each stage in detail, giving you the context and understanding needed to work confidently with the CDTk pipeline.
The pipeline begins with TokenSet, which converts raw text into a stream of typed tokens. In CDTk v8, tokens are defined as fields, and identity is based on reference equality.
- Match text using regex patterns or literal tokens
- Produce strongly typed token objects
- Ignore whitespace, comments, or other non-semantic text
- Guarantee deterministic, left-to-right scanning
- Use field-based identity for symbol resolution
CDTk uses a DFA-based engine capable of 100-200M characters/sec, with:
- No backtracking
- No allocations in hot paths
- Fully safe C# implementation
Next, the RuleSet consumes tokens and constructs a typed Abstract Syntax Tree (AST). Rules are defined as fields, discovered automatically, and resolved using reference equality.
- Define grammar using declarative rule patterns
- Support any grammar shape (recursion, ambiguity, precedence)
- Build strongly typed AST nodes
- Enforce structure and hierarchy
- Use field-based identity for rule references
CDTk's parser is table-driven and arena-allocated:
- 5-10M AST nodes/sec
- Predictable, branch-efficient loops
- Zero allocations during parse
CDTk v8 introduces a pure Model system for semantic analysis and AST transformation. Models receive all data explicitly through constructor parameters and are instantiated lazily inside MapSet.
- Transform AST nodes into semantic objects
- Attach or compute metadata
- Perform validation, type propagation, or custom checks
- Resolve identifiers or relationships using user-defined logic
The semantic engine is optimized for throughput:
- 15M+ operations/sec
- Zero-allocation dispatch loop
- Deterministic, pattern-driven execution
Finally, CDTk transforms the AST or semantic model into output using MapSet. Mappings are strongly typed and reference rule return fields directly. Output may be:
- Source code
- IR
- JSON
- Diagnostics
- Any user-defined format Mappings operate on AST nodes or model-produced objects, producing clean, deterministic output.
The CDTk pipeline is:
- Unified - all stages defined in one place
- Typed - tokens, rules, AST nodes, and models are real C# types
- Fast - lexing, parsing, and semantics are highly optimized
- Safe - implemented entirely in safe C#
- Flexible - supports any grammar and any output format This architecture allows CDTk to scale from tiny DSLs to full programming languages while remaining predictable, expressive, and extremely fast.
- Learn: Learn to use CDTk.