Repository navigation
🚀 Release v2.1.0 (Zero-IndexMap Lexing & Structural Block Tokens)
This release fundamentally simplifies the text-processing layers: the preprocessor is now a pure format adapter, all comment / N-code / whitespace filtering has moved inline into the lexer, and block boundaries travel through the pipeline as first-class NewLine tokens. As a side effect, row counting for diagnostics after multi-line comments is now correct.
✨ Added
- Structural Block Tokens: Introduced
TokenKind.NewLineandNodeKind.NewLineas first-class pipeline entities. The interpreter no longer scans characters or counts rows - frame boundaries are driven exclusively by marker nodes emitted by the parser. - Four Independent Trivia Helpers:
LexingHelpernow exposesSkipWhitespaces,SkipBlockComment,SkipLineComment, andSkipLineNumberas individually testable primitives.SkipWhitespacesdeliberately excludes'\n'so structural boundaries can be emitted by the lexer.
🔄 Changed
- Preprocessor Is Now a Pure Format Adapter:
VeloxMemoryPreprocessoronly copies raw input characters into theMemoryArena. All comment, N-code, and whitespace filtering is performed inline during lexing. PreprocessingResultSlimmed Down: RemovedIndexMapPtr/IndexMapand the associatedint[]buffer. The stored text is now byte-for-byte identical to the original source, so downstream offsets are already absolute.- Parser Collapses Empty Frames:
NodeKind.NewLineis materialized only between blocks that actually contained content. Empty source lines, comment-only lines, and trailing line endings no longer produce spurious frames. - Interpreter Simplified: Removed the
currentTextIndex/currentRow/currentFrameRowtracking state and the per-node character scan. Frame flushing is now driven byNodeKind.NewLine, with a singlependingFrameContentflush at end-of-stream.
🐛 Fixed
- Row Counting After Multi-Line Comments:
IndexMapper.GetRowColumnnow returns correct row numbers for diagnostics emitted after a parenthesized comment that spans multiple physical lines. Previously,\ninside(...)was stripped by the preprocessor and the reported row was shifted.
⚡ Performance
- Lower Preprocessing Overhead: Eliminated per-character
IndexMapwrites - the preprocessor now performs a singleSpan.CopyTointo the arena. Smaller memory footprint, better cache locality. - Cheaper Interpreter Hot Loop: The interpreter no longer walks the source buffer between nodes; block boundaries arrive as tokens. Large files with many blocks benefit the most.
⚠️ Known Limitations
- Comment Inside a Numeric Literal: An input such as
G0(comment)1- invalid per ISO 6983 - now produces two separateNumbertokens instead of being transparently merged. This is a consequence of inline trivia handling and is documented as a deliberate trade-off.
📦 Installation
dotnet add package MachineGate.NG.Velox --version 2.1.0