Skip to content

2.1.0

Latest

Choose a tag to compare

@MachineGate MachineGate released this 21 Oct 20:21
· 1 commit to main since this release

🚀 Release v2.1.0 (Zero-IndexMap Lexing & Structural Block Tokens)

This release fundamentally simplifies the text-processing layers: the preprocessor is now a pure format adapter, all comment / N-code / whitespace filtering has moved inline into the lexer, and block boundaries travel through the pipeline as first-class NewLine tokens. As a side effect, row counting for diagnostics after multi-line comments is now correct.


✨ Added

  • Structural Block Tokens: Introduced TokenKind.NewLine and NodeKind.NewLine as first-class pipeline entities. The interpreter no longer scans characters or counts rows - frame boundaries are driven exclusively by marker nodes emitted by the parser.
  • Four Independent Trivia Helpers: LexingHelper now exposes SkipWhitespaces, SkipBlockComment, SkipLineComment, and SkipLineNumber as individually testable primitives. SkipWhitespaces deliberately excludes '\n' so structural boundaries can be emitted by the lexer.

🔄 Changed

  • Preprocessor Is Now a Pure Format Adapter: VeloxMemoryPreprocessor only copies raw input characters into the MemoryArena. All comment, N-code, and whitespace filtering is performed inline during lexing.
  • PreprocessingResult Slimmed Down: Removed IndexMapPtr / IndexMap and the associated int[] buffer. The stored text is now byte-for-byte identical to the original source, so downstream offsets are already absolute.
  • Parser Collapses Empty Frames: NodeKind.NewLine is materialized only between blocks that actually contained content. Empty source lines, comment-only lines, and trailing line endings no longer produce spurious frames.
  • Interpreter Simplified: Removed the currentTextIndex / currentRow / currentFrameRow tracking state and the per-node character scan. Frame flushing is now driven by NodeKind.NewLine, with a single pendingFrameContent flush at end-of-stream.

🐛 Fixed

  • Row Counting After Multi-Line Comments: IndexMapper.GetRowColumn now returns correct row numbers for diagnostics emitted after a parenthesized comment that spans multiple physical lines. Previously, \n inside (...) was stripped by the preprocessor and the reported row was shifted.

⚡ Performance

  • Lower Preprocessing Overhead: Eliminated per-character IndexMap writes - the preprocessor now performs a single Span.CopyTo into the arena. Smaller memory footprint, better cache locality.
  • Cheaper Interpreter Hot Loop: The interpreter no longer walks the source buffer between nodes; block boundaries arrive as tokens. Large files with many blocks benefit the most.

⚠️ Known Limitations

  • Comment Inside a Numeric Literal: An input such as G0(comment)1 - invalid per ISO 6983 - now produces two separate Number tokens instead of being transparently merged. This is a consequence of inline trivia handling and is documented as a deliberate trade-off.

📦 Installation

dotnet add package MachineGate.NG.Velox --version 2.1.0