Skip to content

v3.3.2

Choose a tag to compare

View all tags
@gbrandtio gbrandtio released this 11 Jul 21:44
v3.3.2
73bdfc5
  • Improvement: Move FsmLexer from hardcoded C-family comment/string masking to injected LexicalProfile objects, enabling language-specific accuracy without code forks. Simultaneously optimize the hot tokenization path to operate exclusively on code units (integers) instead of String indexing, eliminating per-character allocations.
  • Improvement: Add LexicalProfile definitions for 10 languages (C, Python, Go, Ruby, Lua, Shell, XML, etc.), each with correct line-comment prefixes, block delimiters, and string syntax. Extend DefaultProfiles with new language support and file extensions, refactoring extension lookup to iterate through all profiles.
  • Improvement: Improve number literal scanning to handle radix prefixes (0x, 0b, 0o), digit separators, decimals, and scientific notation as atomic tokens. Replace substring-based character tests with precomputed lookup tables.

Full Changelog: v3.3.1...v3.3.2

Assets 10
Loading