Skip to content

0.1.4

Choose a tag to compare

@ursm ursm released this 06 May 22:47
· 50 commits to main since this release
8e1fb3d

Another performance release on the parsing path. No behavioral changes; same 275 tests pass; RBS still validates.

Changed

  • Parsing is ~36% faster again on top of 0.1.3. Combined with the chars-Array fix in 0.1.3, parsing is ~31× faster than 0.1.0.

    Avo bench (278 KiB / 67k tokens / 2080 rules):

    tokenize parse_stylesheet
    0.1.3 406 ms 477 ms
    0.1.4 261 ms 286 ms

    Two independent wins:

    • CSS::CodePoints predicates use ASCII lookup tables. The four predicates (digit?, hex_digit?, ident_start_code_point?, ident_code_point?) now use frozen 128-element boolean tables keyed by c.ord; non-ASCII (c.ord >= 0x80) is always treated as ident-cp per spec. Replaces a chain of String#<=> range tests that were 12.8% of tokenize time. ident_code_point? drops to ~4%.
    • Position is built lazily. The tokenizer plants start_offset / end_offset / a shared @newlines reference on each Token via the new Token#assign_source!; Token#position materializes the Position Data on first read via bsearch_index. Eliminates ~67k Position allocations from the hot path along with the per-token line/column work. Worst case (every token's position queried) tokenize takes 319 ms — still 22% under the 0.1.3 baseline.
  • Internal: small simplifications. CodePoints table builders share a build_table helper. Token#inspect no longer triggers lazy Position materialization as a side effect. Tokenizer caches @chars.length as @length. Eager nil-init of the lazy-position fields removed for cleaner object shape (better for YJIT inline caches).

Removed

  • Token#assign_position! was never called outside its own definition. Hand-built tokens with a known Position should pass it through Token.new(:type, value, position: pos) instead. The RBS sig now advertises Token#assign_source! in its place.

Compatibility

Same as 0.1.0 — Ruby 3.4+, no mandatory runtime dependencies.