Skip to content

perf: lightweight TextSpan for tokens (LocationSpan only for diagnostics) - #41

Merged
R-unic merged 1 commit into
rbx-loom:masterfrom
AlternativeLua:perf/token-textspan
Jul 21, 2026
Merged

perf: lightweight TextSpan for tokens (LocationSpan only for diagnostics)#41
R-unic merged 1 commit into
rbx-loom:masterfrom
AlternativeLua:perf/token-textspan

Conversation

@AlternativeLua

Copy link
Copy Markdown
Contributor

Summary

Continues the recent lexer perf series (number/string/operator lexing): tokens now carry a lightweight TextSpan — just Position + Length — plus a SourceFile reference, instead of a full LocationSpan. Line/character resolution (binary search over line starts) is deferred to Token.GetLocation(), which is only invoked on diagnostic paths.

Changes

  • New TextSpan struct (Loom.Core/Text/TextSpan.cs): position + length with value equality; no file reference, no line/char computation.
  • Token stores TextSpan Span + SourceFile File; GetLocation() materializes a LocationSpan on demand. A LocationSpan-based convenience constructor keeps synthetic-token call sites (TokenFactory, tests) unchanged.
  • Lexer tracks plain int positions through the hot loop — Location/LocationSpan structs are now only constructed inside diagnostic reporting (GetSpan).
  • Keyword lookup uses Dictionary.GetAlternateLookup<ReadOnlySpan<char>>, eliminating the per-identifier substring allocation previously needed for the KeywordMap probe.
  • Token record equality now compares two ints instead of LocationSpans, whose Location.Equals computed line/character lazily via binary search.
  • Consumers that treated a token span as a location (DiagnosticBag token overloads, parser paren-mismatch messages, TypeSolver.CheckCircular, Node.DeriveSpan, function-signature validation) go through GetLocation().
  • Dropped the now-unused LocationSpan.operator+.

Node.Span remains a LocationSpan; its construction from token locations stays lazy, so no line computation happens at node-construction time. AST/Luau snapshots are unchanged.

Testing

  • dotnet build clean
  • dotnet test: 1693 passed, 0 failed

🤖 Generated with Claude Code

…ics)

Tokens now carry a two-int TextSpan (position/length) plus a SourceFile
reference instead of a full LocationSpan. Line/character resolution is
deferred to Token.GetLocation(), which is only called on diagnostic paths.

- new TextSpan struct: position + length, value equality, no file ref
- Lexer tracks plain int positions in the hot loop; Location/LocationSpan
  are only constructed inside diagnostic reporting
- keyword lookup uses Dictionary.GetAlternateLookup<ReadOnlySpan<char>>,
  eliminating the per-identifier substring allocation
- Token record equality now compares two ints instead of LocationSpans
  (which computed line/character via binary search)
- DiagnosticBag token overloads, parser error messages, TypeSolver, and
  Node.DeriveSpan go through GetLocation()
- drop now-unused LocationSpan.operator+

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@R-unic
R-unic merged commit cc3384c into rbx-loom:master Jul 21, 2026
1 of 2 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Loom Release 1.0.0 Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants