perf: lightweight TextSpan for tokens (LocationSpan only for diagnostics) - #41
Merged
Merged
Conversation
…ics) Tokens now carry a two-int TextSpan (position/length) plus a SourceFile reference instead of a full LocationSpan. Line/character resolution is deferred to Token.GetLocation(), which is only called on diagnostic paths. - new TextSpan struct: position + length, value equality, no file ref - Lexer tracks plain int positions in the hot loop; Location/LocationSpan are only constructed inside diagnostic reporting - keyword lookup uses Dictionary.GetAlternateLookup<ReadOnlySpan<char>>, eliminating the per-identifier substring allocation - Token record equality now compares two ints instead of LocationSpans (which computed line/character via binary search) - DiagnosticBag token overloads, parser error messages, TypeSolver, and Node.DeriveSpan go through GetLocation() - drop now-unused LocationSpan.operator+ Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Continues the recent lexer perf series (number/string/operator lexing): tokens now carry a lightweight
TextSpan— justPosition+Length— plus aSourceFilereference, instead of a fullLocationSpan. Line/character resolution (binary search over line starts) is deferred toToken.GetLocation(), which is only invoked on diagnostic paths.Changes
TextSpanstruct (Loom.Core/Text/TextSpan.cs): position + length with value equality; no file reference, no line/char computation.TokenstoresTextSpan Span+SourceFile File;GetLocation()materializes aLocationSpanon demand. ALocationSpan-based convenience constructor keeps synthetic-token call sites (TokenFactory, tests) unchanged.Lexertracks plainintpositions through the hot loop —Location/LocationSpanstructs are now only constructed inside diagnostic reporting (GetSpan).Dictionary.GetAlternateLookup<ReadOnlySpan<char>>, eliminating the per-identifier substring allocation previously needed for theKeywordMapprobe.LocationSpans, whoseLocation.Equalscomputed line/character lazily via binary search.DiagnosticBagtoken overloads, parser paren-mismatch messages,TypeSolver.CheckCircular,Node.DeriveSpan, function-signature validation) go throughGetLocation().LocationSpan.operator+.Node.Spanremains aLocationSpan; its construction from token locations stays lazy, so no line computation happens at node-construction time. AST/Luau snapshots are unchanged.Testing
dotnet buildcleandotnet test: 1693 passed, 0 failed🤖 Generated with Claude Code