fix: lexing utf-8 characters

[tstirrat15]

Fixes #2835

Description

See #2835. A user encountered an issue where non-ASCII characters in string literals were causing a lexer panic. This fixes that by changing how peeks and string literal lexing work.

The issue was that the code implicitly made an assumption that all of the peeked or accepted runes were the same width, which isn't necessarily going to be the case in unicode strings; l.backup() will use the currently-set width, rather than the width of the rune that is being backed up over. Manually tracking the start position works around this problem.

Changes

• Add parser test
• Add lexer test
• Change lexer logic to restore to a starting position rather than calling backup multiple times

Testing

Review. See that tests pass.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.45%. Comparing base (737bdef) to head (a5edb21).

❌ Your project check has failed because the head coverage (74.45%) is below the target coverage (75.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2836      +/-   ##
==========================================
- Coverage   74.48%   74.45%   -0.03%     
==========================================
  Files         482      482              
  Lines       56555    56560       +5     
==========================================
- Hits        42119    42104      -15     
- Misses      11482    11497      +15     
- Partials     2954     2959       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[josephschorr]

LGTM