Advance scanner position by byte length while searching for line

[vinistock]

Motivation

Fixes #3494 and #2446. Second time is the charm 😅

In #3583, we fixed bytesize counting for UTF-8, but didn't advance @pos based on the byte size when finding the correct line, which meant the document would still get corrupted.

Implementation

I split scanning into 3 subclasses since I found the logic for each encoding to be sufficiently different. We also don't want to pay the price of checking the encoding inside the many loops.

For posterity, the spec explains that positions use code unit lengths. For each encoding, that means a different thing:

• UTF-8: code units are equivalent to bytes. We simply work directly with byte sizes
• UTF-16: code units are almost equivalent to code points. The main different is that code points after the surrogate pair are considered length 2 and everything else length 1
• UTF-32: code units is the same as code points

I implemented the logic for each in subclasses.

Automated Tests

Added a bunch of tests for each encoding and some edge cases, which should hopefully help us prevent further regressions.

Manual Tests

Tested on VS Code and NeoVim (UTF-16 and UTF-8).

[vinistock]

• Advance scanner position by byte length while searching for line #3612 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add the label graphite-merge to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[ChallaHalla]

Tested in Neovim with UTF-32 and these changes seem to fix the problem!

Seems to be good for UTF 16 and 8 as well.

[kddnewton]

I thought this functionality was largely taken care of inside of Prism::Location. Is there something missing?

[vinistock]

@kddnewton Prism::Location fully handles the other direction returning the right code units for a given AST node. This one is the reverse, we need to receive the code unit positions from the editor and ensure that we're turning them into the right string indices to update our source representation.

[alexcrocha]

LGTM 🚀

Just one small change, otherwise good to 🚢
(feel free to ignore the nits)