[BUG]: Codebook should report diagnostic indexes based on UTF-16 bytes, and not based on graphene clusters to LSP clients

[JosephTLyons]

Issue Description

Codebook doesn't seem to account for multiple code points

Operating System

macOS

Editor

Zed

Codebook Version

No response

Configuration

Steps to Reproduce

1. Type in text that will trigger a typo
2. Add a multi-code point emoji before the typo

Expected Behavior

I expected the underline diagnostic to be aligned under the text identified as a typo

Actual Behavior

The underline was offset by one column

Code Sample

✍️ Doog

Log Output

Additional Context

No response

[blopker]

Hey! Thanks for submitting this. I have noticed this myself, but only with some emojis, and it's even worse with others. This submission prompted me to look closer at it. I think I've worked out what's happening. Although Codebook is careful to treat graphene clusters as 1 character, this is not actually how the LSP spec's indexing works. The LSP spec expects character indexes to count UTF-16 boundaries, not graphene clusters. To me, it's unintuitive, but I'm sure there's a reason for it.

Anyway, I'm going to need to do some tricky accounting to convert the cluster indexes to UTF-16 indexes before sending diagnostics to the editors. It may take some time before I get this all worked out 😭.

[JosephTLyons]

Sounds brutal! Hopefully it doesn't turn out to be too terrible.

[blopker]

Yo! I was able to get this dialed in: #125. It wasn't that bad, but I did have to rewrite a bunch of tests. Needed to be done anyway.

Putting out a new release soon!