Implement Unicode grapheme clusters

[straight-shoota]

This patch adds a representation and query methods for Unicode grapheme clusters as defined in Unicode Standard Annex #29.

The data type String::Grapheme represent a grapheme cluster. Instances can be aquired through String#graphemes and String#each_grapheme. This API is purely informational and allows no modifications.

This PR builds on top #10721 by @naqvis which is based on the shard https://github.com/naqvis/uni_text_seg. The enhancements trim down a more concise API and optimize some algorithms with inspiration from https://github.com/JuliaStrings/utf8proc.

/ref https://forum.crystal-lang.org/t/unicode-text-segmentation-grapheme-clusters/3274
Closes #10721

[Blacksmoke16]

Would it make sense to merge this script with the other generate_unicode_data script so that there's only 1 thing that generates all unicode data?

[straight-shoota]

I think it makes sense to separate these concerns into individual files. They work with different data. The code is more readable when it's clearly separated.

Instead, I would consider adding a Makefile for generating data. That allows us to easily re-generate data with a single command and manage configuration such as Unicode versions in a single place.

[Blacksmoke16]

👍 Works for me. We also have that Unicode::VERSION constant we could use at least versus a make env var.

[asterite]

Wow, I was just going to ask today or tomorrow if graphemes were going to be considered for the next release, because I think they are very important and useful. This is great!

And thank you so much @naqvis for the original code! 🙏

[straight-shoota]

It has been some time since I did most of the work in this PR, just finished it up recently. It mostly just implements the suggestions made in its predecessor.

But since then I have thought about the internal representation again.

The Char | String union type is a huge improvement for grapheme clusters consisting of a single character. That should be most common in typical use cases, so it saves a big deal of allocations compared to String (or a collection of Char as in the initial propsal).

However, in some contexts, multi-character grapheme clusters are quite ubiquitous. Whether you're dealing with lots of composed emojis or scripts such as Thai, Hebrew or Arabic, or even just many decomposed diacritics, there might be a lot of string allocations for all these graphemes.

I'm going to briefly sketch some ideas for performance improvements. Obviously, we can defer such optimizations to a later point when we might already have some real-world applications for reference. However it's helpful to have some ideas in mind while defining the API because that might influence what options we can easily apply later.

1. We can expose the method String#each_grapheme_boundary as part of the public API. That's trivial and basically cost-free. However, I'd consider it more a low-level API. It can be useful for custom grapheme-based implementations.
2. Employing a StringPool to avoid repeated allocations of identical grapheme clusters. There is no way to release values from a pool, so it would be a bad idea to use a global pool. But individual pools could be useful for single iterations. This could optionally be configurable and the API could allow specifying a custom pool to be used. This would be pretty easy to implement later. We just need to add StringPool parameters to Grapheme.new and the iteration methods.
   A pool means we still need to allocate strings for every grapheme, but we can safe on duplicate allocations of the same graphemes. It is likely that the same graphemes show up repeatedly in longer texts of the same context, so that should be expected to be benefitial.
3. We could avoid allocating a string entirely by keeping a reference of the original string and making Grapheme essentially a pointer to a substring. String allocation would only happen in Grapheme#to_s. The big downside of this approach is that it keeps the original string in memory. That's a serious implication when working with large amounts of texts. If the Grapheme instaces are only used for iteration, this is not a problem. But if you start storing Graphemes somewhere, they keep pointers to the original strings alive. An example would be collecting frequencies of graphemes over a collection of texts. Assuming every evaluated texts contains a multi-character grapheme that was not encountered previously, all these strings will be kept in memory. The user would need to be aware of this behaviour and take appropriate action such as converting grapheme instances to self-owned cluster strings.

[asterite]

Is there a common maximum grapheme cluster length that we could take advantage of? For example if the "שָׁ" in "שָׁלוֹם" is composed of two chars, the letter and the mark below, then maybe we could change the internal representation to be:

Char | {Char, Char} | String

Going with that idea, we could also allow up to three or four chars:

Char | {Char, Char} | {Char, Char, Char} | {Char, Char, Char, Char} | String

I know this doesn't cover all cases, but maybe it covers most of them?

That union is not very nice, so maybe another idea could be to have it be like this:

{size: Int32, data: Char[4]} | String

That is, either a Char buffer with a size, or a string. That should simplify the internal implementation, and it would also allow us to change that 4 to 3 or 2 later on very easily.

[straight-shoota]

Yeah, I wondered about that as well. But I don't think there is a technical limit on how many codepoints can form a grapheme. You can stack many modifiers.

Maybe we could identify a size that covers a good portion of practically relevant grapheme clusters that would frequently come across. But it's hard to tell.

[beta-ziliani]

Minor comments (WIP)

[beta-ziliani]

Not a big deal, but would it be possible to have a more functional implementation of this function (that is, avoid the Pointer here)?

[straight-shoota]

I guess so. The return value could be a Tuple(Bool, Property). Not sure if that would be significantly better.

[beta-ziliani]

To me this looks good. @naqvis, since you're the original creator, would you mind taking a look at @straight-shoota's changes to make sure we're on the same page? Thanks a lot for your effort!
(Same to you @straight-shoota!)

[naqvis]

Thanks @straight-shoota for whole refactoring.

[Sija]

@straight-shoota Wouldn't grapheme_count be a more apt name here, wdyt?

[straight-shoota]

Nope. We always use size.