[Improvement] Word breaking algorithm

[saona-raimundo]

I noticed that the word-breaking algorithm you are using in zspell::dictionary::Dictionary::check is stringmetrics::tokenizers::split_whitespace_remove_punc.
In the code, this is in dictionary.rs.

Word-breaking is actually pretty complex in Unicode, and the best I know of in Rust is unicode-segmentation it implements a trait unicode_segmentation::UnicodeSegmentation for str, following the Unicode Standard Annex #29.

Would you consider changing the algorithm and documenting how you split the words?

Whenever ICU4X is ready (Rust crate for ICU from the Unicode consortium), we could even consider locale-aware word-breaking rules. This would be a full Unicode word-breaking algorithm.

[tgross35]

Absolutely! Thank you for bringing this to my attention, I'll add it to the todo list

[tgross35]

Awesome suggestion 👍 also came with a nice 20% speed boost

Spellcheck: 188 word paragraph                                                                             
                        time:   [50.602 µs 50.869 µs 51.235 µs]
                        change: [-21.704% -20.700% -19.645%] (p = 0.00 < 0.05)
                        Performance has improved.

Change is in #25