Fix generation of multi-token unicode characters

[ai-and-i]

This PR fixes generation of unicode strings that can only be represented by sequences of multiple tokens (closes #725).

Currently, outlines will prevent any such characters from being generated at all, even with '.*' regex. This creates a major problem when generating text in non-latin languages, or generating special characters like emojis.

This PR addresses this problem by converting character-level regex FSMs to byte-level FSMs. More precisely, it augments the FSM by adding byte-by-byte transitions that can be triggered by sub-character tokens generated by the LLM. The full-character transitions are kept as-is, so the performance of generation for normal tokens isn't impacted.

I considered other design choices, including keeping the logic of dealing with such tokens in the RegexGuide class. I don't think it can work at least for GPT2-like tokenizers (which includes gpt2, phi, qwen and other models). Such tokenizers have tokens that combine full utf8 characters followed by parts of the next character (for example, b'\x20\xf0'), and deciding whether to accept such tokens requires walking the FSM.

[rlouf]

We knew we would need something like this at some point. Thank you so much for implementing it!

[ai-and-i]

Great, thanks for merging, it unblocks me from using outlines in my project. Thanks for the great tool!