Fix CJK and emoji stream output

[MeouSker77]

Missing CJK characters is caused by decoding incomplete bytes. This PR shows a way to fix it, hope it can be a reference.

[Equim-chan]

Is there any further progress on this? This not only affects CJK but also wide Unicode characters such as emojis (see #372).

[MeouSker77]

I didn't mark it as ready for review because I'm not sure how to handle the case where logprobs is not None.
But it seems logprobs is unimportant, so I just keep it unchanged.

[MeouSker77]

hi, @gjmulder can you approve running workflow ?

[gjmulder]

@abetlen ping

[LiuYuWei]

I hope to express through this message that the Pull Request (PR) submitted by MeouSker77 can be beneficial to me. This PR is extremely important for the output of Traditional Chinese characters as it can resolve the existing issue of missing characters.

While using the current version, we found that occasionally, there would be missing characters when outputting Traditional Chinese text. This not only affects the user experience but can also lead to the loss or misinterpretation of important information.

In MeouSker77's PR, they made some adjustments and optimizations to the code to ensure that no characters are missed when outputting Traditional Chinese text. We have conducted multiple tests on this PR and confirmed its effectiveness and stability.

Therefore, we strongly recommend that you consider this PR, as it not only solves a significant issue but also enhances the user experience.

Before:

After

[MeouSker77]

@abetlen ping

[abetlen]

@MeouSker77 thank you for the contribution and my apologies for the delay on reviewing this PR, should have a release pushed shortly.