[BUG]: Outputting Chinese characters may result in incomplete UTF8 encoding, causing garbled text

[jxq1997216]

Description

I am building a translation software, and when it translates language into Chinese, it sometimes outputs incomplete UTF8 encoding as tokens, which causes the model to freeze when it is input again as context. I think the problem should be in the Decode part, and it seems that we can detect whether a token is a complete UTF8 encoding type in the Decode part.

Reproduction Steps

1.Using the Sakura13B model to translate a Japanese text to Chinese, while adding some symbols, emojis, etc., here is my original text:
『いいね』\n『ウインナー好き』\n『おじさんのウインナーも食べて欲しいナ（笑）(^_^)😃✋💕』\n『一回茹でるといいらしいぞ』
2.Its output should include symbols like 口口
3.The model will be stuck if you input the text as context into it

Environment & Configuration

• Operating system: Windows11
• .NET runtime version: 8.0
• LLamaSharp version:0.19
• CUDA version (if you are using cuda backend): 12.7
• CPU & GPU device: NVIDIA GeForce RTX 3090

Known Workarounds

No response

[SanftMonster]

Is this behavior model-related?

[jxq1997216]

Is this behavior model-related?

yes,It's model-related and depends on whether the model will output incomplete UTF8 encoding as a token

[github-actions]

This issue has been automatically marked as stale due to inactivity. If no further activity occurs, it will be closed in 7 days.