Skip to content

Emojis are broken #3

@tiguchi

Description

@tiguchi

Problem

When using the LlamaModel.generate() method, the resulting tokens are corrupted in case the model generates emojis as response. Instead of getting the correct emojis, I get question mark blocks instead.

It seems emoji characters are split into multiple tokens, and buffering might be necessary?

Similar Issue

Here's a related issue with a comment from a user who figured out a fix for the C# binding library

ggml-org/llama.cpp#2231 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions