Skip to content

refactor: (server) move server_context::update_chat_msg to HTTP thread #17726

@ngxson

Description

@ngxson

update_chat_msg() calls common_chat_msg_diff::compute_diffs which can negatively impact the performance for text generation

we should find a way to move it into HTTP thread.

not 100% sure how the implement will look like, but I'm creating this issue so that it's not forgotten.

if any maintainers / contributors willing to take this task, please briefly mention your approach below

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions