Skip to content

In HuggingFaceLocalChatGenerator add an async version of HFTokenStreamHandler and update type signature for async streaming callback #9391

Open
@sjrl

Description

@sjrl

Describe the bug
While working on Streaming I relaized we have a few inconsistencies in our HuggingFaceLocalChatGenerator component with regards to the streaming callback.

Expected behavior
Use a async version of HFTokenStreamingHandler and update the type signature

To Reproduce
If you update the types to be the AsyncStreamCallbackT mypy will throw an error saying HFTokenStreamingHandler is not compatible

Metadata

Metadata

Assignees

Labels

P1High priority, add to the next sprint

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @julian-risch@sjrl

    Issue actions

      In `HuggingFaceLocalChatGenerator` add an async version of `HFTokenStreamHandler` and update type signature for async streaming callback · Issue #9391 · deepset-ai/haystack