Open
Description
Describe the bug
While working on Streaming I relaized we have a few inconsistencies in our HuggingFaceLocalChatGenerator component with regards to the streaming callback.
- We define the type of the streaming_callback in the run_async function incorrectly to be a sync signature
- and here
- We require it to be an async method here
- And we use
HFTokenStreamingHandler
which is a sync only streaming handler. So we actually need to create an async version of this for streaming to work properly with run_async. Line in question is here
Expected behavior
Use a async version of HFTokenStreamingHandler
and update the type signature
To Reproduce
If you update the types to be the AsyncStreamCallbackT mypy will throw an error saying HFTokenStreamingHandler
is not compatible
Activity
SyncStreamingCallbackT
andAsyncStreamingCallbackT
where possible #9390HuggingFaceLocalChatGenerator
#9405