Introducing IOStream protocol and adding support for websockets

[davorrunje]

The goals of this PR are:

• introduction of IOStream class and a mechanism to change the default output from the console to alternative outputs such as websockets
• Introduction of IOWebsockets class that can be used to output streams to websocket.

The plan is to accomplish this by the following steps:

• introducing a new protocol IOStream with print and input methods
• creating an IOConsole implementing the protocol using console
• refactoring ConversableAgent, OpenAIClient and OpenAIWrapper to use IOStream internally
• testing ConversableAgent, OpenAIClient and OpenAIWrapper using IOConsole
• creating an IOWebSockets implementing the protocol using web sockets
• testing IOWebSockets with unit tests (a bit tricky because it has to be a multithreaded test)
• testing IOWebSockets with a new notebook
• write documentation for implementing classes implementing IOStream protocol

Why are these changes needed?

Related issue number

Closes #1199 , closes #1143 , closes #217 , closes #394

Related pull requests

#1290, #1414

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[tyler-suard-parker]

@davorrunje Congratulations, I have seen you working hard every day on this feature for months! Excellent work.

[ekzhu]

Merging this to anticipate changes in #2030 for structured output of prints.

Can you also create a topic page under /website/docs/topics/ to explain how to use this streaming util? You can simply copy the notebook to the /website/docs/topics/iostream.ipynb to make it available as a web page. I think the first example should be just about the utils not involving any agents: how to redirect iostream to a different place, etc. to illustrate the usage. Then you can show an example of using with ... context manager to change the output stream to a file, for example.

[davorrunje]

Merging this to anticipate changes in #2030 for structured output of prints.

Can you also create a topic page under /website/docs/topics/ to explain how to use this streaming util? You can simply copy the notebook to the /website/docs/topics/iostream.ipynb to make it available as a web page. I think the first example should be just about the utils not involving any agents: how to redirect iostream to a different place, etc. to illustrate the usage. Then you can show an example of using with ... context manager to change the output stream to a file, for example.

@ekzhu sure, I'll do it tomorrow night.

[beckortikov]

davorrunje Getting problem with this notebook

 - test_setup() with websocket server running on ws://127.0.0.1:8765.
 - on_connect(): Connected to client using IOWebsockets <autogen.io.websockets.IOWebsockets object at 0x0000021BDE10ED70> - Connected to server on ws://127.0.0.1:8765

 - on_connect(): Receiving message from client. - Sending message to server.

 - IOWebsockets._handler(): Error in on_connect: [Errno 22] Cache directory "./C:\Users\erifi\AppData\Local\Temp\tmpa7t3_152/42" does not exist and could not be created
Traceback (most recent call last):

[SsspongeBob]

davorrunje Getting problem with this notebook

 - test_setup() with websocket server running on ws://127.0.0.1:8765.
 - on_connect(): Connected to client using IOWebsockets <autogen.io.websockets.IOWebsockets object at 0x0000021BDE10ED70> - Connected to server on ws://127.0.0.1:8765

 - on_connect(): Receiving message from client. - Sending message to server.

 - IOWebsockets._handler(): Error in on_connect: [Errno 22] Cache directory "./C:\Users\erifi\AppData\Local\Temp\tmpa7t3_152/42" does not exist and could not be created
Traceback (most recent call last):

@beckortikov The author provided the answer above; you can scroll up to see it.

[davorrunje]

davorrunje Getting problem with this notebook

• test_setup() with websocket server running on ws://127.0.0.1:8765.

• on_connect(): Connected to client using IOWebsockets <autogen.io.websockets.IOWebsockets object at 0x0000021BDE10ED70> - Connected to server on ws://127.0.0.1:8765

• on_connect(): Receiving message from client. - Sending message to server.

• IOWebsockets._handler(): Error in on_connect: [Errno 22] Cache directory "./C:\Users\erifi\AppData\Local\Temp\tmpa7t3_152/42" does not exist and could not be created

Traceback (most recent call last):

@beckortikov The author provided the answer above; you can scroll up to see it.

Thanx for reporting it, the workaround is above. I'll update the notebook.

[beckortikov]

@davorrunje Thank you sir! I have also one questions. How can I implement it to AI chat with human_input_mode ALWAYS, where I also have function calls. Looking for a lot of issues, but couldn't find any solution how to deploy it through FastAPI, and make conversable agent. Will begreat if you could provide little code in this case. Thank you!

[davorrunje]

@davorrunje Thank you sir! I have also one questions. How can I implement it to AI chat with human_input_mode ALWAYS, where I also have function calls. Looking for a lot of issues, but couldn't find any solution how to deploy it through FastAPI, and make conversable agent. Will begreat if you could provide little code in this case. Thank you!

@beckortikov can you please check out https://captn.ai/ for an example of a larger system built with this and related code not yet merged into autogen?

I’ll factor out the mechanism to build it and open-source it in the following weeks. One major part is distributing it to multiple workers and streaming results back to clients. The streaming part was merged this week into autogen. The distribution of workers is not yet as it is a bit more complex to factor it out. There is also a code injection mechanism with a bunch of security related features. LLM never get any sensitive information about users, but functions called by LLM have access to it through code injection. There are also helpers like a decorator for checking if the user explicitly approves calling a function with specific parameters (when human mode is never). Anyway, you should be able to easily build something like this very soon.

[harrywang]

any docs on how to use streaming? Thanks a lot for the help.

[sonichi]

any docs on how to use streaming? Thanks a lot for the help.

@ekzhu @jackgerrits could you check if this is in the doc roadmap?

[davorrunje]

any docs on how to use streaming? Thanks a lot for the help.

@ekzhu @jackgerrits could you check if this is in the doc roadmap?

@sonichi there is another PR coming that will make websockets much easier to use, the main idea is to use structured JSON-encoded messages for websockets and convert them to strings for console output. Right now there is an example in notebooks on how to use it, I could quickly make a documentation out of it.

[ekzhu]

@davorrunje thanks, we can covert the notebook to a user guide page under /website/docs/topics/. I have added this to our documentation roadmap: #1597 feel free to convert that task into an issue.

[harrywang]

@davorrunje thanks a lot for adding the streaming feature. We followed the notebook https://github.com/microsoft/autogen/blob/main/notebook/agentchat_websockets.ipynb to setup a fastapi server and deploy on aws using fargate.

we are able to use the endpoint such as ws://x.x.x.x:8000/autogen locally when testing.
once we push the code to vercel, and use HTTPS to access ws://x.x.x.x:8000/autogen, we get the following error:

Mixed Content: The page at 'https://app.domain.com/autogenchat' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://x.x.x.x:8000/autogen'. This request has been blocked; this endpoint must be available over WSS.

we don't know how to setup WSS - any pointers on this? Thanks a lot.

[tyler-suard-parker]

I could be wrong here, but when you are calling a streaming websockets endpoint I think you should be calling it as wss:// rather than ws://

[RuthNjeri6]

@davorrunje Thanks a lot for the streaming feature. I have checked out the notebook that uses FastAPI. I was wondering how the same approach could be used on with a Django server. I have a chatbot that uses autogen with multiple agents including custom agent specific function. The issue is when it comes to setting up the Websocket server given that Django uses channels for this. Please point me to some resources that can get me started.

[davorrunje]

@RuthNjeri6 I am sorry for the late reply, I was travelling for the past four weeks. Do you still need helo with this?

[RuthNjeri6]

Hello Davor, Yes I still need help to integrate streaming on a Django backend.

…

On Mon, Jun 10, 2024 at 2:36 PM Davor Runje ***@***.***> wrote: @RuthNjeri6 <https://github.com/RuthNjeri6> I am sorry for the late reply, I was travelling for the past four weeks. Do you still need helo with this? — Reply to this email directly, view it on GitHub <#1551 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AINASFRLLEWNOS4DJ26AZGLZGWFUVAVCNFSM6AAAAABC25CDYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJYGEYDKMJXHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>