This is super interesting.

@sor4chi @nakasyou @watany-dev @hagishi and others:

Any thoughts? Or can you work on it?

I think it is a good idea!

Also I thought, what do you think about typed JSON SSE?
Now, SSE is used to stream LLM content. If we implement JSON SSE, development experience to create LLM app may get better.

JSON SSE

Great! As you mentioned, it is used in LLM apps often.

following

Hi, @yusukebe
I’m personally working on this development, but EventSource-related APIs are not part of the Web Standards. As a result, writing code that depends on it leads to errors when loading the Hono Client in various runtime environments such as Node.js. Could you suggest a good solution for this?

Hi @sor4chi

As a result, writing code that depends on it leads to errors when loading the Hono Client in various runtime environments such as Node.js. Could you suggest a good solution for this?

This means there are differences between the runtime APIs for SSE, right?

This would be really useful for LLM structured responses. Is it still in the plans to be added?

@yusukebe
Sorry, I didn't realize your response.
Yes, I think so.

I am following this package to implement SSE support for my PoC today.
https://github.com/Azure/fetch-event-source

How about this version to merge into Hono?
https://gist.github.com/elct9620/e7f34dbe22670eec5493b6503769c5a7

P.S. didn't have the error handler and abort controller yet.

I think we have support for the POST method so that users can send messages to LLM and get response streaming.

@sor4chi What do you think of the @elct9620 's implementation?

Are you assuming that you are going to add faet-event-source as a deps in Hono? Or do you mean copying that implementation into Hono and using it?

In my opinion, we need to copy or re-implement a similar behavior into Hono.

The fetch-event-source will call fetch directly and wrapper server event. However, Hono RPC has its API call method, this is why I re-implement it to make it work with Hono RPC.

I assume the API may be

const res = await agentApi[':name'].conversation.$post(...)

await res.sse({ // SSE is a type of response can be formatted like JSON
  onMessage: event => { ... }
})

In my opinion, we need to copy or re-implement a similar behavior into Hono.

The fetch-event-source will call fetch directly and wrapper server event. However, Hono RPC has its API call method, this is why I re-implement it to make it work with Hono RPC.

I assume the API may be

const res = await agentApi[':name'].conversation.$post(...)

await res.sse({ // SSE is a type of response can be formatted like JSON
onMessage: event => { ... }
})

Just adding to the discussion here, an RCP API could also be AsyncGenerator-based.

See Elysia's Eden Treaty client API for streaming responses as a unit of prior art / inspiration for something like this.

Thank you, everyone! I’ll try implementing it next week.

I tried https://github.com/Azure/fetch-event-source, but it depends on the document, so in the end, I had to polyfill global.

Azure/fetch-event-source#39

For now, I will add the eventsource polyfill and present the POC, and then leave the rest to @yusukebe’s discretion.