Dev UI chat page: support images in messages

[jmartisk]

No description provided.

[jmartisk]

Hm, I'm trying to implement attaching images using the Vaadin upload component and then converting to base64 in the frontend and passing it to a jsonrpc invocation, but I'm hitting some kind of limitation where the websocket for JsonRPC invocations can only handle messages up to 100 kB, anything bigger crashes the websocket :(

@phillip-kruger have you perhaps tried anything similar before?

[jmartisk]

(For clarity, the intended use case is creating an image-based chat message to pass to gpt-4-turbo or any other model that has computer vision capabilities)

[phillip-kruger]

In that case just use normal post, not the ws. You will have to use another comment or build your own

[jmartisk]

Yeah that will make things somewhat more complicated because the image will arrive completely independently of the accompanying text message, but I'll give it a try

[phillip-kruger]

Shout if I can help

[jmartisk]

Hm, I now realize, if I want to add a custom endpoint beyond JSON-RPC (either a REST endpoint or a reactive route) then I have to add some kind of HTTP extension, and we recently removed that to make quarkus-langchain4j suitable for CLI applications :(

[phillip-kruger]

This is only for Dev UI. And you have Verx-http in Dev UI

[jmartisk]

We don't have that as a dependency of the extension (neither the deployment, nor the runtime artifact). Quarkus apparently brings it in when the user runs dev mode, but we don't have any compile-time dependency on it (when compiling the extension)

[phillip-kruger]

Ok I see. Maybe I can add something to the Dev UI to make this possible.
How urgent is this ?

[jmartisk]

It would be cool to have the ability to post images for demoing on conferences (the first is in the half of May), but even if it doesn't work out I still have enough material to present :)

[jmartisk]

Simply making the jsonrpc endpoint accept messages over 100 kB isn't possible? I don't know where the limitation is coming from. But lifting it would be the easiest solution.

[phillip-kruger]

Yea I am not sure where that comes from. It's not something I set in Dev UI so must be some default of Vertx Web Sockets. If you can find and increase that, go for it !