Idea: interactive whiteboard in Jupyter AI for scribble prompts

[kolibril13]

A few days ago, I tweeted an idea about generating plots from scribbles in Juypyter:

This idea sparked some interest in the python community, and therefore the question:
Would it be possible to embed a whiteboard into Jupyter AI as well?
I'm currently using an ipywidgets approach, but having this in the chat would be interesting!
That way, users would have both text & vision when debugging code.

And I'm using the React based whiteboard Tldraw:
https://github.com/tldraw/tldraw

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[JasonWeill]

@kolibril13 Thanks for opening this enhancement request! @brichet recently released the first version of Jupyter Chat, based on Jupyter AI's chat interface, but without the requirement to have an AI conversation partner. There is also an open issue in Jupyter AI to use Jupyter Chat, if the latter is installed; see #785. Adding an image input, such as by a whiteboard or an image upload, would be an improvement for language models that accept images as part of prompts.