New Attachments system

[enricoros]

To enable a great UX in attaching documents, enable automatic conversion to the supported types, and enable multi-part attachments, which in turn enables new features.

This feature aims at recreating the attachment system of big-AGI so that documents are more 'live', converted on the fly, inlined in the composer (as option), and enable attachment of images.

Requirements:

• implemented as a useAttachments(...) hook, in a separate folder from the composer
• at least methods such as 'attachFiles(..)', 'attachClipboard(...)', 'async inlineTextAttachments(...)', and all the fitting methods must be implemented
• attachments can come (see the Composer code) from a File dialog, from the Clipboard, from Drag & drop, from CTRL+V, etc.
• inputs that do not have a single way of translating to outputs, need to show a conversion dialog (e.g. image can be attached as image, or as text (via ocr conversion), or as text (via image description conversion)'
• a translation mapping must exist so that every input type, with different degrees of completeness in qualification, can find an output type through a converter
• the Converter should be a stack of conversion operations to be approved by the user
  • for a single file, a 'radio box' must be displayed to select which type, as noted in the image above
  • for the image case, the useAttachment component needs to be aware of whether the caller can attach images or not
• the attachments (or 'outputs') that have been converted and ready, must have a React component to display them as a horizontally-scrollable row of boxes
  • there must be a 'x' sign on each attachment to delete it (with confirmation) from the list
  • there must be options to sort the list, moving attachments around
• Close all the other items, below
• Update the release notes (below) for publishing

Future:

• Text file previews
• Image attachment pipeline
• Image attachment previews
• FileWithHandle (edit/save-over)

Release notes:

• 6 ways of attaching files: open file, drop from File Explorer or the IDE, paste (of a screenshot, for instance), snap form camera, using the 'paste' button (shift + ctrl + v) (may be removed in the future), paste a URL
• Automatic conversion to the best format for the active Chat LLM (and the user can switch converters):
  • PDFs: auto-converted to text
  • Images: auto-OCR (for non-vision networks) and future support for Vision (sending images)
  • Html Tables -> Markdown
  • Text -> Text (within markdown blocks)
• Camera now produces a photo that you can send to OCR (e.g. you're on mobile, looking at a food menu) or to a multimodal network (e.g. you want to ask non-textual, unstructured, information)
• Large performance optimization as a result of not having every attachment inlined, and caching intermediaries

[enricoros]

Large progress:

[enricoros]

Remaining as of Dec 5, 2023:

• Extract menu
• Remove all the code in port.ts, item by item
• Composer:
  • Evaluate whether to call Convert from Decide, or make that chain more explicit
  • Reintroduce the templating system / or swap out all uses with consistency
  • Inline shall return a MultiPart; each Part shall describe whether it's concatenated to the prompt, or sent separately
    • All Text Outputs shall be concatenate'able (make sure the token count for a concat or multiple does not change in shape)
    • All Image outputs shall be standalone
    • FUTURE - actually implement the image pipeline of the Attachment process
  • Make sure 'inlineAttachments()' is stable and synchronous (if possible)
  • Implement inlining of a single attachment
  • Implement inlining of all attachments (by user button)
  • Implement inlining of all attachments (when clicking Send)
  • Disable the attachment system on non-Chat controllers
  • Introduce the concept of Controllers for the current chat, or maybe move to a future issue
• Check the Camera for the old OCR pipeline and decide if/what to do
• Reduce yellow/red tooltips in favor of the text in the menu?
• Decide whether to change the name of 'Rich Text'
• !!! OnNewMessage called by more than sendText...

[enricoros]

Everything that was left here is now in #266