Multimodal Node in Chainforge#352
Merged
ianarawjo merged 84 commits intoianarawjo:ragforgefrom May 7, 2025
Merged
Conversation
…eldsNode to store image infos
…ement in the TextFieldNode
…emo-carousel-node
… handling for image and text properties
… Add backup import unzip that works entirely in the front-end, if back-end not present.
…cipy was fixed in this release
…rontend - Implemented a new Flask API endpoint `/api/proxy-image` to fetch images while avoiding CORS issues. - Updated `ImagePreviewModal` to handle new metadata structure and display additional image details. - Enhanced `MultimediaNode` to manage image metadata and support image uploads from both URLs and local files. - Refactored `UploadFileModal` to fetch and validate image URLs, allowing users to upload images directly. - Improved handling of image data in `ResponseBoxes` to support blob URLs. - Updated `TemplateHooksComponent` to reference the new image column structure. - Added functionality to remove all media entries in `MultimediaNode`.
…ability and structure
added 3 commits
May 5, 2025 18:30
…iple files + added type FileWithContent in typing.tsx file
ianarawjo
approved these changes
May 7, 2025
ianarawjo
added a commit
that referenced
this pull request
Oct 23, 2025
* WIP: RAG Engineering in ChainForge (#350) * Added UploadNode * Added ChunkingNode * Added RetrievalNode * Added upload chunking and retrieval backend * Added multivar grouping on Join Node * Added reasoning mode to Eval Nodes * Added RAG presets and output to Multi-Eval Node * Fixed state persistance * Added RAG example flow * Add files via upload * Added FAISS vector store --------- Co-authored-by: Ilias Zouine <146049321+izouiny@users.noreply.github.com> Co-authored-by: izouiny <ilias.zouine@umontreal.ca> * Refactor rag handlers within rag dir, and move imports inside method bodies * Add MediaLookup and backend endpoints for uploading/retrieving documents. * Fixes to Chunk Node. Upgrade req Python version to 3.10. Use Markitdown for doc to text. * Add Chonkie chunkers. Fix bug-ridden existing chunker code. * Typing fix MediaLookup. Add Chonkie to requirements. * WIP: Bundle media files via .cfzip files. Add import/export of cfzip. Add backup import unzip that works entirely in the front-end, if back-end not present. * Add pytest and tests folder. Test all Chonkie chunking methods. * Fix remaining chunking methods. * Move func-specific imports from top of retrieval handlers file into specific funcs * Lock gensim to 4.3.3 and above, since a major dependency issue with scipy was fixed in this release * Bug and UI fixing for chunking methods * Refactor embeddings handlers into separate file from retrieval methods. * Added LocalVectorStore with LanceDB and tests for it. Added dependencies for lancedb. WARNING: Adding lancedb caused dependency hell between pydantic and spacy dependencies. Fix this before release * Work way out of dependency hell * Add distance metric to LanceDB vector store tests * Add numpy<2.0 to setup.py * wip testing retriever endpoint * Multimodal Node in Chainforge (#352) --------- Co-authored-by: lolo md <65371299+loloMD@users.noreply.github.com> Co-authored-by: Roy Eyono <royeyono@Roys-MBP.office1.mila.quebec> Co-authored-by: Ian Arawjo <fatso784@gmail.com> * Remove Taskfiles. Cleanup. * Many fixes and refactorings to MultimediaNode * More bug fixing * Improved Media Node UI. Renamed Multimedia node to Media Node for simplicity. * Even more bug fixing. Add ability to paste images. Pass images to custom providers. Rate limit media uploads. * Add multiple files at once. Fix issue with unique rows for images in response inspector. * Verify that other provider calls work with images. * Updated keyword-based retrieval methods and NestedMenu toggle behavior (#365) * bug fixing eval score display * side-by-side retriever display; fix carrying LLM name through retrieved chunks * Embeddings: - Local folder for huggingface - AzureOpenAI implementation - Fix pour OpenAI * - Change in lanceDB to work as one of the vector_stores - Implementation of faiss as a vector store - Front for lanceDB * local path for chunkers that need HF repo * fix error in sdpm for car type of min_chunk_size * fix include delim in SDPMChunker (should be Optional[Literal["prev", "next"]] = "prev") keep it as default : prev * local path for neural chonkie (req modification of NeuralChunker) * adding ; split in SplitNode.tsx * add metadata with python processor * -fix var input for cosine in retriver -parallelization of azure openai embeddings -adding * wip * Implement custom provider protocols and update retriever/chunker logic (#370) * Fix retrieval response inspector display * adding chunkmethod as a parameters for visualisation in inspect * - fix markitdown with {} - add Filter variable node * new retriever logic that: - allow the use of local db for vector database - return empty list if no chunk is retrieved (very important for eval) Fix in vector store: - use of local db (was always create) - use similarity threshold in faiss - front end adding * fixed custom provider setting buton logic; added 2 RAG example flows; added markdown chunker method (#383) * Added 2 example flows * fixed custom provider setting buton logic; added 2 RAG example flows; added markdown chunker method * Fix viewport node origin bug --------- Co-authored-by: MkYacine <96208478+MkYacine@users.noreply.github.com> Co-authored-by: Ilias Zouine <146049321+izouiny@users.noreply.github.com> Co-authored-by: izouiny <ilias.zouine@umontreal.ca> Co-authored-by: Roy Eyono <72818357+RoyHEyono@users.noreply.github.com> Co-authored-by: lolo md <65371299+loloMD@users.noreply.github.com> Co-authored-by: Roy Eyono <royeyono@Roys-MBP.office1.mila.quebec> Co-authored-by: Derin Akay <82316603+dxa204@users.noreply.github.com> Co-authored-by: freddupo <frederic.dupont-marillia@lautorite.qc.ca>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 This PR introduces a new node to support Vision-Language Models (VLMs) in Chainforge.
✨ Features
🧠 Supports Multiple Vision Language Model Providers
📷 Image Input Flexibility
.tsvfile in base64 format — batch mode!🕵️♂️ Vibe Check Images
🏷️ Tagging & Evaluation Tools