[Feature Request] Native support for multi-modal RAG (text + images)

[rehan243]

Feature Description

With the rise of vision-language models (GPT-4V, LLaVA, CogVLM), it would be valuable to have native multi-modal document support in Haystack pipelines.

Current Limitation

Currently, image content in PDFs/documents is lost during ingestion. Users need custom extractors to handle images alongside text.

Proposed Enhancement

1. Multi-modal document parser that extracts text AND images
2. Multi-modal embeddings (CLIP-style) for image chunks
3. Multi-modal retriever that searches across text and image content
4. VLM integration for answer generation from mixed context

Use Case

Technical documentation with diagrams, medical records with scans, financial reports with charts - all common enterprise use cases where image understanding is critical for accurate retrieval.

Would love to hear the team's thoughts on this direction!

[sjrl]

Hey @rehan243 we do have multimodal support for images.

Take a look at these resources:

• Introduction to Multimodal Text Generation
• Tutorial: Creating Vision+Text RAG Pipelines
• Image Converters API: https://docs.haystack.deepset.ai/reference/image-converters-api
• All of our ChatGenerators already support ImageContent as input and can support generation from mixed content (check the tutorials above).
• We also have an LLM powered extractor like LLMDocumentContentExtractor that uses a vision LLM to extract textual content from images

[rehan243]

Hi @sjrl,
Thank you for sharing this I'll go through this and definitely it brings a lot of value to the product with this capability.

[julian-risch]

I'll close this issue given that @sjrl provided links to our documentation that address how Haystack has built-in support for multi-modal RAG. @rehan243 Feel free to reopen the issue/open a new issue if you miss a feature for building your multi-modal RAG application