Skip to content

Commit

Permalink
Update Ollama multi-modal multi-vector template README.md (#14995)
Browse files Browse the repository at this point in the history
  • Loading branch information
rlancemartin committed Dec 21, 2023
1 parent 94586ec commit 535db72
Showing 1 changed file with 9 additions and 5 deletions.
14 changes: 9 additions & 5 deletions templates/rag-multi-modal-mv-local/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@

# rag-multi-modal-mv-local

Visual search is a famililar application to many with iPhones or Android devices: use natural language to search across your photo collection.
Visual search is a famililar application to many with iPhones or Android devices. It allows user to serch photos using natural language.

With the release of open source, multi-modal LLMs it's possible to build this kind of application for yourself and have it run on your personal laptop.
With the release of open source, multi-modal LLMs it's possible to build this kind of application for yourself for your own private photo collection.

This template demonstrates how to perform visual search and question-answering over a collection of photos.
This template demonstrates how to perform private visual search and question-answering over a collection of your photos.

It uses an open source multi-modal LLM of your choice to create image summaries for each photos, embeds the summaries, and stores them in Chroma.

Given a set of photos, it will produce image summaries and index them, retrieve photos relevant to user question using the summaries, and use Ollama to run a local, open-source multi-modal LLM to answer questions about the retrieved photos.
Given a question, relevat photos are retrieved and passed to the multi-modal LLM for answer synthesis.

![mm-caption-local](https://github.com/langchain-ai/langchain/assets/122662504/cd9b3d82-9b06-4a39-8490-7482466baf43)

## Input

Expand Down Expand Up @@ -115,4 +119,4 @@ We can access the template from code with:
from langserve.client import RemoteRunnable

runnable = RemoteRunnable("http://localhost:8000/rag-multi-modal-mv-local")
```
```

0 comments on commit 535db72

Please sign in to comment.