Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ It defines an index flow like this:
| [Embeddings to Qdrant](examples/text_embedding_qdrant) | Index documents in a Qdrant collection for semantic search |
| [FastAPI Server with Docker](examples/fastapi_server_docker) | Run the semantic search server in a Dockerized FastAPI setup |
| [Product_Taxonomy_Knowledge_Graph](examples/product_taxonomy_knowledge_graph) | Build knowledge graph for product recommendations |
| [Image Search with Vision API](examples/image_search_example) | Generates detailed captions for images using a vision model, embeds them, enables semantic search via FastAPI and served on a React frontend.|
| [Image Search with Vision API](examples/image_search_example) | Generates detailed captions for images using a vision model, embeds them, enables live-updating semantic search via FastAPI and served on a React frontend|

More coming and stay tuned 👀!

Expand Down
4 changes: 3 additions & 1 deletion examples/image_search_example/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
- QDrant for Vector Storage
- Ollama Gemma3 (Image to Text)
- CLIP ViT-L/14 - Embeddings Model
- Live Update

## Make sure Postgres and Qdrant are running
```
Expand Down Expand Up @@ -42,12 +43,12 @@ pip install -r requirements.txt
```

### Place your images in the `img` directory.
- No need to update manually. CocoIndex will automatically update the index as new images are added to the directory.


## Run Backend
```
python main.py cocoindex setup
python main.py cocoindex update
uvicorn main:app --reload --host 0.0.0.0 --port 8000
```

Expand All @@ -59,3 +60,4 @@ npm run dev
```

Go to `http://localhost:5174` to search.

8 changes: 6 additions & 2 deletions examples/image_search_example/main.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from dotenv import load_dotenv
import cocoindex
import datetime
import os
import requests
import base64
Expand All @@ -14,7 +15,7 @@
OLLAMA_MODEL = "gemma3"

# 1. Extract caption from image using Ollama vision model
@cocoindex.op.function()
@cocoindex.op.function(cache=True, behavior_version=1)
def get_image_caption(img_bytes: bytes) -> str:
"""
Use Ollama's gemma3 model to extract a detailed caption from an image.
Expand Down Expand Up @@ -55,7 +56,8 @@ def caption_to_embedding(caption: cocoindex.DataSlice) -> cocoindex.DataSlice:
@cocoindex.flow_def(name="ImageObjectEmbedding")
def image_object_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
data_scope["images"] = flow_builder.add_source(
cocoindex.sources.LocalFile(path="img", included_patterns=["*.jpg", "*.jpeg", "*.png"], binary=True)
cocoindex.sources.LocalFile(path="img", included_patterns=["*.jpg", "*.jpeg", "*.png"], binary=True),
refresh_interval=datetime.timedelta(minutes=1) # Poll for changes every 1 minute
)
img_embeddings = data_scope.add_collector()
with data_scope["images"].row() as img:
Expand Down Expand Up @@ -101,6 +103,8 @@ def startup_event():
query_transform_flow=caption_to_embedding,
default_similarity_metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY,
)
app.state.live_updater = cocoindex.FlowLiveUpdater(image_object_embedding_flow)
app.state.live_updater.start()

@app.get("/search")
def search(q: str = Query(..., description="Search query"), limit: int = Query(5, description="Number of results")):
Expand Down