Using Milvus for video highlights: time-ranged evidence rows instead of text chunks

[sungbin1015]

Sharing how we use Milvus to build highlight reels from baseball broadcasts. The retrieval result has to be a time range to cut from the video, not a paragraph, so the usual text-chunking approach did not fit.

One row = one piece of evidence with a time range

Instead of chunking by length, each row is a piece of "evidence" with start_sec / end_sec:

kind	Content
stt	commentary transcript
shot	caption of what is on screen
etc	OCR of the lower-third caption

Each row also carries the key of the play it belongs to, so a hit maps straight back to the source row in our relational DB (MariaDB is the source of truth, Milvus is a derived search copy).

Collection

Milvus 2.6.18, 18 fields, dynamic field off. Definition: milvus/sm_sport_baseball.json

• vector: 2560-dim, Qwen3-Embedding-4B, AUTOINDEX / COSINE
• Only the text field is embedded (truncated at 1024 bytes)
• Everything else is scalar metadata for filtering: video id, stream chunk id, kind, inning, teams, labels, scoreboard tags, score, score delta

The metadata is copied from the play onto every evidence row. It is redundant, but it lets one request do the vector search and the scalar filter together.

What we learned at query time

• Search each kind separately. In a single mixed search, shot and etc rows outnumber stt and push the commentary out of the top results.
• Always scope by video id. We never search across games.
• Skip vector search for abstract queries. A query like "highlights" has nothing concrete to match, and searching with it gave poor results. For those we use the scalar filters only.
• Use the same embedding model for indexing and querying. Indexing and search live in different services, so this is enforced through configuration.

One thing to watch for

We first created the collection through the RESTful v2 API and found that field descriptions were not kept, so we create it with pymilvus instead. This is already listed in #52163.

Try it

git clone https://github.com/SceneMakerAI/sm_db.git && cd sm_db
cp .env.example .env
docker compose up -d --wait        # MariaDB + Milvus standalone (embedded etcd, local storage)
pip install -r requirements.txt
python3 milvus/create.py

Search code: https://github.com/SceneMakerAI/agent-compose (src/domains/baseball/graph/retrieve_evidence.py)

If anyone indexes video or audio by time range, I would like to hear how you handle the per-kind imbalance. We run one search per kind; is a hybrid search with a ranker a better fit here?

[xiaofan-luan]

Have you considered grouping the evidence by time window or scene, with one row per segment and separate vector fields for STT, shot captions, and OCR (etc)?

You could then retrieve top candidates from each vector field and use hybrid_search() with RRF to fuse the three ranked lists by the same segment ID. Each retrieval path would contribute at most one hit per segment, and RRF would combine scores when the same segment appears in multiple lists. This could help balance the evidence types without repeated OCR or shot rows occupying multiple candidate slots for one segment.

Is there a reason you chose separate evidence rows over this segment-level multi-vector schema?

[xiaofan-luan]

bTW， this is really a good sharing. Thanks for the detailed background