M3-ID: Multi-Modal Movie Identification System

M3-ID (Multi-Modal Movie Identification) is a Python-based microservice project designed to build an information retrieval system capable of identifying movies from diverse, multi-modal user queries. Users can search for a movie using a line of dialogue (text), a memorable scene (image), or a piece of music (audio).

The system uses a state-of-the-art hybrid retrieval strategy, combining dense vector (semantic) search with sparse vector (keyword) search in a Qdrant vector database.

---

Table of Contents

1. Project Goal & Description
2. System Architecture & Design
   • Query Flow (Online)
   • Ingestion Flow (Offline)
3. Methodology
4. Technology Stack
5. Data Source
6. System Requirements
   • Functional Requirements
   • Non-Functional Requirements
7. Project Deliverables
8. References
9. Project Plan & Backlog (Sprint Stories)

---

1. Project Goal & Description

Project Goal

The primary goal of this project is to design, implement, and evaluate a robust, multi-modal information retrieval system. This project aims to demonstrate mastery of modern IR concepts, including:

• Multi-modal feature extraction (text, audio, vision).
• Vector embeddings and indexing (via Qdrant).
• Advanced hybrid retrieval models (sparse + dense search).
• Microservice architecture (FastAPI + gRPC).
• Quantitative system evaluation (mAP, R@K).

Description

Identifying a specific movie from a vague memory is a common user problem. Traditional search engines struggle with multi-modal queries (e.g., "What's that movie that looks like this [image] and has this sound in it [audio]?").

M3-ID bridges this gap. It's an end-to-end system that allows users to submit text, image, or audio "clues." These clues are converted into vector embeddings by a dedicated machine-learning service. A retrieval service then queries a vector database using a novel hybrid search, which combines the "vibe" (semantic meaning) of the clues with specific keywords (like transcribed dialogue or names) to provide highly accurate and robust results.

---

2. System Architecture & Design

The system is designed as a containerized set of Python microservices, promoting scalability and separation of concerns.

                     +------------------------------------------------+
                     |                USER (e.g., Postman)            |
                     +------------------------------------------------+
                                       |
                                       | (1) REST API Request (JSON + Files)
                                       v
+-----------------------------------------------------------------------------------+
|                  (Docker Network)                                                 |
|                                                                                   |
|    +-------------------------+      (2) gRPC Request       +---------------------+
|    |   API & Retrieval       | ---------------------------> |   Feature Extractor |
|    |   Service (FastAPI)     |      (Raw Data)            |   Service (gRPC)    |
|    |                         | <--------------------------- | (Holds all ML Models)|
|    |   - Public /search API  |      (3) gRPC Response     +---------------------+
|    |   - gRPC Client         |      (Query Vectors)               |
|    |   - Qdrant Client       |                                    | (Offline Ingestion)
|    |   - Orchestrates flow   |                                    |
|    +-------------------------+                                    |
|          |         ^                                             v
|          |         | (5) Results                 +--------------------------------+
|          |         |                             |      scripts/ingest.py         |
|          | (4) Hybrid Search Query             |      (Offline Script)          |
|          |         |                             +--------------------------------+
|          v         |
|    +-------------------------+
|    |   Vector Database       |
|    |   (Qdrant)              |
|    |                         |
|    | - Stores Dense Vectors  |
|    | - Stores Sparse Vectors |
|    +-------------------------+
|                                                                                   |
+-----------------------------------------------------------------------------------+

Query Flow (Online)

1. Request: A User sends a POST /search request to the API Service (FastAPI), containing any combination of text, image files, or audio files.
2. Vectorize: The API Service acts as a gRPC client, sending the raw data to the Feature Extractor Service (gRPC).
3. Return Vectors: The Feature Extractor uses its loaded ML models (CLIP, AST, Whisper, etc.) to process the data and returns the resulting dense and sparse query vectors.
4. Search: The API Service sends the hybrid search query (with both vectors) to the Qdrant Database.
5. Response: Qdrant returns a ranked list of results, which the API Service formats as JSON and sends back to the user.

Ingestion Flow (Offline)

1. An offline script (scripts/ingest.py) is run once.
2. It iterates through the entire MSR-VTT dataset (videos, captions).
3. For each video, it calls the Feature Extractor Service to get all embeddings.
4. It uploads the fused dense vector, sparse vector, and metadata for each video to the Qdrant Database.

---

3. Methodology

Retrieval Strategy: Hybrid Search

This project's core novelty is its hybrid retrieval.

• Dense Search (via HNSW index) captures semantic meaning or "vibe." This is crucial for matching a scene's description or an image's visual content.
• Sparse Search (via inverted index, e.g., TF-IDF/BM25) captures exact keywords. This is critical for matching specific lines of dialogue, actor names, or titles.
• Fusion: We use Qdrant's native hybrid search to fuse the scores: $Score = \alpha \cdot Score_{dense} + (1 - \alpha) \cdot Score_{sparse}$ The weight $\alpha$ will be empirically tuned during the evaluation phase to maximize mAP.

Feature Fusion: Late Fusion

We employ Late Fusion. Each modality (text, image, audio) is first processed by its own "expert" model to create an embedding. These embeddings are then combined (e.g., via simple concatenation) after extraction.

• Why? This is a practical and flexible approach. It allows us to use the best available pre-trained models for each modality (e.g., CLIP for vision, AST for audio) and simplifies handling partial queries (e.g., a user only provides text).
• Alternative (Rejected): Early Fusion, which involves building a single, complex transformer to process all raw modalities simultaneously. This is difficult to train and less flexible.

---

4. Technology Stack

• Backend & API: Python 3.10+, FastAPI
• Microservice Communication: gRPC (grpcio, grpcio-tools)
• Vector Database: Qdrant (qdrant-client)
• Feature Extraction (ML/IR):
  • Vision: transformers (e.g., openai/clip-vit-base-patch32)
  • Audio (Semantic): transformers (e.g., MIT/ast-finetuned-audioset)
  • Audio (ASR): openai-whisper (for transcribing dialogue)
  • Text (Dense): sentence-transformers (e.g., all-MiniLM-L6-v2)
  • Text (Sparse): scikit-learn (for TfidfVectorizer) or a sparse model (e.g., SPLADE).
• Containerization & Tooling: Docker, Docker Compose

---

5. Data Source

• Dataset: MSR-VTT (A Large Video Description Dataset for Bridging Video and Language)
• Description: A large-scale benchmark dataset containing 10,000 video clips (totaling 41.2 hours) and 200,000 descriptive sentences (20 per clip).
• Project Usage:
  • The video clips will be treated as the "movie" documents to be retrieved.
  • The text descriptions will serve as the ground-truth queries for evaluation.

---

6. System Requirements

Functional Requirements

• FR1: Multi-modal Query Input

  • The system shall provide a REST API endpoint (POST /search).
  • The endpoint shall accept text queries.
  • The endpoint shall accept audio file uploads (e.g., .wav, .mp3).
  • The endpoint shall accept image file uploads (e.g., .png, .jpg).
  • The system shall gracefully handle queries with any combination of modalities.

• FR2: Feature Extraction

  • The system shall transcribe uploaded audio to text (for sparse search).
  • The system shall generate dense vector embeddings for text, audio, and image inputs.
  • The system shall generate sparse vector embeddings for text inputs.
  • The system shall fuse dense vectors from different modalities into a single query vector.

• FR3: Data Ingestion

  • The system shall provide an offline script to process and index the entire MSR-VTT dataset into the Qdrant database.

• FR4: Retrieval

  • The system shall query Qdrant using a hybrid (sparse + dense) search.
  • The system shall combine the scores from both searches using a weighted fusion.
  • The system shall return a ranked list of relevant video clips as a JSON response.

Non-Functional Requirements

• NFR1: Performance (Latency)
  • The end-to-end p95 search latency (API request to response) should be under 3 seconds.
• NFR2: Accuracy
  • Retrieval performance must be measured using Mean Average Precision (mAP) and Recall@K (R@K).
  • The final hybrid model must demonstrate a quantitative performance improvement over sparse-only and dense-only baselines.
• NFR3: Technology Stack
  • The system must be implemented exclusively in Python, using FastAPI, gRPC, and Qdrant.
• NFR4: Scalability
  • The FeatureExtractor gRPC service must be stateless and horizontally scalable.
• NFR5: Maintainability & Deployment
  • The entire application stack must be containerized via docker-compose.yml for one-command setup.

---

7. Project Deliverables

1. Source Code: A complete GitHub repository containing all Python code for the FastAPI API, gRPC service, and ingestion/evaluation scripts.
2. Containerized Application: A docker-compose.yml file that builds and launches the entire M3-ID system (API, Feature Extractor, Qdrant).
3. Evaluation Harness: A standalone evaluate.py script to run the test queries against the API and calculate mAP/R@K.
4. Technical Report: A final REPORT.pdf (in IEEE format) detailing the system design, architecture, methodology, experiments, and results (including performance charts).
5. Final Presentation: A PRESENTATION.pdf summarizing the project and its findings.

---

8. References

• Bose, D. et al. (2022). MovieCLIP: Visual Scene Recognition in Movies. arXiv preprint arXiv:2202.01692.
• Gabeur, V. et al. (2020). Multi-modal Transformer for Video Retrieval. ECCV 2020.
• Gong, Y. et al. (2021). AST: Audio Spectrogram Transformer. INTERSPEECH 2021.
• Mandikal, V. et al. (2024). Sparse Meets Dense: A Hybrid Approach to Enhance Scientific Document Retrieval. AAAI-SDU 2024.
• OpenAI. (2022). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv preprint arXiv:2212.04356.
• Qdrant. (n.d.). Qdrant Vector Database Documentation. Retrieved from https://qdrant.tech/documentation/
• Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML 2021.
• Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval.
• Xu, J. et al. (2016). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. CVPR 2016.

---

9. 🚀 Project Plan & Backlog

Sprint 1: Core Services & Ingestion Pipeline

Sprint Goal: Establish the foundational architecture. By the end of this week, the Qdrant database and the gRPC FeatureExtractor service will be operational and a debug dataset will be ingested.

User Stories:

• As the System Admin, I want to provision the Qdrant database service.

  • Functional Requirements: (NFR) This is setup for FR3.
  • Expected Procedures:
    1. Add the qdrant/qdrant image to docker-compose.yml.
    2. Run docker-compose up and verify the container is running and the web UI is accessible.
    3. Run a Python script using qdrant-client to create the "msr-vtt" collection.
    4. Verify the collection is configured with two vector fields: dense_fused (HNSW index) and sparse_text (inverted index).

• As the Developer, I want to implement the gRPC FeatureExtractor service.

  • Functional Requirements: FR2 (Feature Extraction - the service itself).
  • Expected Procedures:
    1. Create the feature_extractor/server.py file.
    2. Implement the gRPC server boilerplate (e.g., using grpcio-tools).
    3. Add the new service to docker-compose.yml.

• As the ML Engineer, I want to load all pre-trained models into the FeatureExtractor service.

  • Functional Requirements: FR2 (all sub-points about generating embeddings/transcriptions).
  • Expected Procedures:
    1. Add transformers, sentence-transformers, openai-whisper, etc. to requirements.txt.
    2. Write code (e.g., a singleton class) to load all models (CLIP, AST, Whisper, MiniLM, TF-IDF) into memory when the gRPC server starts.

• As the Developer, I want to define the .proto contract for multi-modal feature extraction.

  • Functional Requirements: FR1 (Query Input - defining the contract for it).
  • Expected Procedures:
    1. Create a features.proto file.
    2. Define QueryRequest (with fields like text_query, image_bytes, audio_bytes).
    3. Define QueryResponse (with fields for the resulting dense and sparse vectors).
    4. Generate the Python gRPC code from the .proto file and integrate it into the server.

• As the Data Engineer, I want to create an ingestion script that processes a 100-video debug subset and populates Qdrant.

  • Functional Requirements: FR3 (Data Ingestion), FR2 (Feature Extraction).
  • Expected Procedures:
    1. Create the scripts/ingest.py script.
    2. The script will act as a gRPC client to the FeatureExtractor service.
    3. It will loop through 100 videos from MSR-VTT, extract data (frames, audio), and send it to the gRPC service to get vectors.
    4. It will then use the qdrant-client to upload the vectors and metadata to the database.
    5. Verify the 100 items are visible and searchable in the Qdrant UI.

Sprint 2: End-to-End (E2E) Retrieval API

Sprint Goal: Implement the user-facing API and orchestrate the full retrieval pipeline. By the end of this week, a user can send a multi-modal query to the API and receive a ranked list of results.

User Stories:

• As the Developer, I want to create the FastAPI API service with a /search endpoint.

  • Functional Requirements: FR1 (API endpoint).
  • Expected Procedures:
    1. Create the api/main.py FastAPI application.
    2. Add this new service to docker-compose.yml so it runs alongside the other services.
    3. Verify the API runs and the /docs page is accessible.

• As a User, I want the /search endpoint to accept text, image, and audio queries.

  • Functional Requirements: FR1 (all sub-points about accepting inputs).
  • Expected Procedures:
    1. Implement the POST /search endpoint.
    2. Define the endpoint to accept Form data for text_query (FR1-text).
    3. Define the endpoint to accept UploadFile for image_query (FR1-image).

4. Define the endpoint to accept UploadFile for audio_query (FR1-audio).

• As the System, I want the FastAPI service to orchestrate the query-to-vector pipeline.

  • Functional Requirements: FR1, FR2 (Orchestration).
  • Expected Procedures:
    1. Implement the gRPC client logic inside the FastAPI app.
    2. In the /search endpoint, add the logic to: a. Read raw data from the request. b. Send the data to the FeatureExtractor service via gRPC. c. Receive the dense and sparse query vectors back.

• As the System, I want to execute a hybrid (sparse + dense) search against Qdrant.

  • Functional Requirements: FR4 (Hybrid retrieval, score combination).
  • Expected Procedures:
    1. Implement the qdrant-client logic inside the FastAPI app.
    2. Using the vectors from the previous step, build a hybrid Qdrant search query.
    3. Implement a simple weighted score fusion (e.g., α=0.5 to start).

• As a User, I want to receive a ranked JSON list of movie results.

  • Functional Requirements: FR4 (Ranked list).
  • Expected Procedures:
    1. The /search endpoint must return a 200 OK with a JSON list of ranked results.
    2. Each result should include its ID, score, and relevant metadata.
    3. Test the full E2E flow using Postman (send text, get JSON back).

Sprint 3: Full Ingestion, Evaluation, & Reporting

Sprint Goal: Scale the system to the full dataset, quantitatively evaluate its performance against baselines, and document all findings.

User Stories:

• As the Data Engineer, I want to run the ingestion script on the full MSR-VTT dataset.

  • Functional Requirements: FR3 (Data Ingestion).
  • Expected Procedures:
    1. Run the scripts/ingest.py script and monitor it until it successfully processes all 10,000 videos.
    2. Verify the Qdrant collection count matches the full dataset size.

• As the Developer, I want to build an evaluate.py script to measure mAP and R@K.

  • FunctionalRequirements: NFR2 (Accuracy).
  • Expected Procedures:
    1. Create the scripts/evaluate.py script.
    2. The script must load the MS-VTT test set (queries + ground truth answers).
    3. The script must programmatically call the live POST /search API.
    4. Implement logic to compare API results to the ground truth and calculate mAP and R@K.

• As the Researcher, I want to run the evaluation harness for all baselines.

  • Functional Requirements: NFR2 (Accuracy).
  • Expected Procedures:
    1. Modify the evaluation script (or API) to support search modes.
    2. Run the evaluate.py script in "sparse-only" mode and save the metrics.
    3. Run the evaluate.py script in "dense-only" mode and save the metrics.
    4. Run the evaluate.py script in the default "hybrid" mode and save the metrics.

• As the Researcher, I want to tune the hybrid search α (alpha) weight to find the optimal mAP.

  • Functional Requirements: NFR2 (Accuracy).
  • Expected Procedures:
    1. Parameterize the α (alpha) weight in the API's search logic.
    2. Run the evaluate.py script in a loop with different α values (e.g., 0.25, 0.5, 0.75).
    3. Identify and record the α value that produces the highest mAP.

• As the Author, I want to write the final technical report and presentation.

  • Functional Requirements: (Project Completion).
  • Expected Procedures:
    1. Generate plots (bar charts) comparing the mAP/R@K of the baselines vs. the tuned hybrid model.
    2. Write the final REPORT.pdf (e.g., in IEEE format), detailing the project architecture, methodology, and results.
    3. Create the final PRESENTATION.pdf summarizing the project.