Movie Recommendation System — Deployment & Monitoring Guide

This document provides a practical, streamlined guide for running the complete recommendation system in production, including initial setup, data collection, training, container deployment, monitoring, and provenance tracking.

---

1. Initial Setup

Clone the Repository

git clone <repo-url>
cd group-project-f25-deploy-hard/movie_recommendation_data

Create Virtual Environment

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python setup.py

This creates necessary directories, configuration files, and prepares the project structure.

---

2. Collect Initial Training Data

The system trains on event streams from Kafka. You must open an SSH tunnel first.

Open SSH Tunnel (new terminal)

ssh -L 9092:localhost:9092 tunnel@128.2.220.241 -N

Run Data Collection

python data_processing/data_collection_system.py --duration-hours 0.05

This pulls recommendation events, watches, and ratings into:

data/collected/movie_data.db

This SQLite file becomes your training dataset.

---

3. Initial Training & Evaluation

Run Full Offline Pipeline

python -m pipeline.cli run-all --evaluate

This executes:

1. Preprocessing
2. Matrix Factorization training
3. Data drift check
4. Offline evaluation (RMSE, MAE, Precision@K)

Key Outputs

models/matrix_factorization_model.pkl
models/baseline_metrics.json
reports/eval_<timestamp>.json

---

4. Build & Start Production Containers

Build All Services

docker-compose build

Start System (Backend + Load Balancer + Telemetry + Prometheus + Grafana)

docker-compose up -d

Service Ports

Component	Port	Description
Backend v1	8082	Model inference API
Backend v2	8082	A/B testing alternate model
Load Balancer	8082	Splits traffic between v1/v2
Prometheus	9090	Metrics scraping backend
Grafana	3000	Monitoring UI
Telemetry Monitor	9100	Drift + quality metrics

Backend containers load models directly from:

./models:/app/models

---

5. Monitoring Dashboard & Alerts

Grafana runs automatically at:

http://<vm-ip>:3000

A dashboard monitors:

• Availability via telemetry_recommendation_status_total
• Latency via lb_request_latency_seconds_bucket
• Online accuracy via watch-hit and rating-hit metrics
• Data drift via telemetry_drift_overall_score and drift recommendation code
• Traffic split for A/B testing

Prometheus scrapes rules from:

prometheus.yml
telemetry.rules.yml

Alerts trigger when:

• Availability drops below threshold
• Drift score exceeds allowed bounds
• Error rate increases abnormally

---

6. Provenance Tracking

Every component logs the model_id used:

• serve.py exposes it in /health and in X-Model-ID response headers
• Telemetry consumer stores model_id in every event
• Retraining pipeline creates versioned artifacts under:

  models/releases/<model_id>/
  

• Model promotions update:

  models/current_model.txt
  

This ensures every prediction, evaluation, and metric is traceable to:

• Model version
• Dataset version
• Pipeline version

---

7. Retraining & Hot Reload

Run retraining:

python scripts/run_retrain.py

Backend automatically reloads models through:

POST /admin/reload-model

or via retraining script automation.

---

8. Restarting Containers After Updates

docker-compose down
docker-compose up -d --build

Backend containers stay lightweight because only model artifacts, not training data, are mounted in.

---

9. Testing

pytest tests/ --cov --cov-report=term

Expect: ~73 passing tests and ~45% overall coverage (≥85% for core models).

---

10. Health Check Verification

curl http://localhost:8082/health
curl "http://localhost:8082/recommend/12345"

---

This README serves as the authoritative guide for running, deploying, monitoring, and retraining your production recommendation system end-to-end.