VQA Guider — Complete Setup & Deployment Guide

What is VQA Guider?

A multimodal Video Question Answering system. Upload a video, ask a question, get an AI-powered answer with task routing (Action / Tracking / Scene). Built with FastAPI + Streamlit, powered by CLIP, DistilBERT, VQAGuiderCore, and Phi-2.

---

⚠️ Python Version Requirement

Use Python 3.10.x

• Python 3.11 also works
• Python 3.12+ is NOT compatible with transformers==4.37.2

How to check your Python version

python --version

If it shows 3.12 or higher, install Python 3.10 from https://python.org/downloads/ and use that version.

---

🚀 Setup — Every Time on a New Device

Step 1 — Clone / Copy the project

# If using git:
git clone <your-repo-url>
cd VQAguider

# Or just copy the folder to the new machine

Step 2 — Create a virtual environment

# Windows
python -m venv .venv
.venv\Scripts\activate

# Linux / macOS
python3.10 -m venv .venv
source .venv/bin/activate

Always activate the venv before running anything!

Step 3 — Install PyTorch (CUDA or CPU)

Run one of these depending on your GPU:

# ── NVIDIA GPU (CUDA 11.8) ──────────────────────────────────
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

# ── NVIDIA GPU (CUDA 12.1) ──────────────────────────────────
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

# ── NVIDIA GPU (CUDA 12.4) ──────────────────────────────────
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

# ── No GPU / CPU only ───────────────────────────────────────
pip install torch torchvision

How to find your CUDA version:

nvidia-smi        # shows CUDA Version in top-right corner
# or
nvcc --version    # shows release version

Step 4 — Install remaining dependencies

pip install -r requirements.txt

Step 5 — Place the model checkpoint

Copy vqa_model_final.pt into the models/ folder:

VQAguider/
  models/
    vqa_model_final.pt   ← put it here

Step 6 — Run the project (2 terminals)

Terminal 1 — Backend:

# From the VQAguider/ project root
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # Linux/macOS

python -m uvicorn backend.main:app --host 0.0.0.0 --port 8000

Terminal 2 — Frontend:

.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # Linux/macOS

streamlit run frontend/app.py

Then open your browser: http://localhost:8501

---

⚡ Windows One-Click Setup (first time only)

Double-click setup.bat — it will:

• Create the virtual environment
• Auto-detect your GPU and install the correct PyTorch wheel
• Install all dependencies

Then follow Step 6 above to start the servers.

---

🔍 Verify CUDA is being used

After starting the backend, open: http://localhost:8000/health

You should see:

{
  "status": "ok",
  "model_ready": true,
  "device": "cuda",
  "cuda_available": true,
  "torch_version": "2.x.x+cu121"
}

Or in the Streamlit sidebar click "Check Status".

---

❓ Troubleshooting

Problem	Cause	Fix
Python version not compatible	Python 3.12+ installed	Install Python 3.10 and create venv with it
Server runs on CPU (even with GPU)	Wrong PyTorch wheel installed	Uninstall torch, reinstall with CUDA wheel (Step 3)
500 Internal Server Error on Generate	dtype mismatch or model not loaded	Check server logs; ensure model checkpoint exists
Equal probabilities ~33% on all tasks	Fixed in latest code — update files	Pull latest code and restart server
Cannot connect to backend	Backend not running	Start the uvicorn server (Terminal 1)
transformers not found	venv not activated	Run .venv\Scripts\activate first
CLIP install fails	Network issue with GitHub zip	Run: pip install git+https://github.com/openai/CLIP.git

---

📁 Project Structure

VQAguider/
├── backend/
│   ├── main.py              ← FastAPI app + lifespan startup
│   ├── routes/
│   │   ├── video.py         ← POST /api/v1/upload_video
│   │   └── query.py         ← POST /api/v1/ask_question
│   ├── services/
│   │   ├── model_loader.py  ← loads all models at startup
│   │   ├── vqa_service.py   ← inference pipeline (video → answer)
│   │   └── cache_service.py ← file-based pickle cache
│   ├── models/
│   │   ├── architectures.py ← VQAGuiderCore, LLMProjector, etc.
│   │   └── schemas.py       ← Pydantic request/response schemas
│   ├── database/
│   │   └── db.py            ← SQLite CRUD layer
│   └── utils/
│       ├── logger.py
│       └── video_utils.py
├── frontend/
│   ├── app.py               ← Streamlit UI
│   └── api_client.py        ← HTTP client to backend
├── models/
│   └── vqa_model_final.pt   ← trained checkpoint (you provide this)
├── config.py                ← all paths, dims, hyperparams
├── requirements.txt
├── setup.bat                ← Windows one-click setup
└── README.md

---

🔌 API Endpoints

Method	Endpoint	Description
POST	/api/v1/upload_video	Upload a video file
POST	/api/v1/ask_question	Ask a question about an uploaded video
GET	/api/v1/result/{id}	Fetch a stored result by ID
GET	/health	Server and model status
GET	/docs	Interactive API documentation (Swagger)

---

🎓 Tech Stack

Component	Library
Video encoder	OpenAI CLIP ViT-B/32 + Temporal Attention Pooling
Question encoder	DistilBERT (frozen)
Task router + fusion	VQAGuiderCore (trained)
Answer generator	Microsoft Phi-2 (frozen)
Backend API	FastAPI + Uvicorn
Frontend	Streamlit
Database	SQLite (stdlib)
Cache	File-based pickle