OpenLab - NotebookLM-Inspired Research Paper Q&A System

A modern web application for managing, querying, and analyzing research papers. Features a clean NotebookLM-inspired UI with notebook management, paper organization, and AI-powered chat backed by a full RAG pipeline.

🎯 Project Status

Frontend: ✅ Fully Functional | Backend: ✅ Functional (Agentic RAG Pipeline + Paper to Code Active)

The frontend is complete with a production-ready UI. The backend runs a full agentic vision RAG pipeline: upload a PDF → pages are extracted by a VLM (metadata including a paper description is stored in memory_store.json, text chunks go to Qdrant) → at query time a planner LLM decides which actions to run (read_metadata, retrieve with optional per-paper scoping) → retrieved results are reranked by a cross-encoder → the top result's surrounding pages per paper are passed as images to a VLM for answering. The Paper to Code Lab feature is also fully implemented — a 3-stage LLM pipeline generates a runnable code repository from any uploaded paper, downloadable as a ZIP.

✨ Implemented Features

📚 Notebook Management System

• Multiple Notebooks: Create, rename, delete notebooks (ChatGPT-style sidebar)
• Isolated Data: Each notebook has its own sources, chat history, and notes
• Smart Navigation: Toggle between "Your Notebooks" and "Sources" views

📄 Document Management

• PDF Upload: Drag-and-drop upload with real backend processing
• Paper Actions: Rename and delete papers via 3-dot menu
• Upload Feedback: Shows chunks indexed after successful upload

💬 AI Chat (Agentic RAG Pipeline)

• Agentic Planner: A small LLM decides which actions to run before answering — read_metadata (fetch stored paper info) and/or retrieve (vector search, optionally scoped to a specific paper by ID)
• Multi-Paper Awareness: For comparison questions, the planner issues one retrieve action per paper; Qdrant search and reranking run independently per paper so results from all targeted papers are included
• VLM-Powered Q&A: Final answer generated by a vision model reading page images
• Hybrid Answers: When both metadata and retrieved content are needed, the metadata block is prepended to the question so the model cites both (metadata) and (Page X) sources
• Semantic Search: OpenRouter embeddings (openai/text-embedding-3-small, 4096-dim, via API) + Qdrant vector search with optional per-paper paper_id filter
• Cross-Encoder Reranking: Retrieved chunks reranked by BAAI/bge-reranker-base before answer generation
• 3-Page Context Window: VLM receives pages N-1, N, N+1 around the best match per paper
• Duplicate Upload Guard: Re-uploading the same filename returns HTTP 409
• Markdown Support: Rich text rendering with marked.js
• Citation Badges: Clickable [1], [2] badges linked to source pages
• Message History: Persistent per-notebook chat history

🧪 Lab (Generation Features)

• Paper to Code ✅ — 3-stage LLM pipeline (Planning → Analyzing → Coding) generates a runnable code repository from a paper; progress bar during generation; download result as ZIP; cancel support
• Paper to Poster, Paper to Web — UI complete, generation logic TBD

---

🏗️ Architecture

RAG Pipeline

── INGESTION (Upload) ──────────────────────────────────────────────────
PDF
  └─ PyMuPDF → page PNGs saved to disk

  For each page (one at a time via VLM):
    VLM (RAG_VISION_MODEL)
      ← page image
      → plain text  (tables described in prose)
      → metadata on page 0: title, authors, year, venue, abstract,
                             keywords, description (2-3 sentence summary)
         stored in memory_store.json (not Qdrant)

    RecursiveCharacterTextSplitter (chunk_size=500, overlap=75)
      → N chunks

    OpenRouter Embeddings API (openai/text-embedding-3-small, 4096-dim)
      → dense vector per chunk

    Qdrant (local on-disk)
      ← upsert {type, paper_id, page_num, content, page_text, vector}

── RETRIEVAL (Chat) ─────────────────────────────────────────────────────
Question
  └─ Planner LLM (RAG_PLANNER_MODEL)
       → list of actions: read_metadata / retrieve (with paper_id scope)
       (actions run in parallel via asyncio.gather)

  read_metadata action:
    └─ fetch paper(s) from memory_store.json
       → title, authors, year, abstract, description, keywords

  retrieve action (one per targeted paper for multi-doc queries):
    └─ OpenRouter Embeddings API → 4096-dim query vector
    └─ Qdrant cosine search (optionally filtered by paper_id) → top-50
    └─ Cross-encoder reranker (BAAI/bge-reranker-base, ONNX) → top-5
    └─ best result at page N per paper
         → load images: page N-1, page N, page N+1 from disk

  VLM (RAG_ANSWER_MODEL)
    ← images from all targeted papers + question (+ metadata block if read_metadata ran)
    → answer citing (Page X) for image facts, (metadata) for bibliographic facts

Key Design Decisions

Decision	Reason
Agentic planner instead of classifier router	Explicit paper_id-scoped actions enable correct multi-doc retrieval; old router couldn't target specific papers
Metadata stored in memory_store.json, not Qdrant	Metadata is structured (title/authors/year) and fetched wholesale — doesn't benefit from vector search
Paper description extracted on upload	Lets the planner identify which papers to retrieve for a given query without reading all abstracts
Per-paper independent reranking	Pooling results from all papers before reranking would consistently suppress lower-scored papers
Best result per paper for image selection	Multi-doc comparison queries need visual evidence from each paper, not just the globally highest-scored one
VLM reads images for answering	Avoids lossy text extraction for final answer; model sees original layout, tables, and figures
Tables described in prose during extraction	Avoids Markdown table embedding issues; prose embeds better
3-page window (N-1, N, N+1)	Catches content that spans a page boundary
OpenRouter embeddings instead of local fastembed	No local model to load; consistent with all other API calls; higher-dim vectors (4096) capture richer semantics
fastembed local embeddings removed	Replaced by OpenRouter API embeddings
Qdrant local on-disk	No Docker needed; resets cleanly on re-upload

---

🛠️ Tech Stack

Frontend

• Vue 3 (Composition API with <script setup>)
• Vite · Pinia · Vue Router · Tailwind CSS v3
• Lucide Vue Next · Marked.js

Backend

• FastAPI + Uvicorn (ASGI)
• Pydantic / pydantic-settings
• PyMuPDF — PDF → page images
• fastembed TextCrossEncoder — reranking only (BAAI/bge-reranker-base, ONNX)
• Qdrant Client — local on-disk vector store (4096-dim)
• OpenAI SDK — OpenRouter-compatible client (chat, vision, embeddings)
• LangChain Text Splitters — RecursiveCharacterTextSplitter
• aiofiles · python-multipart

AI / Models (via OpenRouter)

Role	Default Model	Config Key
Page extraction (VLM)	google/gemini-flash-1.5	RAG_VISION_MODEL
Answer generation (VLM)	google/gemini-flash-1.5	RAG_ANSWER_MODEL
Planner + metadata answers (text-only)	openai/gpt-4o-mini	RAG_PLANNER_MODEL
Text embeddings	openai/text-embedding-3-small	RAG_EMBEDDING_MODEL
Paper to Code generation	anthropic/claude-3.5-sonnet	PAPER2CODE_CODE_MODEL

---

📁 Project Structure

VibeProject/
├── frontend/
│   └── src/
│       ├── views/Home.vue          # Main UI (3-column layout)
│       ├── stores/app.js           # Pinia store + API calls
│       └── router/index.js
├── backend/
│   ├── app/
│   │   ├── main.py                 # FastAPI app, CORS, logging
│   │   ├── config.py               # Settings (env vars + defaults)
│   │   ├── routers/
│   │   │   ├── papers.py           # Upload, list, delete, /chunks debug
│   │   │   ├── chat.py             # RAG chat endpoint
│   │   │   └── generate.py         # Paper to Code: start/status/cancel/download
│   │   └── services/
│   │       ├── openrouter_service.py     # VLM extraction, planner, answer generation
│   │       ├── paper2code_service.py     # 3-stage Paper2Code pipeline
│   │       ├── embedding_service.py      # OpenRouter embeddings API (async)
│   │       ├── reranker_service.py       # Cross-encoder reranking (bge-reranker-base)
│   │       ├── qdrant_service.py         # Qdrant local client + search (paper_id filter)
│   │       ├── memory_store.py           # JSON persistence for paper metadata
│   │       └── pdf_service.py            # PDF → PIL page images
│   ├── requirements.txt
│   ├── .env                        # API keys (gitignored)
│   └── .env.example                # Template for .env
├── paper2code_outputs/             # Generated repos + ZIPs (outside backend/ to avoid reload)
└── README.md

---

🚀 Getting Started

Prerequisites

• Node.js 20+
• Python 3.11+
• OpenRouter API key — get one at https://openrouter.ai/keys

Backend Setup

cd backend

# Create and activate virtual environment
python -m venv venv
venv\Scripts\activate        # Windows
# source venv/bin/activate   # macOS/Linux

# Install dependencies
pip install -r requirements.txt

# Configure environment
copy .env.example .env
# Edit .env and set OPENROUTER_API_KEY=your_key_here

# Start server
uvicorn app.main:app --reload
# API available at http://localhost:8000
# Docs at http://localhost:8000/docs

Frontend Setup

cd frontend
npm install
npm run dev
# App at http://localhost:5173

Environment Variables

OPENROUTER_API_KEY=sk-or-...
RAG_VISION_MODEL=google/gemini-flash-1.5         # for page extraction
RAG_ANSWER_MODEL=google/gemini-flash-1.5         # for VLM answer generation
RAG_PLANNER_MODEL=openai/gpt-4o-mini             # for planner + metadata answers
RAG_EMBEDDING_MODEL=openai/text-embedding-3-small  # for text embeddings (4096-dim)
PAPER2CODE_CODE_MODEL=anthropic/claude-3.5-sonnet       # for Paper to Code generation

---

🔌 API Endpoints

Method	Endpoint	Description
GET	/api/v1/health	Health check
GET	/api/v1/notebooks/{id}/papers	List papers in notebook
POST	/api/v1/notebooks/{id}/papers/upload	Upload PDF (triggers ingestion)
DELETE	/api/v1/notebooks/{id}/papers/{pid}	Delete paper + Qdrant points
POST	/api/v1/notebooks/{id}/chat	Ask a question (RAG)
GET	/api/v1/notebooks/{id}/chunks	Debug: browse indexed chunks
POST	/api/v1/notebooks/{id}/papers/{pid}/generate/code	Start Paper to Code job → returns job_id
GET	/api/v1/generate/code/{job_id}/status	Poll job progress (running/done/error/cancelled)
POST	/api/v1/generate/code/{job_id}/cancel	Cancel a running job
GET	/api/v1/generate/code/{job_id}/download	Download generated repo as ZIP

---

🔧 Development Notes

Known Limitations

• In-memory metadata: Notebooks and paper metadata reset on server restart (no database)
• No auth: Notebook IDs passed directly in URL
• Qdrant storage reset required: If RAG_EMBEDDING_MODEL is changed, delete qdrant_storage/ and re-upload all papers (vector dimensions must match)

Debug Endpoint

Browse stored chunks at:

GET /api/v1/notebooks/{id}/chunks?type=text&limit=20

Logging

Per-request debug logs show the planner actions and which pages are sent to the answer VLM:

DEBUG app.routers.chat: Actions planned: [{"action": "retrieve", "paper_id": "abc...", "query": "..."}, ...]
DEBUG app.services.openrouter_service: images sent: ['327dcba8/page_1.png', '326633b7/page_2.png', ...]

---

📝 Conventions

• Naming: camelCase for JS/Vue state, snake_case for Python
• Icons: Lucide Vue Next throughout
• Event handling: @click.stop to prevent bubbling on 3-dot menus
• Async: All OpenRouter calls are async/await; page processing uses asyncio.gather

---

Last Updated: March 2026
Status: Frontend complete · Agentic RAG pipeline active · Reranking active · Multi-doc comparison active · Paper to Code active