JS-RAG: Document Parsing & Retrieval Pipeline in Node.js

A modular, JavaScript/Node.js Retrieval-Augmented Generation (RAG) engine designed to ingest PDF documents, parse and extract structured taxonomy, index semantic chunks into vector stores (Qdrant & ChromaDB), and query via local (Ollama) or hosted (Google Gemini) LLMs.

---

Key Features

• Document Ingestion: Automated PDF downloading, streaming, and markdown conversion (@opendocsg/pdf2md).
• Structure-Aware Chunking: Uses LangChain's MarkdownTextSplitter and RecursiveCharacterTextSplitter to preserve document hierarchy and section boundaries.
• Taxonomy & Entity Extraction: Custom extraction pipelines (src/extractor.js) to pull structured metadata, key concepts, and taxonomies prior to vectorization.
• Dual Vector Store Integration: Built-in connectors for both Qdrant (@qdrant/js-client-rest) and ChromaDB (chromadb).
• Flexible Model Backends: Supports local inference via Ollama as well as cloud-hosted embeddings and completions via Google Generative AI (Gemini).
• Resilient Processing: Checkpointing system (checkpoints/) allowing large ingestion runs to resume safely without reprocessing.

---

Architecture Flow

PDF Document ──► Stream & Download ──► PDF-to-Markdown ──► Text Splitter (Chunking)
                                                                    │
Vector Query ◄── Qdrant / ChromaDB ◄── Embeddings (Gemini/Ollama) ◄─┘
       │
Grounded LLM Response

---

Project Structure

├── checkpoints/       # Persisted states for resumable ingestion
├── src/
│   ├── extractor.js   # Taxonomy and structured data extraction logic
│   ├── extractor2.js  # Alternative parsing strategies
│   └── ollamHelper.js # Local Ollama service health check and client wrapper
├── main.js            # End-to-end ingestion and query pipeline runner
├── package.json       # Project dependencies and script definitions
└── .env               # Environment configuration

---

Getting Started

1. Prerequisites

• Node.js 18+
• pnpm or npm
• (Optional) Running instance of Qdrant or ChromaDB
• (Optional) Local Ollama instance

2. Installation

git clone https://github.com/tobiebenezer/JS-RAG.git
cd JS-RAG

pnpm install
# or: npm install

3. Configuration

Create a .env file in the project root:

GEMINI_API_KEY=your_gemini_api_key
QDRANT_URL=http://localhost:6333
CHROMA_URL=http://localhost:8000
OLLAMA_BASE_URL=http://localhost:11434

4. Running the Pipeline

Run the main ingestion and query script:

pnpm start
# or: node main.js

---

Dependencies

• Vector Stores: @qdrant/js-client-rest, chromadb
• Text & Document Processing: @opendocsg/pdf2md, pdfjs-dist, langchain, remark, unified
• LLM Integrations: @google/generative-ai, ollama
• Utilities: axios, dotenv, uuid

---

License

Created by Tobi Ebenezer. MIT License.