🔍 Vector Space Information Retrieval System

A comprehensive Search Engine and Indexing system implementing the Vector Space Model, Positional Indexing with PySpark, and Ranked Retrieval using Cosine Similarity .

📌 Overview

This project is a practical implementation of core Information Retrieval (IR) concepts. It consists of two main pipelines:

1. Indexing Pipeline: Uses Apache Spark (PySpark) to process text documents, perform tokenization and stemming, and generate a Positional Inverted Index.
2. Search Engine GUI: A Tkinter-based desktop application that allows users to query the dataset using Boolean logic (AND, OR, NOT) and view results ranked by Cosine Similarity with a detailed breakdown of TF-IDF calculations.

🚀 Key Features

• Distributed Indexing: Utilizes PySpark RDDs to handle tokenization, stop-word filtering, and positional index creation efficiently.
• Custom Preprocessing: Implements specific stemming rules (e.g., handling irregulars like "worser" → "worse") while preserving proper nouns (e.g., "Brutus").
• Vector Space Model: Calculates Term Frequency (TF), Inverse Document Frequency (IDF), and TF-IDF weights.
• Ranked Retrieval: Ranks search results using Cosine Similarity between query vectors and document vectors.
• Educational GUI: The search interface provides a transparent view of the math, displaying a dynamic table of TF, IDF, and Normalized weights for every query term.
• Boolean Logic: Supports complex queries combining free text with boolean operators.

📂 Project Structure

File	Description
positional_index.py	The Indexer. A PySpark script that reads the corpus, processes text (tokenization/stemming), and outputs the positional_index.txt.
IR_GUI.py	The Search Engine. A Python Tkinter application that loads the dataset, builds the TF-IDF matrix in-memory, and provides the search interface.
Term_Frequency.ipynb	Analysis Notebook. A Jupyter Notebook used for validating math, calculating document lengths, and visualizing the TF-IDF matrices and normalization factors.
positional_index.txt	The Artifact. The output file generated by Spark containing the mapping of terms to documents and positions.
1.txt - 10.txt	The Corpus. A dataset of 10 text files containing Shakespearean text segments used for testing.

⚙️ Theory & Logic

1. Preprocessing (Spark)

The system reads raw text and applies specific rules:

• Tokenization: Splits text into words and tracks their position.
• Stemming: Maps irregular words to their root (e.g., worser → worse) and handles plurals, while safeguarding proper nouns.
• Positional Indexing: Stores data in the format: <term> doc_id: pos1, pos2; ...

2. Weighting & Ranking (GUI & Notebook)

The system uses the TF-IDF weighting scheme:

• TF (Term Frequency): $1 + \log_{10}(tf)$
• IDF (Inverse Document Frequency): $\log_{10}(N / df)$
• TF-IDF: $TF \times IDF$

Similarity Score: Documents are ranked based on the Cosine Similarity between the normalized Query Vector ($\vec{q}$) and Document Vector ($\vec{d}$): $$\text{Similarity}(\vec{q}, \vec{d}) = \frac{\vec{q} \cdot \vec{d}}{||\vec{q}|| \times ||\vec{d}||}$$

🛠️ Installation & Usage

Prerequisites

• Python 3.x
• Apache Spark (for running positional_index.py)
• Java (required for Spark)
• Required Python Libraries:

  pip install pandas numpy scikit-learn tabulate pyspark

Step 1: Generate the Index

Run the PySpark script to generate the positional index from your text files.

# Syntax: python positional_index.py <path_to_data_folder>
python positional_index.py ./data/