WebCraze: Unleash the Power of Information Retrieval! 🌐

Welcome to the Info Retrieval Wonderland! This project is an exhilarating journey through the realms of web data, index compression, and data analysis. Whether you're a coding wizard or just getting started, there's something magical for everyone! 🚀✨

📚 Section 1: Web Adventures

Part 1: Index Construction 🏗️

Explore the web with Python magic! Crawl top 20 pages for queries like "Forests of India" and build an inverted index. Witness the dance of Boolean queries and unveil the power of Tiger AND Safari! 🐯🌐

Part 2: Merge Posting-Lists 🔄

Master the art of intersecting postings with Boolean queries. Get ready for a wildlife adventure with queries like Wildlife AND Poaching. The most restrictive intersections first - because efficiency is key! 🦓🔍

Part 3: Adding Skip-Pointers ⚡

Re-index with skip-pointers and witness the speed boost! Run queries 100 times and compare time taken for skip-pointers vs. without. Skip into the future of efficient searching! 🕰️🚀

Part 4: Spelling Correction 📝

Embark on a spelling adventure! Create a 3-gram index and correct queries like Tiger AND Saphari. Explore how our correction techniques impact the quality of retrieved documents. 🐅📚

Part 5: Scoring 🌟

Level up! Extend the system to perform TF-IDF scoring. Witness the magic of sorted document IDs matching queries. Because scoring adds a touch of enchantment! ⚖️📊

📦 Section 2: Index Compression Magic

Part 1: Medical Abstracts Compression 💊

Dive into the world of medical abstracts. Create an inverted index and perform TF-IDF scoring. Unravel the secrets of 20 queries and see the space taken by the dictionary! 📑💡

Part 2: Compression Variations 🛠️

Apply dictionary string compression with and without blocking. Witness the evolution of dictionary sizes and query resolution times. Because compression is an art! 🗜️💽

🚀 Section 3: Data Analysis Odyssey

Part 1: Dataset Evaluation 📊

Analyze the dataset using Python magic! Evaluate inter-annotator agreement, build inverted indices, and compare Elasticsearch performance. It's a data analysis odyssey! 📈🔍

Part 2: Pseudo-Relevance Feedback and Query Expansion 🔍

Explore pseudo relevance feedback and query expansion. Find the alpha maximizing MAP and witness the impact on IR engine performance. Because relevance is the name of the game! 🎮💬

Part 3: Document Ranking Showdown 🥋

Witness the battle of document ranking models - TF-IDF, BM25, Language Model, and LSI. Precision, recall, MAP - the metrics showdown begins! 📊🏆

Part 4: Clustering Magic 🌐

Apply K-Means and hierarchical clustering on chosen documents. Discover the secrets of RSS plots and compare purity/NMI values. Because clustering adds an extra layer of magic! ✨🔗

🔗 Section 4: Querying Beyond Boundaries 🎮

Explore the universe of video game sales with Elasticsearch. Pose questions, extract insights, and unveil the gaming legends. Because every game has a story! 🚀🎮

🚀 Getting Started

1. Clone the repository.
2. Navigate to the respective sections you want to explore.
3. Follow the instructions in each section's README to unleash the magic.

Explore the wonders of information retrieval and data analysis! Each section is a new adventure, so grab your keyboard and embark on a journey through the code. Happy coding! 🚀🎮🕵️‍♂️