Here is a comprehensive and visually appealing README.md file tailored specifically to the code you’ve shared. You can copy and paste this directly into the "Add a README" section on your GitHub repository.

---

# 🔍 WeirdSecurity Search Engine

A lightweight, custom-built Web Scraper and Search Engine written entirely in Python. This project utilizes Natural Language Processing (NLP) to crawl web pages, extract metadata, and build a highly efficient inverted index for rapid, ranked text searching.

---

## ✨ Features

* **🕸️ Intelligent Web Crawler:** Uses `BeautifulSoup` to traverse web links, scrape HTML content, and extract meaningful metadata (titles and snippets).
* **🧠 NLP-Powered Indexing:** Leverages the `NLTK` library for text normalization, tokenization, stemming (Porter Stemmer), and stop-word removal.
* **⚡ Inverted Index:** Stores data in a highly optimized inverted index structure (`index.json`), allowing for lightning-fast search queries.
* **📊 Ranked Results:** Ranks search results based on term frequency, prioritizing pages where your search terms appear most often.
* **💾 Persistent Storage:** Saves both the search index and webpage metadata locally in JSON format for quick loading without re-crawling.

---

## 🛠️ Tech Stack

* **Language:** Python 3
* **HTML Parsing:** `beautifulsoup4`
* **NLP Processing:** `nltk` (Natural Language Toolkit)
* **Network Requests:** `urllib`

---

## 🚀 Getting Started

### 1. Clone the Repository
```bash
git clone [https://github.com/WeirdSecurity/Search_Engine.git](https://github.com/WeirdSecurity/Search_Engine.git)
cd Search_Engine

2. Install Dependencies

Create a virtual environment (recommended) and install the required Python packages. If you don't have a requirements.txt, you can install them directly:

pip install nltk beautifulsoup4

3. Download Required NLTK Data

Before running the engine, you need to download the necessary NLP models used for tokenization and stop-word filtering:

python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords')"

---

📖 Usage

Using the search engine is a two-step process: first, you crawl the web to build your index, and then you query that index.

Step 1: Crawl and Index

Run the web scraper to start building your database. You will be prompted to enter a "seed URL" to start crawling from.

python Web_Scraper.py

Note: The crawler respects standard web scraping delays to avoid overloading servers and will save its progress to index.json and metadata.json.

Step 2: Search

Once your index is built, run the search interface to query your database.

python run_search.py

• Simply type your query into the prompt.
• Type exit or quit to close the search engine.

---

📂 Project Structure

WeirdSecurity/Search_Engine/
│
├── Inverted_Indexing.py   # Core logic for NLP parsing and building the index
├── Web_Scraper.py         # Web crawling, HTML parsing, and metadata extraction
├── run_search.py          # The CLI interface for querying the index
├── index.json             # (Generated) The inverted index mapping words to URLs
└── metadata.json          # (Generated) Stored titles and snippets for the UI

---

🌐 Deployment Note (Coolify/Docker)

If you are deploying this project to a platform like Coolify, note that the default scripts (run_search.py) use interactive CLI inputs (input()), which will freeze in a server environment.

To host this as a web service, wrap the Inverted_Indexing logic in a lightweight web framework like Flask or FastAPI to serve the results over HTTP!

---

*Built with ❤️ by WeirdSecurity*