Mini Search Engine

A lightweight web search engine built with Go (backend) and Vue.js (frontend). The system can crawl websites, index content, and provide search functionality.

Features

• Web crawler with configurable depth and page limits
• In-memory indexing with persistence
• Full-text search
• RESTful API
• Modern Vue.js frontend with Tailwind CSS

Prerequisites

• Go 1.25.1 or higher
• Node.js 20.19.0+ or 22.12.0+
• npm or yarn

Getting Started

1. Clone the Repository

git clone <repository-url>
cd mini_search_engine

2. Backend Setup

Navigate to the backend directory:

cd backend

Install dependencies:

go mod download

3. Run the Backend Server

Start the server with crawling:

go run cmd/titan/main.go -crawl "https://example.com" -max-pages 100 -max-depth 3 -workers 5 -port 8080

Or start without crawling (uses existing index):

go run cmd/titan/main.go -port 8080

Available flags:

• -crawl: URL to start crawling from
• -max-pages: Maximum number of pages to crawl (default: 100)
• -max-depth: Maximum crawl depth (default: 3)
• -workers: Number of concurrent workers (default: 5)
• -port: HTTP server port (default: 8080)

The backend will be available at http://localhost:8080

4. Frontend Setup

Open a new terminal and navigate to the frontend directory:

cd frontend

Install dependencies:

npm install

5. Run the Frontend

Start the development server:

npm run dev

The frontend will be available at http://localhost:5173 (or another port if 5173 is busy)

6. Build for Production

Backend:

cd backend
go build -o titan cmd/titan/main.go
./titan -crawl "https://example.com" -port 8080

Frontend:

cd frontend
npm run build
npm run preview

Debug Tool

To inspect the index contents:

cd backend
go run cmd/debug/main.go

This will display index statistics and test searches.

Project Structure

.
├── backend/
│   ├── cmd/
│   │   ├── titan/      # Main server
│   │   └── debug/      # Debug tool
│   └── internal/
│       ├── analysis/   # Text tokenization
│       ├── api/        # HTTP handlers
│       ├── crawler/    # Web crawler
│       ├── index/      # Search indexing
│       └── utils/      # Utilities
├── frontend/
│   ├── src/
│   │   ├── App.vue     # Main app component
│   │   └── main.js     # Entry point
│   └── public/
└── README.md

Usage

1. Start the backend server with a URL to crawl
2. Wait for the crawler to finish indexing pages
3. Open the frontend in your browser
4. Enter search queries to find indexed content

Notes

• The index is persisted to index_data.bin after crawling
• Subsequent runs will load the existing index unless you delete the file
• The crawler respects robots.txt and implements rate limiting