Disee: Distributed Search Engine - Gateway & Worker Architecture (Phase 1–3)

[AnshMNSoni]

🌐 Disee v0.1.0 - First Release

Disee is a containerized, distributed search engine built with FastAPI and Docker. This first release establishes the core Gateway–Worker architecture - a working distributed system that fetches real-time data from Wikipedia & StackOverflow, partitions it into chunks, and fans it out across multiple worker nodes for parallel processing and aggregation.

What's Shipped

Phase 1 - Single Node (Single Machine)

• Standalone node capable of independently managing search logic
• Handles query intake, data fetching, and result return on a single machine
• Foundation for the Gateway–Worker split introduced in later phases

Phase 2 - Multiple Nodes (Single Machine)

• Introduction of the Gateway Service as the central orchestrator
• Multiple Worker containers spun up via Docker Compose on a single host
• Gateway fans out processed data chunks to workers in parallel using httpx + asyncio

Phase 3 - Static Nodes (Multiple Machines)

• Full distribution across 3 static worker nodes on separate machines
• Gateway routes queries concurrently to all active workers and aggregates results
• Groundwork laid for future dynamic node registration and fault tolerance

Architecture

Tech Stack

Layer	Technology
Language	Python 3.9+
Web Framework	FastAPI
Frontend	React, Tailwind CSS, Framer Motion
HTTP Client	HTTPX (async)
Containerization	Docker & Docker Compose
Data Sources	Wikipedia API, StackOverflow API

Quick Start

# Clone the repo
git clone https://github.com/AnshMNSoni/Disee.git
cd Disee

# Spin up the cluster
docker-compose up --build

# Gateway search endpoint
http://localhost:8000/search?q=your_keyword

# Swagger UI
http://localhost:8000/docs/

# Frontend (optional)
cd frontend && npm install && npm run dev

Roadmap

• Phase 1: Standalone Search Node
• Phase 2: Dockerized Multi-Node Aggregation
• Phase 3: Static Distribution Across Multiple Machines
• Inverted Index with write-through distributed caching
• Performance benchmarking & latency metrics
• Dynamic Node Registration & Heartbeats
• Fault-tolerant querying (graceful node timeout handling)

---

Known Limitations

• No inverted index in this release — full sequential scan only
• No performance metrics or benchmarking dashboard yet
• Node discovery is static; dynamic registration is planned
• Local development scope — not yet production-grade

---

Contributing

Pull requests and issues are welcome!

1. Fork the project
2. Create your branch: git checkout -b feature/YourFeature
3. Commit: git commit -m 'Add YourFeature'
4. Push: git push origin feature/YourFeature
5. Open a Pull Request

Thankyou

---

This discussion was created from the release Disee: Distributed Search Engine - Gateway & Worker Architecture (Phase 1–3).

[pdeu-princepatel]

Finally we have achieved Great milestone
great work team 😃

0000000000000000001111111111111111111100000000000000000
0000000000011111110000000000000000000011111110000000000
0000000111110000000000000000000000000000000111110000000
0000111100000000000000000000000000000000000000011110000
0011100000000000000000000000000000000000000000000011100
0110000000001111111111100000000011111111111000000000110
0110000000001111111111100000000011111111111000000000110
0110000000001111111111100000000011111111111000000000110
0110000000001111111111100000000011111111111000000000110
0110000000000000000000000000000000000000000000000000110
0110000000000000000000000000000000000000000000000000110
0110001111111111111111111111111111111111111111111000110
0110001111111111111111111111111111111111111111111000110
0110001111111111111111111111111111111111111111111000110
0110000111111111111111111111111111111111111111110000110
0110000011111111111111111111111111111111111111000000110
0011100000111111111111111111111111111111111100000011100
0000111100000111111111111111111111111111110000011110000
0000000111110000011111111111111111111100000011111000000
0000000000011111110000000000000000000011111110000000000
0000000000000000001111111111111111111100000000000000000