OffsetDB

A lightweight, multi-threaded key-value database server built in C++17, paired with a Full-Stack ecosystem.

Description

Originally designed as a standalone, high-performance storage engine using an append-only log and in-memory offset indexing, OffsetDB has evolved into a complete end-to-end architecture. The ecosystem bridges the gap between raw TCP socket programming and modern web technologies by introducing a Python FastAPI middleware and a zero-build Vanilla JS web interface for seamless data exploration.

How It Works

OffsetDB operates through a clean separation of concerns, allowing modern web clients to communicate seamlessly with a low-level TCP socket server. Here is the step-by-step lifecycle of a database query:

1. The Client Request (Frontend): When a user searches for a key in the Web Explorer, the Vanilla JavaScript triggers an asynchronous HTTP GET request (e.g., /api/get/user_1) to the Python middleware.

2. The Translation (Middleware): FastAPI receives the HTTP request, handles the CORS policies, and opens a raw TCP socket connection to the C++ engine. It translates the web request into OffsetDB's custom text protocol (e.g., sending the bytes GET user_1\n).

3. The Execution (Core Engine): The C++ Thread Pool assigns an available worker to handle the incoming TCP connection.

   • A Reader Lock (shared_mutex) is acquired, allowing multiple clients to read simultaneously without blocking each other.
   • The engine checks its in-memory hash map for the exact byte-offset of the requested key.
   • Using seekg(), the engine jumps instantly to that specific position on the physical disk, reads the value, and sends it back through the TCP stream.

4. The Response: The Python middleware receives the raw string from the C++ server, wraps it in a properly formatted JSON object, and returns it to the browser (HTTP 200 OK), where the frontend renders it with syntax highlighting.

Project Structure

OffsetDB/
├── core/  # The C++ Database Engine (Visual Studio Solution)
├── api/   # The Python FastAPI Middleware
├── web/   # The Vanilla JS/CSS Frontend Interface
├── cli/   # The Python Command Line Interface
└── data/  # Storage directory for the append-only database logs

Key Features

• Custom Thread Pool written from scratch to handle TCP connections.
• Reader-Writer Locks (shared_mutex) allows fully parallel reads from multiple clients, only locking the database when performing a write operation.
• Low RAM footprint: Instead of loading values into RAM, the database keeps the keys and their offsets in memory.
• Asynchronous Web UI: Vanilla JavaScript frontend using the fetch API for non-blocking database queries.
• Zero-Build Web Setup: No frameworks required.

Run the Ecosystem

1. Start the Database Engine

• Open core/OffsetDB.sln in Visual Studio and build the project.
• Run the compiled binary. The server will start listening for raw TCP connections on port 8080.

2. Start the API Middleware

• Open a terminal in the api/ folder
• Activate your virtual environment and install dependencies: pip install -r requirements.txt
• Run the server: python -m uvicorn api:app --port 5000

3. Open the Web Explorer

• Open web/index.html in any browser
• Enter a key and query the database visually

For details, check the specific README.md files inside each folder

Future Improvements (TODO)

• Implement signal handling for graceful shutdown to safely drain the thread pool queue
• Replace manual compaction with an automated background worker thread
• Add a "Time to Live" option for keys
• Containerize the API and Web interface using Docker