🗄️ PyHybridDB - Hybrid Database System

A Python-based hybrid database system combining SQL and NoSQL paradigms with a modern web-based admin panel

---

📋 Table of Contents

• Features
• Quick Start
• Installation
• Usage
• Authentication
• Configuration
• API Documentation
• CLI Commands
• Examples
• Project Structure
• Testing
• Security
• License

---

✨ Features

Core Features

• 🔄 Hybrid Data Model - SQL tables + NoSQL collections in one database
• 💾 Custom Storage Engine - Efficient .phdb file format with B-Tree indexing
• 🔍 Unified Query Language - Execute both SQL and MongoDB-style queries
• 🌐 REST API - Complete FastAPI backend with auto-generated docs
• 🎨 Web Admin Panel - Beautiful, responsive UI for database management
• 🔐 JWT Authentication - Secure token-based authentication
• 🔒 Role-Based Access Control - Admin, user, and readonly roles
• 📊 Real-time Statistics - Dashboard with database metrics
• 🔄 ACID Transactions - Transaction support with commit/rollback
• 📦 Import/Export - JSON and CSV format support

Advanced Features ✨ NEW!

• 💾 Backup & Restore - Automated backup with compression and rotation
• 📝 Audit Logging - Complete activity tracking and compliance
• 👥 User Management - Full CRUD API for user administration
• 🔗 JOIN Operations - INNER, LEFT, RIGHT, FULL OUTER joins
• 📊 Data Visualization - Charts and statistics generation
• 🔄 PostgreSQL Migration - Import from PostgreSQL databases
• 🔄 MongoDB Migration - Import from MongoDB collections
• 🔐 Encrypted Storage - AES encryption for data at rest

Technical Features

• B-Tree indexing for fast lookups
• Block-based storage with checksums
• Transaction logging with ACID compliance
• Query caching and optimization
• CORS support with configurable origins
• Environment-based configuration
• Comprehensive error handling
• SQLite-based audit logging
• Automatic backup rotation
• Password-based encryption

---

🚀 Quick Start

1. Installation

# Create virtual environment
python -m venv venv

# Activate virtual environment
.\venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install package
pip install -e .

2. Run Demo

python DEMO.py

3. Start Server

python -m pyhybriddb.cli serve
# Server runs at http://localhost:8000
# API docs at http://localhost:8000/docs

4. Open Admin Panel

Open admin/index.html in your web browser

Default Login:

• Username: admin
• Password: admin123

---

💻 Usage

Python API

from pyhybriddb import Database

# Create database
with Database(name="my_app", path="./data") as db:
    
    # SQL-like tables
    users = db.create_table("users", {
        "name": "string",
        "age": "integer",
        "email": "string"
    })
    
    # Insert records
    users.insert({"name": "Alice", "age": 30, "email": "alice@example.com"})
    
    # Query records
    all_users = users.select()
    young_users = users.select(where={"age": 25})
    
    # Update records
    users.update(where={"name": "Alice"}, updates={"age": 31})
    
    # NoSQL-like collections
    posts = db.create_collection("posts")
    
    # Insert documents
    posts.insert_one({
        "title": "Hello World",
        "tags": ["intro", "hello"],
        "author": {"name": "Alice"}
    })
    
    # Query documents
    all_posts = posts.find()
    alice_posts = posts.find({"author.name": "Alice"})

SQL Queries

from pyhybriddb.core.connection import Connection

with Database("my_db") as db:
    conn = Connection(db)
    
    # CREATE TABLE
    conn.execute("CREATE TABLE products (name string, price float)")
    
    # INSERT
    conn.execute("INSERT INTO products (name, price) VALUES ('Laptop', 999.99)")
    
    # SELECT
    result = conn.execute("SELECT * FROM products WHERE price > 500")
    
    # UPDATE
    conn.execute("UPDATE products SET price = 899.99 WHERE name = 'Laptop'")
    
    conn.commit()

NoSQL Queries

# MongoDB-style queries
conn.execute('db.posts.insertOne({"title": "Hello", "tags": ["intro"]})')
conn.execute('db.posts.find({"tags": "intro"})')
conn.execute('db.posts.updateOne({"title": "Hello"}, {"$set": {"views": 100}})')
conn.execute('db.posts.aggregate([{"$sort": {"views": -1}}, {"$limit": 10}])')

---

🔐 Authentication

Overview

PyHybridDB uses JWT (JSON Web Token) authentication to secure the API and admin panel.

Default Credentials

• Username: admin
• Password: admin123

⚠️ IMPORTANT: Change these in production!

API Authentication

import requests

# Login
response = requests.post('http://localhost:8000/api/auth/login', json={
    'username': 'admin',
    'password': 'admin123'
})

data = response.json()
token = data['access_token']

# Use token for authenticated requests
headers = {'Authorization': f'Bearer {token}'}

response = requests.post(
    'http://localhost:8000/api/databases',
    json={'name': 'my_db'},
    headers=headers
)

---

⚙️ Configuration

Environment Variables

PyHybridDB uses environment variables for configuration. After installing via pip, you can configure it in multiple ways:

Method 1: Create .env File (Recommended)

Create a .env file in your project directory:

# Create .env file
touch .env  # Linux/Mac
# or
New-Item .env  # Windows PowerShell

Add your configuration:

SECRET_KEY=your-super-secret-key-change-this
ADMIN_PASSWORD=your-secure-password
API_PORT=8000
DEFAULT_DB_PATH=./data

Method 2: Set Environment Variables

# Windows PowerShell
$env:SECRET_KEY = "my-secret-key"
$env:ADMIN_PASSWORD = "secure-password"
$env:API_PORT = "8080"

# Linux/Mac
export SECRET_KEY="my-secret-key"
export ADMIN_PASSWORD="secure-password"
export API_PORT="8080"

Method 3: Programmatic Configuration

import os
os.environ['SECRET_KEY'] = 'my-secret-key'
os.environ['DEFAULT_DB_PATH'] = './my_data'

from pyhybriddb import Database
db = Database("my_app")

Available Settings

# Security
SECRET_KEY=your-super-secret-key-change-this
JWT_ALGORITHM=HS256
ACCESS_TOKEN_EXPIRE_MINUTES=30

# Admin Credentials
ADMIN_USERNAME=admin
ADMIN_PASSWORD=admin123
ADMIN_EMAIL=admin@pyhybriddb.com

# API Configuration
API_HOST=0.0.0.0
API_PORT=8000

# Database
DEFAULT_DB_PATH=./data
LOG_LEVEL=INFO
CORS_ORIGINS=*

View Configuration

python -m pyhybriddb.cli config

Generate Secure SECRET_KEY

import secrets
print(secrets.token_urlsafe(32))

---

🆕 Advanced Features Usage

Backup & Restore

from pyhybriddb.utils.backup import BackupManager

backup_mgr = BackupManager()

# Create backup
backup_file = backup_mgr.create_backup("./data/my_db.phdb", compress=True)

# List backups
backups = backup_mgr.list_backups("my_db")

# Restore backup
restored = backup_mgr.restore_backup(backup_file)

# Auto-backup with rotation
backup_mgr.auto_backup("./data/my_db.phdb", max_backups=5)

Audit Logging

from pyhybriddb.utils.audit import get_audit_logger, AuditAction

audit = get_audit_logger()

# Log action
audit.log(
    action=AuditAction.CREATE_DATABASE,
    user="admin",
    database_name="my_db",
    success=True
)

# Get logs
logs = audit.get_logs(action=AuditAction.INSERT, limit=100)

# Get statistics
stats = audit.get_statistics()

JOIN Operations

from pyhybriddb.query.joins import JoinExecutor, JoinType

# Execute JOIN
result = JoinExecutor.execute_join(
    left_table=users.select(),
    right_table=orders.select(),
    left_key="id",
    right_key="user_id",
    join_type=JoinType.INNER
)

PostgreSQL Migration

from pyhybriddb.migration import PostgreSQLMigration
from pyhybriddb import Database

# Connect to PostgreSQL
pg_migration = PostgreSQLMigration({
    'host': 'localhost',
    'port': 5432,
    'database': 'mydb',
    'user': 'postgres',
    'password': 'password'
})

# Migrate to PyHybridDB
with Database("migrated_db") as db:
    results = pg_migration.migrate_database(db)
    print(f"Migrated {sum(results.values())} records")

MongoDB Migration

from pyhybriddb.migration import MongoDBMigration
from pyhybriddb import Database

# Connect to MongoDB
mongo_migration = MongoDBMigration({
    'host': 'localhost',
    'port': 27017,
    'database': 'mydb'
})

# Migrate to PyHybridDB
with Database("migrated_db") as db:
    results = mongo_migration.migrate_database(db)
    print(f"Migrated {sum(results.values())} documents")

Encrypted Storage

from pyhybriddb.utils.encryption import EncryptionManager

# Setup encryption
encryption = EncryptionManager()

# Encrypt data
encrypted = encryption.encrypt_string("sensitive data")

# Decrypt data
decrypted = encryption.decrypt_string(encrypted)

# Encrypt files
encryption.encrypt_file("data.phdb", "data.phdb.encrypted")
encryption.decrypt_file("data.phdb.encrypted", "data.phdb")

---

📚 API Documentation

Base URL

http://localhost:8000/api

Interactive Docs

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Key Endpoints

Authentication

• POST /api/auth/login - Login and get JWT token
• GET /api/auth/me - Get current user info

Databases

• POST /api/databases - Create database
• GET /api/databases - List databases
• GET /api/databases/{name} - Get database details
• DELETE /api/databases/{name} - Delete database

Backup & Restore ✨ NEW!

• POST /api/databases/{name}/backup - Create backup
• GET /api/databases/{name}/backups - List backups
• POST /api/databases/{name}/restore - Restore backup

Audit Logs ✨ NEW!

• GET /api/audit/logs - Get audit logs (admin only)
• GET /api/audit/statistics - Get audit statistics (admin only)

User Management ✨ NEW!

• POST /api/users - Create user (admin only)
• GET /api/users - List users (admin only)
• GET /api/users/{username} - Get user details
• PUT /api/users/{username} - Update user (admin only)
• DELETE /api/users/{username} - Delete user (admin only)

Data Visualization ✨ NEW!

• GET /api/databases/{db}/tables/{table}/visualize - Generate charts

Tables

• POST /api/databases/{db}/tables - Create table
• GET /api/databases/{db}/tables - List tables
• POST /api/databases/{db}/tables/{table}/records - Insert record
• GET /api/databases/{db}/tables/{table}/records - Get records

Collections

• POST /api/databases/{db}/collections - Create collection
• POST /api/databases/{db}/collections/{coll}/documents - Insert document
• GET /api/databases/{db}/collections/{coll}/documents - Get documents

Query

• POST /api/databases/{db}/query - Execute query (SQL or NoSQL)

---

🖥️ CLI Commands

Database Management

# Create database
python -m pyhybriddb.cli create my_database

# Database info
python -m pyhybriddb.cli info my_database

Server Management

# Start server
python -m pyhybriddb.cli serve

# Custom host and port
python -m pyhybriddb.cli serve --host 127.0.0.1 --port 8080

# Enable auto-reload
python -m pyhybriddb.cli serve --reload

Interactive Shell

# Start shell
python -m pyhybriddb.cli shell my_database

# In shell:
phdb> CREATE TABLE users (name string, age integer)
phdb> INSERT INTO users (name, age) VALUES ('Alice', 30)
phdb> SELECT * FROM users
phdb> db.posts.insertOne({"title": "Hello"})
phdb> exit

Configuration

# View configuration
python -m pyhybriddb.cli config

---

📝 Examples

Example 1: Basic CRUD

from pyhybriddb import Database

db = Database("example_db", path="./data")
db.create()

# Create table
users = db.create_table("users", {"name": "string", "age": "integer"})

# Insert
user_id = users.insert({"name": "Alice", "age": 30})

# Select
all_users = users.select()

# Update
users.update(where={"name": "Alice"}, updates={"age": 31})

# Delete
users.delete(where={"name": "Alice"})

db.close()

Example 2: NoSQL Collections

from pyhybriddb import Database

with Database("blog_db") as db:
    posts = db.create_collection("posts")
    
    # Insert
    posts.insert_one({
        "title": "My First Post",
        "tags": ["intro"],
        "views": 0
    })
    
    # Find
    all_posts = posts.find()
    intro_posts = posts.find({"tags": "intro"})
    
    # Update
    posts.update_one(
        {"title": "My First Post"},
        {"$inc": {"views": 1}}
    )
    
    # Aggregate
    popular = posts.aggregate([
        {"$sort": {"views": -1}},
        {"$limit": 5}
    ])

See examples/basic_usage.py for more examples.

---

📁 Project Structure

D:\python_db\
├── pyhybriddb/              # Main package
│   ├── config.py            # Configuration
│   ├── core/                # Database core
│   │   ├── database.py
│   │   ├── table.py
│   │   └── collection.py
│   ├── storage/             # Storage engine
│   │   ├── engine.py
│   │   ├── file_manager.py
│   │   └── index.py
│   ├── query/               # Query layer
│   │   ├── parser.py
│   │   ├── sql_parser.py
│   │   ├── nosql_parser.py
│   │   └── joins.py         # ✨ JOIN operations
│   ├── api/                 # REST API
│   │   ├── server.py
│   │   ├── models.py
│   │   ├── auth.py
│   │   └── users.py         # ✨ User management
│   ├── utils/               # Utilities
│   │   ├── backup.py        # ✨ Backup & restore
│   │   ├── audit.py         # ✨ Audit logging
│   │   ├── encryption.py    # ✨ Encryption
│   │   ├── visualization.py # ✨ Data visualization
│   │   ├── serializer.py
│   │   └── logger.py
│   ├── migration/           # ✨ Migration tools
│   │   ├── postgresql.py    # PostgreSQL migration
│   │   └── mongodb.py       # MongoDB migration
│   └── cli.py               # CLI
├── admin/                   # Web admin panel
│   ├── index.html
│   ├── app.js
│   └── auth.js
├── examples/                # Examples
├── tests/                   # Tests
├── DEMO.py                  # Demo script
├── config.env               # Config template
├── requirements.txt
├── setup.py
└── README.md                # This file

---

🧪 Testing

Run Tests

# Run all tests
python -m unittest discover tests

# Run specific test
python -m unittest tests.test_database.TestDatabase

Example Test

import unittest
from pyhybriddb import Database

class TestDatabase(unittest.TestCase):
    def test_create_database(self):
        db = Database("test_db", path="./test_data")
        db.create()
        self.assertTrue(db.db_file.exists())
        db.close()

---

🔒 Security

Best Practices

1. Change Default Credentials

   ADMIN_PASSWORD=YourSecurePassword123!

2. Use Strong SECRET_KEY

   import secrets
   print(secrets.token_urlsafe(32))

3. Enable HTTPS in Production

4. Restrict CORS Origins

   CORS_ORIGINS=https://yourdomain.com

5. Set Appropriate Log Level

   LOG_LEVEL=WARNING

Security Features

• ✅ JWT token authentication
• ✅ Password hashing with bcrypt
• ✅ Token expiration (30 minutes)
• ✅ CORS protection
• ✅ Input validation
• ✅ Environment-based secrets
• ✅ Audit logging for compliance
• ✅ Encrypted storage option

---

📊 Implementation Status

✅ 100% Feature Complete

All features from the original PRD have been successfully implemented!

Feature	Status	File Location
Core Features
Hybrid Data Model	✅ Complete	core/database.py, core/table.py, core/collection.py
Custom Storage Engine	✅ Complete	storage/engine.py, storage/file_manager.py
B-Tree Indexing	✅ Complete	storage/index.py
SQL Query Support	✅ Complete	query/sql_parser.py
NoSQL Query Support	✅ Complete	query/nosql_parser.py
ACID Transactions	✅ Complete	core/database.py
REST API	✅ Complete	api/server.py
Web Admin Panel	✅ Complete	admin/index.html
JWT Authentication	✅ Complete	api/auth.py
CLI Tools	✅ Complete	cli.py
Environment Config	✅ Complete	config.py
Advanced Features
Backup & Restore	✅ Complete	utils/backup.py
Audit Logging	✅ Complete	utils/audit.py
User Management	✅ Complete	api/users.py
JOIN Operations	✅ Complete	query/joins.py
Data Visualization	✅ Complete	utils/visualization.py
PostgreSQL Migration	✅ Complete	migration/postgresql.py
MongoDB Migration	✅ Complete	migration/mongodb.py
Encrypted Storage	✅ Complete	utils/encryption.py
Import/Export	✅ Complete	admin/app.js

Total: 20/20 Features (100%)

📈 Project Metrics

• Python Modules: 40+
• Lines of Code: ~10,000+
• API Endpoints: 25+
• Test Coverage: Core functionality
• Documentation: Comprehensive

🎯 Production Ready

✅ All core features implemented
✅ All advanced features implemented
✅ Security features complete
✅ Operational tools ready
✅ Migration tools available
✅ Comprehensive documentation
✅ Working examples provided
✅ Server tested and running

---

🚀 Quick Start Guide

Installation

# Clone repository
git clone https://github.com/Adrient-tech/PyHybridDB.git
cd PyHybridDB

# Create virtual environment
python -m venv venv
.\venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install package
pip install -e .

Start Server

python -m pyhybriddb.cli serve

Server will start at: http://localhost:8000

Access Points

• API Docs: http://localhost:8000/docs
• Admin Panel: Open admin/index.html in browser
• Default Login: admin / admin123

Run Demo

python DEMO.py

---

📞 Support & Contributing

Support

• GitHub Issues: Report bugs and request features
• Documentation: See this README and inline docs
• Examples: Check examples/ directory

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

Development Setup

# Install dev dependencies
pip install -r requirements.txt

# Run tests
python -m unittest discover tests

# Start with auto-reload
python -m pyhybriddb.cli serve --reload

---

🎓 Learning Resources

1. Quick Start: This README
2. API Reference: http://localhost:8000/docs (when server running)
3. Examples: examples/basic_usage.py
4. Demo Script: python DEMO.py
5. Original PRD: project.md

---

📄 License

MIT License

Copyright (c) 2025 Adrient.com - Developed by Infant Nirmal

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

---

📞 Support

• Issues: Report bugs on GitHub Issues
• Documentation: See this README and project.md
• Examples: Check examples/ directory
• Demo: Run python DEMO.py

---

🎯 Roadmap

Phase 1: Core Features ✅ COMPLETE

• ✅ Core storage engine
• ✅ Hybrid data model
• ✅ SQL & NoSQL query support
• ✅ REST API
• ✅ Admin panel
• ✅ JWT Authentication
• ✅ CLI Tools

Phase 2: Advanced Features ✅ COMPLETE

• ✅ Backup & Restore
• ✅ Audit Logging
• ✅ User Management
• ✅ JOIN Operations
• ✅ Data Visualization
• ✅ PostgreSQL Migration
• ✅ MongoDB Migration
• ✅ Encrypted Storage
• ✅ Import/Export

Phase 3: Future Enhancements (Optional)

• Multi-Factor Authentication (2FA)
• Full-Text Search
• Compound Indexes
• Advanced Query Optimization
• Replication & High Availability
• Sharding & Horizontal Scaling
• GraphQL API
• Real-time Subscriptions
• Cloud Storage Backends (S3, Azure, GCP)
• Plugin System & Extensions

---

🙏 Acknowledgments

Inspired by:

• PostgreSQL - Relational model
• MongoDB - Document model
• SQLite - Embedded database
• phpMyAdmin - Admin interface

---

📊 Project Stats

• Lines of Code: ~10,000+
• Python Modules: 40+
• Total Files: 45+
• API Endpoints: 25+
• Features: 20/20 (100% Complete)
• Version: 1.0.0 (Production Ready)
• Python: 3.10+
• Platform: Cross-platform
• License: MIT

---

Built with ❤️ by Infant Nirmal at Adrient.com

GitHub: https://github.com/Adrient-tech/PyHybridDB.git

Last Updated: October 25, 2025