Contributing

Thank you for your interest in contributing to SQL Query Engine! This page covers development setup, project conventions, and how to submit changes.

Development Setup

Prerequisites

• Python 3.10+
• Docker and Docker Compose
• A running PostgreSQL instance (for integration testing)
• A running Redis instance
• Access to an OpenAI-compatible LLM server (Ollama is the easiest for local dev)

Clone and Install

git clone https://github.com/codeadeel/sqlqueryengine.git
cd sqlqueryengine

# Create virtual environment
python -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Start Supporting Services

The quickest way to get PostgreSQL and Redis running locally:

# Start just Redis from the compose file
docker compose up redis -d

# Or use standalone Redis
docker run -d --name redis -p 6379:6379 redis:latest

# For Ollama (local LLM)
ollama serve
ollama pull qwen2.5-coder:7b

Configure Environment

export LLM_BASE_URL="http://localhost:11434/v1"
export LLM_MODEL="qwen2.5-coder:7b"
export LLM_API_KEY="ollama"
export LLM_TEMPERATURE="0.7"
export POSTGRES_HOST="localhost"
export POSTGRES_PORT="5432"
export POSTGRES_DB="your_dev_db"
export POSTGRES_USER="postgres"
export POSTGRES_PASSWORD="your_password"
export REDIS_HOST="localhost"
export REDIS_PORT="6379"
export REDIS_PASSWORD=""
export REDIS_DB="0"

Run the Server

python run.py

Project Structure

sqlqueryengine/
├── sqlQueryEngine/                    # Main Python package
│   ├── __init__.py                    # Public exports
│   ├── main.py                        # FastAPI app + native routes
│   ├── engine.py                      # Pipeline orchestrator
│   ├── queryGenerator.py              # Stage 1: NL → SQL
│   ├── queryEvaluator.py              # Stage 2: execute + repair
│   ├── openaiCompat.py                # OpenAI-compatible API
│   ├── dbHandler.py                   # PostgreSQL handler (read-only)
│   ├── sessionManager.py              # Redis session manager
│   ├── connConfig.py                  # Configuration loading
│   ├── promptTemplates.py             # LLM prompt definitions (4 templates)
│   └── sqlGuidelines.py               # PostgreSQL best-practices (2 corpora)
├── evaluation/                        # Evaluation framework
│   ├── shared/                        # Shared utilities
│   │   ├── resultComparator.py        # Order-independent result comparison
│   │   └── resourceMetrics.py         # Wall time, memory, throughput tracking
│   ├── synthetic/                     # Synthetic evaluation (controlled environment)
│   │   ├── entrypoint.py              # Pipeline orchestrator
│   │   ├── evalRunner.py              # 3-config ablation runner
│   │   ├── evalConfig.py              # Environment-driven config
│   │   ├── seedData.py                # Database seeding (Faker)
│   │   ├── schemaDefinitions.py       # DDL for 3 evaluation databases
│   │   ├── questionRunner.py          # Gold query executor
│   │   ├── scoreReport.py             # Summary table generation
│   │   ├── questions/                 # 120 gold questions (40 per DB)
│   │   └── results/runs/              # Archived results per model
│   └── bird/                          # BIRD benchmark evaluation
│       ├── birdEntrypoint.py          # Pipeline orchestrator
│       ├── birdDataLoader.py          # Dataset loading + SQL dialect conversion
│       ├── sqliteToPostgres.py        # SQLite → PostgreSQL migration
│       ├── birdEvalRunner.py          # 3-config ablation runner for BIRD
│       ├── birdScoreReport.py         # BIRD-specific scoring + baselines
│       ├── birdConfig.py              # BIRD environment-driven config
│       ├── bird_data/                 # BIRD dataset (gitignored, download separately)
│       └── bird_results/runs/         # Archived results per model
├── Dockerfile                         # Multi-stage Docker build (3 stages)
├── docker-compose.yml                 # Production stack (engine + Redis + OpenWebUI)
├── docker-compose-synthetic-evaluation.yml  # Synthetic evaluation stack
├── docker-compose-bird-evaluation.yml       # BIRD benchmark stack
├── requirements.txt                   # Python dependencies
├── run.py                             # Uvicorn entry point
├── curlCommands.sh                    # API usage examples
└── .gitignore

See the Module Reference page for detailed documentation of each module.

Code Conventions

Python Style

• Follow PEP 8 with the exception of camelCase for variable and function names (project convention)
• Use type hints for function signatures
• Use Pydantic models for request/response validation
• Use LangChain patterns for LLM interactions

Naming

• Files: camelCase (queryGenerator.py, dbHandler.py)
• Classes: PascalCase (SQLQueryEngine, QueryGenerator)
• Methods: camelCase (getUserChatContext, queryExecutor)
• Constants: UPPER_SNAKE_CASE (SPLIT_IDENTIFIER, DEFAULT_RETRY_COUNT)

Architecture Principles

• Separation of concerns: Each module has a single responsibility
• Dependency injection: Connection params passed down from config, not imported globally
• Multi-strategy response parsing: LLM responses are parsed via a 5-strategy cascade (JSON → embedded JSON → code blocks → regex → raw text) rather than relying on structured output or function calling — this ensures compatibility with any model
• Read-only safety: Database connections always enforced as read-only via conn.set_read_only(True)
• Graceful degradation: Evaluator has 3-tier fallback for schema context resolution
• Early-accept: Queries returning rows are accepted immediately without an LLM call, preventing regressions
• Best-result tracking: If retries exhaust, the best result seen across all attempts is returned

Making Changes

Adding a New Endpoint

1. Define the Pydantic request model in main.py
2. Add the route handler in main.py (native) or openaiCompat.py (OpenAI-compat)
3. If needed, add new methods to engine.py
4. Update curlCommands.sh with example calls
5. Update the wiki API Reference and Usage Guide

Modifying Prompt Templates

Prompt templates live in promptTemplates.py. When changing prompts:

1. Test with multiple LLM models to ensure broad compatibility
2. Verify response parsing still extracts SQL correctly (check _parseResponse() and _parseEvalResponse())
3. Test with various database schemas (simple and complex)
4. Check that the repair loop still functions

Adding a New LLM Feature

If adding features that require new LLM capabilities:

1. Define the Pydantic output schema in the relevant module
2. Add a corresponding parser method (follow the multi-strategy pattern in _parseResponse())
3. Add appropriate error handling for malformed LLM responses
4. Test with different model sizes (small models may struggle with complex schemas)

Submitting Changes

Pull Request Process

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature-name
3. Make your changes
4. Test the endpoints using curlCommands.sh as a reference
5. Commit with a clear message describing the change
6. Push and open a Pull Request

PR Guidelines

• Keep PRs focused — one feature or fix per PR
• Include curl examples demonstrating the change (if API-related)
• Update documentation (wiki pages, curlCommands.sh, README) as needed

Areas for Contribution

Here are some areas where contributions are welcome:

• Additional LLM providers: Direct integrations beyond the OpenAI-compatible interface
• Query result formatting: Better markdown/HTML rendering of results
• Schema change detection: Automatic invalidation of cached schema when the database schema changes
• Token counting: Implement actual token usage tracking for the OpenAI-compat endpoint
• Write mode: Optional write-capable mode for controlled INSERT/UPDATE operations
• Test coverage: Unit tests for individual modules (currently only integration testing via curl exists)
• Multi-database support: Ability to query multiple databases in a single session
• Query history: Persistent storage of past queries and results
• Rate limiting: Request throttling for the API endpoints