Skip to content
BranchesTags

Repository files navigation

AxisSQL

ICLR 2027 · Mapping the Axes of Inference-Time Scaling for Text-to-SQL

ICLR 2027 Task Benchmarks

Architecture Python License

Highlights · Results · Scaling Axes · Installation · Quick Start · Evaluation · Citation

What is AxisSQL?

AxisSQL is a controlled measurement study that maps where inference-time compute converts into accuracy in a multi-stage Text-to-SQL pipeline. Rather than optimizing a single design choice, AxisSQL systematically varies five scaling knobs and measures their impact on execution accuracy.

Built on the DeepEye-SQL pipeline, AxisSQL identifies the bottleneck as selection, not sampling, and demonstrates that an execution-grounded aggregator significantly outperforms traditional voting-based selection. Results on BIRD mini-dev with four open coder models provide actionable insights for inference scaling.

Key Findings

  • Parallel width saturates early: sampling more candidates hits diminishing returns
  • Selection is the bottleneck: aggregation beats majority voting and tournaments
  • Execution-grounded synthesis: run-and-verify outperforms pure voting by 4%
  • Metadata scales efficiently: profiling data is cheaper than candidate sampling
  • Inference and parameters compose: test-time compute buys ~one model-size tier

Highlights

📊 Systematic scaling study of five inference-time knobs across a fixed pipeline.
⚙️ Two primary axes: parallel width (candidates sampled) and sequential depth (revise/verify rounds).
🧠 Execution-grounded aggregation: run candidates, synthesize from correct fragments, verify results.
📈 Practical insights: selection strategy matters more than pool size on moderate-difficulty queries.
🛠️ Four open coder models: Qwen2.5-Coder-32B, Qwen3-Coder-30B, Gemma-3-27B, Gemma-4-31B.
📦 Scaling-curve harness, per-run outputs, and reproducible configuration suite included.

Results

AxisSQL demonstrates significant improvements in execution accuracy on BIRD mini-dev through inference-time scaling. Execution-grounded aggregation closes the gap between oracle upper bounds and realized accuracy.

Configuration Model EX Accuracy Improvement Notes
Baseline (single-shot) Qwen3-Coder-30B-A3B 65.8% — DeepEye-SQL pipeline
Pairwise tournament Qwen3-Coder-30B-A3B 69.8% +4.0pp Pass@16 with voting
Agentic aggregation Qwen3-Coder-30B-A3B 73.8% +4.0pp Execution-grounded synthesis
Moderate-only (agg) Qwen3-Coder-30B-A3B 73.2% +6.4pp Gains concentrated here
Gemma-4-31B (single) Gemma-4-31B 61.2% — Baseline 2.6× smaller
Gemma-4-9B + scaling Gemma-4-9B ~61.2% Matches With inference scaling

Scaling Axes

AxisSQL studies five scaling knobs that affect inference-time accuracy:

Primary Axes (the core 2D plane):

  • Parallel width ($N$): number of independent SQL candidates sampled per stage
  • Sequential depth ($R$): number of revise/verify/adjudicate rounds applied to candidates

Auxiliary Knobs (reshape the axes):

  • Stage-wise model scaling: assign stronger models to specific pipeline stages
  • Domain-specific metadata: profiling and LLM-summarized column metadata
  • Fine-tuning scaling: task-specific model adaptations

The study reveals that the operating path—from sampling more candidates to realizing accuracy—passes through a competent agentic aggregator that executes, compares, and synthesizes SQL queries.

Built on DeepEye-SQL

AxisSQL is a scaling study of the DeepEye-SQL five-stage Text-to-SQL pipeline:

Natural Language Question
        |
        v
1. Value Retrieval (grounding)
2. Schema Linking (relevant context)
3. SQL Generation (diverse candidates)
4. SQL Revision (checker-style repair)
        |
        v
5. SQL Selection ← AxisSQL focuses here
   Parallel width: sample N candidates
   Sequential depth: revise R rounds
   Aggregator: execute, compare, synthesize

Why inference-time scaling matters

  • Single-shot generation misses valid alternatives.
  • Parallel sampling raises the oracle ceiling but selection is still the bottleneck.
  • Execution feedback is underexploited: a run-and-verify aggregator can synthesize new queries from correct fragments.
  • Different scaling knobs have asymmetric cost-benefit profiles (metadata is cheap, sampling is expensive).

Repository Tour

AxisSQL (built on DeepEye-SQL)
├── app/
│   ├── config/          # lazy config loading and typed settings
│   ├── dataset/         # BIRD datasets + structured snapshots
│   ├── db_utils/        # SQL execution, schema loading
│   ├── llm/             # OpenAI-compatible LLM wrapper
│   ├── pipeline/        # five-stage Text-to-SQL pipeline + agg_agent
│   │   └── sql_selection/
│   │       └── agg_agent.py  # execution-grounded aggregation ← AxisSQL contribution
│   ├── services/        # schema service, execution service, artifact store
│   ├── prompt/          # prompt templates
│   └── vector_db/       # vector index creation for value retrieval
├── config/              # model configs (Gemma3/4, Qwen variants) + example configs
├── runner/              # reproducible entry scripts
├── results/             # released predictions and few-shot seeds
├── script/              # helper shell scripts + scaling_curve.py (AxisSQL evaluation)
├── paper/               # ICLR 2027 submission (main.tex, references, style)
└── workspace/           # generated snapshots and intermediate outputs

Key entry points

AxisSQL-specific:

DeepEye-SQL pipeline (foundation):

Installation

Requirements

  • Python >= 3.12
  • Linux/macOS environment recommended
  • OpenAI-compatible LLM endpoint for each stage
  • Embedding endpoint or local embedding model for value retrieval

1. Clone

git clone https://github.com/ShaikNagurShareef/AxisSQL.git
cd AxisSQL

2. Install dependencies

We recommend uv.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync

3. Optional cloud dependencies

Spider2 cloud evaluation may require valid:

  • BigQuery credentials
  • Snowflake credentials

The corresponding paths are configured in config/config-spider2-example.toml.

Dataset Setup

AxisSQL evaluates on BIRD mini-dev (a manageable subset for systematic scaling studies).

BIRD (required)

Use the provided helper script:

bash script/download_dataset.sh

This downloads the BIRD dev split. AxisSQL primarily uses BIRD mini-dev for controlled scaling experiments.

Configuration notes

  • All AxisSQL configs use root_path = "data/bird/data_minidev/MINIDEV" by default
  • Ensure the BIRD directory structure matches the config paths
  • No cloud credentials required for BIRD experiments

Configuration

AxisSQL includes model-specific BIRD configurations for four open coder models:

Legacy example configs (if running on Spider/Spider2):

Important config blocks

Dataset

[dataset]
type = "bird"                # spider | bird | spider2
split = "dev"
root_path = "data/bird"
save_path = "workspace/dataset/bird/dev.snapshot"

Embedding / vector DB

[vector_database]
api_type = "openai"          # or local
embedding_model_name_or_path = "your-embedding-model"
store_root_path = "workspace/vector_database/bird/dev"
embedding_device = "auto"       # auto | cpu | cuda | cuda:0
db_parallel = 2
column_parallel = 8

Stage LLMs

[sql_generation.llm]
model = "your-model-name"
base_url = "https://your-openai-compatible-endpoint/v1"
api_key = "your-api-key"
max_tokens = 4096
temperature = 0.7
api_type = "openai"
max_model_len = 128000

Selection Strategy (AxisSQL-specific)

The [sql_selection] block supports two modes:

[sql_selection]
# Default: pairwise tournament
strategy = "pairwise"  # classical voting

# AxisSQL mode: execution-grounded aggregation
strategy = "agg_agent"
agg_agent_mode = "both"           # "pick" | "synthesize" | "both"
agg_agent_sampling_budget = 3     # self-consistency samples
agg_agent_max_refine_rounds = 1   # verify-once (0=off, 1=verify, N=multi-round)
agg_agent_decomposition_enabled = false  # question decomposition (future)

Notes

  • Each stage can use a different model (configure per [stage.llm] block).
  • All stage outputs are stored as structured .snapshot manifests.
  • Only structured .snapshot manifests are supported for reproducibility.

Quick Start

AxisSQL Scaling Experiment (recommended)

# Set up the environment
export CONFIG_PATH=config/config-bird-vllm-qwen3coder.toml

# Run the full scaling curve harness
uv run script/scaling_curve.py \
  --config $CONFIG_PATH \
  --model_scale_budget 32000 \
  --selection_strategies pairwise agg_agent \
  --aggregator_modes pick synthesize both

Full Pipeline (single run)

export CONFIG_PATH=config/config-bird-vllm-qwen3coder.toml
bash script/run_pipeline.sh

Stage-by-Stage

export CONFIG_PATH=config/config-bird-vllm-qwen3coder.toml

uv run runner/preprocess_dataset.py
uv run runner/create_vector_db_parallel.py
uv run runner/run_value_retrieval.py
uv run runner/run_schema_linking.py
uv run runner/run_sql_generation.py
uv run runner/run_sql_revision.py
uv run runner/run_sql_selection.py  # includes agg_agent if configured

Outputs

  • Dataset snapshot: workspace/dataset/bird/dev.snapshot
  • Stage snapshots: workspace/{value_retrieval,schema_linking,sql_generation,sql_revision,sql_selection}/bird/dev.snapshot
  • Scaling curves (from scaling_curve.py): JSON logs per model and strategy

Reproducibility

AxisSQL uses structured snapshots for checkpoint-and-resume, making long-running scaling studies repeatable.

Checkpoint workflow

  1. Preprocess: uv run runner/preprocess_dataset.py → creates workspace/dataset/bird/dev.snapshot
  2. Vector index: uv run runner/create_vector_db_parallel.py → creates workspace/vector_database/bird/dev/
  3. Pipeline: Run each stage in order; each consumes the previous snapshot and writes a new one
  4. Selection variants: Re-run runner/run_sql_selection.py with different agg_agent_* settings without re-running prior stages

Resume and compare

# After stage N, you can compare selection strategies on the same candidates:
# - Keep workspace/sql_revision/bird/dev.snapshot
# - Modify [sql_selection] config
# - Re-run: uv run runner/run_sql_selection.py

This enables efficient cost-benefit analysis of aggregation strategies without re-running generation and revision.

Evaluation

AxisSQL evaluates on BIRD mini-dev using execution accuracy (EX) as the primary metric.

Single evaluation run

uv run runner/evaluation.py \
  --snapshot_path workspace/sql_selection/bird/dev.snapshot \
  --dataset_type bird

Scaling curve analysis

The scaling_curve.py harness generates comparative results across:

uv run script/scaling_curve.py \
  --config config/config-bird-vllm-qwen3coder.toml \
  --model_scale_budget 32000 \
  --selection_strategies pairwise agg_agent \
  --aggregator_modes pick synthesize both \
  --width_values 1 2 4 8 16 32 \
  --depth_values 0 1 2 3

Metrics

  • EX (Execution Accuracy): exact match between predicted and gold SQL execution results
  • Pass@N: oracle upper bound (does at least one of N candidates match gold?)
  • Synthesis gain: delta between tournament and execution-grounded synthesis
  • Moderate-difficulty focus: AxisSQL highlights gains concentrated on mid-difficulty queries

Artifacts

Code and configurations

Paper and results

  • ICLR 2027 submission: paper/main.tex
  • Per-run outputs: Available in workspace/ after experiment completion (structured snapshots)

FAQ

What is the difference between AxisSQL and DeepEye-SQL?

DeepEye-SQL is the five-stage Text-to-SQL pipeline (grounding, linking, generation, revision, selection). AxisSQL is a scaling study of DeepEye-SQL that measures where inference-time compute converts to accuracy. AxisSQL's core contribution is the agentic aggregator in the selection stage, which executes, synthesizes, and verifies SQL candidates.

Can I compare aggregation strategies on the same candidates?

Yes! The snapshot-based workflow lets you:

  1. Run stages 1–4 once (value retrieval through revision)
  2. Keep the revision snapshot
  3. Re-run selection with different agg_agent_* configs without re-running generation

Why focus on BIRD mini-dev?

BIRD mini-dev is a curated 11-database subset of BIRD dev. It's large enough to reveal scaling patterns yet small enough for systematic grid search. AxisSQL studies a 5D parameter space (width, depth, model scale, metadata, fine-tuning) on a fixed, manageable dataset.

Can I use local models?

Yes. Any OpenAI-compatible endpoint works. Update the [*.llm] blocks in config files with your local server's base_url and api_key.

Citation

If you find AxisSQL useful in your research, please cite:

@article{shareef2027axissql,
  author  = {Shaik Nagur Shareef},
  title   = {{AxisSQL:} Mapping the Axes of Inference-Time Scaling for Text-to-SQL},
  journal = {Proc. Int. Conf. Learn. Represent.},
  year    = {2027},
  note    = {ICLR 2027 Submission}
}

Built on:

  • DeepEye-SQL — the underlying five-stage pipeline
  • BIRD — benchmark for Large-Scale Database Grounded Text-to-SQL

License

This project is released under the MIT License. See LICENSE.

Acknowledgement

AxisSQL builds on DeepEye-SQL (the five-stage pipeline) and the BIRD benchmark. We thank:

  • The BIRD team for curating a challenging and realistic text-to-sql evaluation set
  • The DeepEye-SQL authors for the software-engineering-inspired pipeline architecture
  • Maintainers of OpenAI-compatible serving frameworks (vLLM, etc.)
  • The broader Text-to-SQL research community for benchmarks and baselines

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages