Datum

A semantic context layer that keeps data agents honest.

Datum answers natural-language questions over a database by grounding an LLM on a governed semantic layer. It writes SQL that is validated against the schema, cites the tables and columns it used, and refuses instead of guessing when a question needs data that doesn't exist, is personal (PII), or isn't certified for analytics. It ships with an MCP server, a reliability eval harness, and unit tests for the guardrail.

A confident wrong answer is worse than no answer.

---

Results

Datum vs. a naive single-prompt text-to-SQL baseline, gpt-4o-mini, averaged over 3 runs. CWR = Confidently-Wrong Rate (wrong answers plus answers that should have been refused — PII, uncertified, out-of-scope) — the number a trust layer exists to minimize.

Benchmark	Engine	Execution Acc. ↑	Refusal Recall ↑	CWR ↓	Answer faithfulness ↑
Synthetic (30 Q)	Datum	96.3%	100%	0.0%	100%
	Baseline	88.9%	27.8%	35.6%	81.5%
Chinook (20 Q, real DB)	Datum	92.9%	100%	0.0%	—
	Baseline	100%	33.3%	20.0%	—

The critic + semantic layer drives confidently-wrong answers to zero and catches 100% of PII / uncertified / out-of-scope queries — without lowering execution accuracy. The baseline, by contrast, leaks PII (SELECT email FROM employees), queries uncertified tables, and even aliases AVG(csat_score) AS average_nps to answer a question about a metric that doesn't exist.

Reproduce:

python evals/run_evals.py --runs 3 --judge        # synthetic
python evals/run_external.py evals/chinook_eval.yaml --runs 3   # real Chinook DB

Why

Point an LLM at a database and it will happily invent an nps_score column, join on the wrong key, leak an email address, or return a confident number that's just wrong. Datum puts two things between the model and the data:

1. A governed semantic layer — definitions, ownership, trust levels, metrics and PII flags — so the agent reasons over meaning, not raw table names.
2. A deterministic critic — a real SQL parser (sqlglot, not the LLM) that blocks hallucinated columns, PII, uncertified tables and any non-read-only statement before a query ever runs.

If neither can make the question safe to answer, Datum abstains — with a typed reason.

Architecture

flowchart TD
    Q[Natural-language question] --> R[Retriever<br/>relevant semantic assets]
    R --> P[Planner<br/>answerable? which assets?]
    P -->|no| X[Abstain<br/>pii · missing · untrusted · out-of-scope]
    P -->|yes| A[SQL Author<br/>grounded SELECT]
    A --> C{Critic<br/>sqlglot guardrail}
    C -->|violations| A
    C -->|clean| E[(SQLite<br/>read-only)]
    E --> Ans[Answerer<br/>NL answer + citations]
    SM[[Semantic layer<br/>trust · PII · metrics]] -.governs.-> P
    SM -.governs.-> A
    SM -.governs.-> C

Loading

Multi-agent pipeline: Planner → SQL Author → Critic → Answerer, with a repair loop (the Critic's feedback goes back to the Author). Citations are extracted from the validated SQL's AST, so they reflect what the query actually touched.

Features

• Grounded NL→SQL over SQLite, with citations and typed abstention.
• Governed semantic layer (context/semantic_model.yaml): trust levels, PII flags, ownership, business metrics with reference SQL.
• Deterministic critic (sqlglot): rejects unknown/hallucinated columns, PII columns (even hidden behind a CTE or renamed alias), uncertified tables, and non-read-only SQL.
• Schema-linking: FK-neighbour expansion pulls join-partner tables into context.
• Context retrieval over semantic assets — OpenAI embeddings, with a lexical fallback so the pipeline runs with no key.
• MCP server — expose the engine to Claude Desktop / Cursor.
• Auto-introspection adapter — build a semantic layer from any SQLite schema.
• Reliability eval harness with a naive baseline, an LLM-as-judge faithfulness check, and multi-run averaging.
• Unit tests for the guardrail (pytest, no API needed).

Quickstart

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env          # add your OPENAI_API_KEY

python -m datum seed          # build the demo database
python -m datum ask "What is our monthly recurring revenue?"
python -m datum ask "List the email addresses of our account owners."   # abstains (PII)
python -m datum eval          # reliability scorecard vs baseline
pytest -q                     # guardrail unit tests (no API)

Example

$ python -m datum ask "What is our monthly recurring revenue?"
╭─ Answer ─────────────────────────────────────────────────────────╮
│ Our monthly recurring revenue is $1,241,366.00.                   │
│ Source: plans.monthly_price, subscriptions.seats                  │
╰───────────────────────────────────────────────────────────────────╯
SELECT SUM(p.monthly_price * s.seats) AS mrr
FROM subscriptions s JOIN plans p ON p.plan_id = s.plan_id
WHERE s.status = 'active'

$ python -m datum ask "List the email addresses of our account owners."
╭─ Abstained · pii ────────────────────────────────────────────────╮
│ I can't answer that because it requires personal data (PII) that  │
│ policy does not expose.                                            │
╰───────────────────────────────────────────────────────────────────╯

Benchmarks

Metrics (standard text-to-SQL + governance):

Metric	Meaning
EX Execution Accuracy	answerable questions whose result matched gold
RP Refusal Precision	of everything it refused, how much should be refused
RR Refusal Recall	of the should-refuse questions, how many it caught
CWR Confidently-Wrong	wrong answers + policy leaks, over all questions (lower better)
Answer faithfulness	LLM-as-judge: are the answer's numbers grounded in the rows?

Synthetic (evals/eval_set.yaml, 30 questions, deterministic DB) is the primary benchmark. Chinook (evals/chinook_eval.yaml, 20 questions) runs the same harness against the real third-party Chinook DB via schema auto-introspection.

Adding a Spider dev database (or any SQLite): the harness is dataset-agnostic. Drop the .sqlite file in benchmarks/, copy chinook_eval.yaml, point database: at it, list its pii_columns, and paste the Spider dev questions + gold SQL as items — then python evals/run_external.py evals/your_spec.yaml. (Public benchmarks ship no trust/PII labels; hand-annotating them in the YAML is exactly what the semantic layer is for.)

# fetch the Chinook demo DB used above (~1 MB)
curl -sL -o benchmarks/chinook.sqlite \
  https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite

Tests

pytest -q      # 37 tests, ~0.4s, no API

The guardrail is tested directly with adversarial SQL — stacked-statement injection, write keywords hidden in comments, PII behind a renamed CTE alias, hallucinated columns, unqualified columns shared across two tables — so the "trust layer" claim is proven, not just asserted.

Use it from Claude Desktop / Cursor (MCP)

python -m datum serve starts an MCP server exposing list_certified_tables, describe_table, and ask:

{
  "mcpServers": {
    "datum": {
      "command": "/ABSOLUTE/PATH/datum/.venv/bin/python",
      "args": ["/ABSOLUTE/PATH/datum/mcp_server.py"]
    }
  }
}

Project layout

datum/
├── datum/
│   ├── db.py            # sqlite access + read-only guardrail + result compare
│   ├── seed.py          # deterministic synthetic SaaS database
│   ├── semantic.py      # loads/queries the governed semantic layer
│   ├── introspect.py    # auto-build a semantic layer from any SQLite schema
│   ├── retrieval.py     # embeddings (or lexical fallback) over assets
│   ├── agents.py        # Planner · Author · Critic · Answerer + Engine
│   ├── baseline.py      # naive single-prompt baseline (the comparison point)
│   └── cli.py           # `python -m datum ...`
├── context/semantic_model.yaml   # the governed context layer
├── evals/
│   ├── eval_set.yaml            # synthetic benchmark (incl. must-refuse questions)
│   ├── chinook_eval.yaml        # real-DB benchmark spec
│   ├── run_evals.py             # harness + baseline + faithfulness judge
│   └── run_external.py          # run the harness on any external SQLite DB
├── tests/                       # pytest guardrail suite (no API)
└── mcp_server.py                # MCP server (FastMCP)

Design notes, limitations & next steps

• Benchmarks are small and partly self-authored (30 synthetic + 20 Chinook). They demonstrate the mechanism and the delta vs. baseline; they are not a large-scale study.
• Single dialect. Only SQLite is implemented and tested. The design (semantic layer + AST critic) is dialect-agnostic, but porting to a warehouse dialect is future work, not a claim being made here.
• The critic proves validity and governance, not semantic correctness. It guarantees SQL is read-only and touches only real, certified, non-PII columns; whether the query answers the user's intent is what the eval's execution accuracy measures.
• Columns behind derived-table/CTE aliases are skipped by the critic to avoid false positives (the read-only + certified-table checks still apply).
• Next: Spider/BIRD subsets through run_external.py; a warehouse dialect; caching embeddings; expanding the eval sets.

License

MIT