qfeval

Scores a QueryForge config against a golden file of natural-language sentences and the queries they should compile to.

It answers the question the library's unit tests cannot: how well does the model handle real sentences? The deterministic half of the pipeline (AST → validate → generate) is already covered by the library's own suite, so what is left to measure is comprehension — and comprehension is a statistical property of a remote, non-deterministic service. That needs a corpus and a pass rate, not an assertion.

qfeval -in golden.csv -config ../queryforge/examples/orders.config.json

It scores QueryForge, which compiles natural language into SQL or MongoDB through a validated AST.

---

Results from a real run

Everything below is a live run against Google's Gemini API — not a simulation and not a projection. The raw per-case output is committed at results/2026-08-04-gemini-3.1-flash-lite.csv, one row per case with the AST the model returned, so every number here can be recomputed from the file.

Date	2026-08-04
Model	gemini-3.1-flash-lite (Google Gemini, OpenAI-compatible endpoint)
Temperature	0
Corpus	golden.example.csv — 25 cases, 21 SQL + 4 MongoDB
Config	orders.config.json — 7 fields, enum/boolean/date/number/array/string
Result	24 / 25 passed — 96%

Where the 25 cases landed

EXACT        ████░░░░░░░░░░░░░░░░░░░░░   4    byte-identical to the expectation
EQUIVALENT   █████████████████░░░░░░░░  17    different text, same meaning
REFUSED      ███░░░░░░░░░░░░░░░░░░░░░░   3    correctly declined an impossible request
FAIL         █░░░░░░░░░░░░░░░░░░░░░░░░   1    produced a query, but the wrong one
ERROR        ░░░░░░░░░░░░░░░░░░░░░░░░░   0    no query at all (transport / parse)
             └─────── 24 pass ────────┘

pie showData
    title Grade distribution — 25 cases, gemini-3.1-flash-lite
    "EQUIVALENT (pass)" : 17
    "EXACT (pass)" : 4
    "REFUSED (pass)" : 3
    "FAIL" : 1

Loading

EQUIVALENT being the most common grade is expected, not a weakness — it means the model wrote a semantically identical query in different words, which is what a language model does. See what "equivalent" covers for exactly which differences are forgiven and which are not.

Pass rate by category

Cases carry ;-separated tags, so one case can appear in two rows below.

enum            ████████████████████  5/5    status = 'CANCELLED', IN (...)
conjunction     ████████████████████  5/5    A and B and C
simple          ████████████████████  3/3    one field, one operator
relative-date   ████████████████████  3/3    "last 30 days" → a pinned timestamp
sort            ████████████████████  3/3    ORDER BY + direction
refusal         ████████████████████  3/3    request the config cannot express
numeric         ████████████████████  2/2    >, <, BETWEEN
synonym         ████████████████████  2/2    "state" → the status field
boundary        ████████████████████  2/2    inclusive vs exclusive edges
array           ████████████████████  2/2    tags contains / contains-any
limit           ████████████████████  2/2    "top 10" and the default limit
scope           ████████████████████  1/1    caller-imposed tenant filter
disjunction     ███████████████░░░░░  3/4    A or B
negation        ██████████░░░░░░░░░░  1/2    not / neither

Both dented rows are the same single failure — it is tagged negation;disjunction.

The one failure, and why it happened

Case	S10
Sentence	"orders that are neither cancelled nor refunded"
Expected	... WHERE status NOT IN ('CANCELLED', 'REFUNDED') LIMIT 50
Model produced	... WHERE refunded = FALSE AND status <> 'CANCELLED' LIMIT 50

This is arguably a config-design finding rather than a model failure. The config declares two different things that the word "refunded" can point at:

{ "name": "status",   "type": "enum",    "values": ["PLACED","DELIVERED","CANCELLED","REFUNDED"] }
{ "name": "refunded", "type": "boolean", "synonyms": ["was refunded","is refunded"] }

So "neither cancelled nor refunded" has two faithful readings. The golden file asserts one (both words are order statuses); the model chose the other (the second word is the boolean flag literally named refunded). Its query is well-formed, validated, and a reasonable answer to an ambiguous sentence — it just is not the answer the corpus expects.

Three things are worth drawing out of it:

1. It is deterministic, not flaky. A separate -repeat 5 run produced the same reading 5 times out of 5. The model is not guessing; it consistently prefers the field whose name matches the word.
2. It is the failure mode this harness exists to find. A string-diff test would have reported "query mismatch" and left you to guess. The graded output plus the returned AST names the actual problem: an ambiguous vocabulary.
3. The fix is in the config, not the prompt. Renaming the boolean to refundIssued, or dropping the overlapping synonyms, removes the collision for every future sentence — which is the point of a config-driven design.

What else the run showed

• Latency — 713 ms fastest, 943 ms median, 1409 ms slowest, per model call.
• Zero repairs. The library retries the model with the validator's complaint when an AST comes back invalid. The repair budget was never touched: all 25 ASTs were valid on the first attempt.
• Zero transport retries. No rate-limit or network backoff was needed at the default pacing.
• All 3 refusal cases passed. Asking for a field the config never exposes produced a typed refusal, not a plausible-looking query against a lookalike column — the failure mode that is genuinely dangerous, since it returns the wrong data silently.
• All 3 relative-date cases passed. "Last 30 days" resolved against the pinned clock to 2026-07-05T00:00:00Z on every backend.

Reproduce it

export QF_API_KEY=<your-gemini-key>
qfeval -in golden.example.csv \
       -config ../queryforge/examples/orders.config.json \
       -model gemini-3.1-flash-lite \
       -out results.csv

25 model calls, well inside a free-tier key. Your numbers may move by a case or two — the model is a remote service, and measuring that variance is what -repeat is for.

A larger corpus ships too

testdata/ contains a much bigger evaluation set — 573 cases / 1073 golden rows against a 35-field config that exercises every capability flag QueryForge supports. It is built and validated (-dry-run clean; the expected queries are compiled by the library itself, never hand-typed), but the numbers above are from the 25-case corpus. A full pass on the large set has not been scored here — it is 1073 model calls, roughly 72 minutes at -rpm 15. Run it yourself with testdata/README.md; use -only <tag> for a cheap subset first.

---

Run it on your own data

Five steps, no service to deploy and no database to connect.

1. Install Go 1.26+ and clone

git clone https://github.com/awsaman-ai/qfeval.git
cd qfeval
go build -o qfeval .

2. Get a model key. Any OpenAI-compatible endpoint works. A free Google AI Studio key is the quickest start:

export QF_API_KEY=<your-key>

3. Point it at a QueryForge config. This is the file describing your entity — its fields, types, operators and permissions. Use one of the examples, or build one in the browser with the config builder.

4. Write a golden file — your sentences and the queries they should compile to. Copy golden.example.csv and edit it; the format is below.

5. Dry-run, then score.

# validates the golden file and every config it names — spends zero model calls
./qfeval -in mygolden.csv -config myconfig.json -dry-run

# score it for real
./qfeval -in mygolden.csv -config myconfig.json -out results.csv

Open results.csv in Excel or a spreadsheet: one row per case, with the grade, the expected and actual query, and the raw AST the model returned.

Then the questions worth asking

# how STABLE are the answers? (the sharpest signal about a model)
./qfeval -in mygolden.csv -config myconfig.json -repeat 5 -rpm 25

# which model should you pay for? Same corpus, two models.
./qfeval -in mygolden.csv -config myconfig.json -model gemini-3.5-flash      -out flash.csv
./qfeval -in mygolden.csv -config myconfig.json -model gemini-3.1-flash-lite -out lite.csv

# only the cases you care about right now
./qfeval -in mygolden.csv -config myconfig.json -only refusal,relative-date

On a free-tier key add -rpm 15, which spaces the calls out to stay under the quota. -h lists every flag.

Exit status is 0 when every case passed, 1 when any failed, 2 on a usage error — so it drops into CI or a shell loop without parsing the output.

The golden file

A CSV with a header row. id and nl are required; exactly one of expect_sql / expect_mongo / expect_error must be set per row.

column	meaning
id	unique identifier; the results CSV joins back on it
nl	the natural-language request handed to the model
backend	sql or mongo (default: -backend)
config	path to a config (default: -config) — lets one file span entities
tags	;-separated labels, giving per-category pass rates
expect_sql	the SQL it should compile to, values written inline
expect_args	optional JSON array, to assert the parameterized form instead
expect_mongo	the Mongo filter, or a full find() envelope, as JSON
expect_error	unsupported — assert the config cannot express the request
scope	JSON object of caller-imposed filters (tenant, subscription)
now	pinned reference date, so relative-date cases do not rot
notes	ignored — a place for your reasoning

Unknown columns are an error, not a warning. A silently ignored expect_sqll would mean the row asserts nothing while still reporting a pass, which is the worst failure mode a test harness can have. The loader reports every problem in the file at once rather than one per run.

The smallest file that works

Three columns is a valid golden file. Everything else has a default or is optional:

id,nl,expect_sql
S01,cancelled orders,SELECT <default> FROM orders WHERE status = 'CANCELLED' LIMIT 50
S02,orders over 500,SELECT <default> FROM orders WHERE amount > 500 LIMIT 50

qfeval -in mygolden.csv -config myconfig.json

Sample rows, one per shape

Real rows from golden.example.csv. Note the quoting: a value containing a comma must be wrapped in ", and a " inside such a value is doubled — standard CSV, and exactly what Excel writes when you save.

id,nl,backend,tags,expect_sql,expect_mongo,expect_error,scope,now,notes
S07,orders that are cancelled or refunded,sql,disjunction;enum,"SELECT <default> FROM orders WHERE status IN ('CANCELLED','REFUNDED') LIMIT 50",,,,2026-08-04T00:00:00Z,Two readings are defensible; the canonicalizer accepts both.
M01,cancelled orders,mongo,simple;enum,,"{""status"":""CANCELLED""}",,,2026-08-04T00:00:00Z,A bare filter: collection/limit/sort are not asserted.
R01,orders shipped from the Ohio warehouse,sql,refusal,,,unsupported,,2026-08-04T00:00:00Z,No warehouse field exists. The model must refuse.
P01,orders over 500,sql,scope,"SELECT <default> FROM orders WHERE customer_name = 'acme' AND amount > 500 LIMIT 50",,,"{""customerName"":""acme""}",2026-08-04T00:00:00Z,Scope is AND-ed at the root; the model never sees it.

Reading those four rows in order:

• S07 — a plain SQL case. Exactly one of the three expect_* columns is filled, the other two stay empty.
• M01 — the same sentence on MongoDB. backend switches, expect_sql empties and expect_mongo fills.
• R01 — a negative case: the config has no warehouse field, so the correct behaviour is a refusal. expect_error is set instead of a query, which is how you test that the model declines rather than quietly answering with a lookalike column.
• P01 — the same sentence as a positive case, plus a scope filter the application imposes (a tenant, a subscription). It is AND-ed onto the query after validation and never appears in the model's prompt, so the expectation carries a predicate the sentence never mentioned.

Three things worth knowing before you write cases

1. Pin now on anything involving a relative date. "Last 30 days" resolves against the run clock, so an expectation written today stops matching tomorrow. With now set, the harness fixes both the generation clock and the date in the model's prompt (see Clock pinning below). -dry-run tells you how many cases are unpinned.

2. Write <default> for the projection. When a config marks any field returnable:false, the library stops emitting SELECT * and emits an explicit allow-list instead (the BUG-004 guarantee). That list is fully determined by the config, so repeating six column names on every row costs effort and measures nothing about the model:

SELECT <default> FROM orders WHERE amount > 500 LIMIT 50

The placeholder resolves by asking the library what the config compiles to, so it cannot drift from the real rule. Writing an actual column list still works and is still compared strictly — so the returnable:false guarantee stays testable.

3. For Mongo, a bare filter is usually what you want. If the expected JSON has no envelope key, only the filter is compared:

{"status":"CANCELLED"}

To assert the collection, sort or limit as well, write the envelope — and note that only the keys you actually write are compared:

{"collection":"orders","sort":[{"field":"createdAt","order":-1}],"limit":10}

Grades

The result is a ladder, not a boolean, because "did it match" collapses situations you want to tell apart.

grade	meaning	counts as
EXACT	byte-identical to the expectation	pass
EQUIVALENT	different text, same meaning	pass
REFUSED	correctly refused a request the config cannot express	pass
FAIL	produced a query, but the wrong one — the model misread the sentence	fail
ERROR	no query: transport failure, unparseable output, or an unexpected refusal	fail

EQUIVALENT is the normal grade for a passing case, not a lesser one. ERROR is kept separate from FAIL because it usually says something about the run (a rate limit, a dead endpoint) rather than about the model's comprehension.

What "equivalent" covers, and what it deliberately does not

A plain string comparison against generator output would report failures that have nothing to do with comprehension. The canonicalizer absorbs exactly these:

• Predicate order. The library reorders an AND/OR node's children by the config's indexed/priority hints before rendering, so emitted clause order is a property of the config, not of the sentence.
• Parameterization. total > $1 + args vs the inline total > 500.
• Nesting and parentheses. The generator nests as the AST nests; you write flat.
• Number, date and quoting spellings. 500/500.0, 2026-07-05 vs 2026-07-05T00:00:00Z, 'x' vs "x". Timestamps are compared to the second.
• Set membership. status IN (A,B) ≡ status = A OR status = B; NOT IN ≡ an AND of <>; tags && ARRAY[a,b] ≡ an OR of singleton @>. Measured live, the model picks between these spellings unpredictably for the same sentence.
• Projection and IN-list order, which are cosmetic.

It deliberately does not absorb, because each of these is a real difference:

• SELECT * vs an explicit column list (the returnable:false guarantee)
• ORDER BY key order and direction
• BETWEEN bound order
• inclusive vs exclusive bounds (>= vs >)
• contains-all vs contains-any at two or more elements
• f = a AND f = b (a contradiction) — never merged into a set
• f <> a OR f <> b (a tautology) — never merged into NOT IN
• the number 500 vs the string "500"

Every equivalence above is paired in sqlnorm_test.go / mongonorm_test.go with the neighbouring non-identity it must not swallow. That pairing is the point: a canonicalizer that is too aggressive turns a wrong query into a passing grade, and nothing ever surfaces it.

Measuring stability

-repeat N runs each sentence N times and reports how many passed:

  ⚠ 3 case(s) were FLAKY — they passed some repeats and failed others.

A case passes only if every repeat passed. This is the honest reading: a sentence the model gets right three times in five is not a sentence the model handles, and a single-shot harness cannot tell the two apart. When comparing models, this is usually the column that decides it.

Quota and rate limits

-rpm spaces calls evenly to stay under a free-tier ceiling (it is a plain gate rather than a token bucket — a burst is exactly what trips a quota). Transport failures are retried with exponential backoff (-retries) and are not counted against the model until the budget is gone; a refusal or unparseable output is never retried, because both are answers about the model.

-parallel controls how many cases are in flight. Ctrl-C reports whatever finished rather than discarding the run.

Clock pinning

Engine.Now fixes the generation clock, where a relative date is resolved into a timestamp. But the library's Planner keeps its own clock and Engine.planner is unexported, so an outside caller cannot reach it — the prompt would still tell the model today's real date, and any sentence the model chooses to resolve itself ("in July", "since last Tuesday") would drift.

qfeval closes that by wrapping the ModelProvider and rewriting the Today (UTC): … line in the system prompt. This depends on a string the library emits rather than on a documented seam, so the dependency is confined to one regexp and the harness warns if the line is ever absent instead of silently leaving the clock unpinned.

The clean fix is three lines in the library — have NewWithProvider point the planner's clock at the engine's:

eng := &Engine{ /* … */ }
eng.planner.Now = eng.now
return eng

With that in place the wrapper becomes redundant and can be deleted.

Testing

go test ./...

The suite runs fully offline: a stub provider stands in for the model, so the whole pipeline — golden-file load, config load, engine, translate, grade, aggregate, report — is exercised with no network and no API key. Those tests also pin the harness's assumptions about the library's wire format, so a change to the AST shape or the refusal envelope fails here rather than showing up as a mystifying mass failure on a live run.

Related

qfeval scores QueryForge — the Go library that turns a natural-language sentence into SQL or MongoDB by having the model fill in a validated Query AST rather than write a query string.

Library	https://github.com/awsaman-ai/queryforge
Documentation	https://awsaman-ai.github.io/queryforge/
Config builder	https://awsaman-ai.github.io/queryforge/config-builder.html
Live demo	https://queryforge-demo.amtry.in

qfeval is deliberately a separate module: it is a QA tool, so keeping it out of the library leaves the published package a library with no evaluation machinery in its public surface. It depends on the released github.com/awsaman-ai/queryforge the same way any other consumer does.

License

Licensed under the Apache License, Version 2.0 — the same licence as QueryForge itself.