BIRD dev (bird_sql_dev_20251106): 23 golds still cut at a tie, sort NULL first, or sort numbers as text

[ivermin1123]

This is about the dev set, not Mini-Dev: the 2025-11-06 pass published as birdsql/bird_sql_dev_20251106 on Hugging Face. Posting here because this is where the BIRD team answers; happy to move it if there is a better place.

I ran three mechanical checks over every gold of the 2024 dev.json and of this 2025-11-06 pass, executing each statement on the dev_databases that ship with dev.zip: does a LIMIT cut through rows the data leaves tied, does an ORDER BY key hold NULLs that sort first, and is a numeric-looking ORDER BY key stored as text. Nothing semantic; the checks only ask whether the shipped data determines the answer the gold returns.

Of the 399 golds this pass rewrote, 31 fired on the old text and 29 of those stop firing on the new one, so the rewrite and the checks mostly agree. Of the 963 golds left unchanged, 25 fired. I read all 25 against the data by hand: 23 do not answer their question on this data, 1 is harmless (q893, two drivers tied on points, same set either way), 1 is my check being stricter than the question (q423 asks for alphabetical order of a text column that happens to hold digits).

The 23, by what goes wrong:

Mechanism	Questions	What happens on the shipped data
LIMIT cuts through a tie	30, 57, 82, 392, 766, 802, 906, 1002, 1034, 1090, 1117, 1144, 1290, 1365, 1389, 1517	the rows tied at the cut carry different values for the column asked; q766 has 63 rows tied on the largest strength naming 36 heroes, q392 has 418 cards on the earliest ruling date and returns 3 of them, q57 asks for the 333rd highest inside a seven-row tie
NULL sorts first under ASC	81, 847	the gold returns a row with no value in the ordering key (a school with no latitude; a driver who set no qualifying time) instead of the lowest one
numbers sorted as text	879, 927	fastestLapSpeed is a text column, so 9.5 sorts above 10 and the fastest speed is not the one returned; q879 is the case from bird-bench/mini_dev issue 24, fixed in Mini-Dev's Hugging Face copy but still here
aggregate beside ungrouped columns	1004	a bare SUM with no GROUP BY next to two ungrouped columns: the total is every driver's wins added up (7,890) and the name beside it is one arbitrary row's
no cut where the question asks for the top	484	the gold orders by converted mana cost and never limits, returning all 155 Italian cards of the set where 12 carry the highest cost
average of ids	523	the projected average is a sum of ids divided by a count of ids, and several release dates share the count the gold cuts at

Why it matters for scoring: the evaluator compares result sets, so on a tie-cut gold a prediction that picks another of the tied rows, which is just as correct, scores 0, and on q81 and q847 a prediction that filters the NULLs out scores 0 for being right.

Per-question reasons, the probe outputs, and a script that reruns all of it against dev.zip are here: https://github.com/ivermin1123/attestql/tree/main/plans/reports/bird-dev-sqlite-260907 (the hand reading is classification.json). Happy to open one issue per question or send a corrected SQL for each if that is easier for you.