Add joined queries (join= on search/count/lucky) and the $col special key

[roed-math]

Implements the two query-language extensions from the pre-1.0 plan, per the joins design doc, now including the v2 lifts from review: joined sorts, RIGHT/FULL joins, and cross-table $col / $or / $and / $not.

join= on search, count, lucky (and lookup)

db.ec_nfcurves.search(
    {"rank": 1, "nf_fields.r2": 1},
    ["label", "nf_fields.degree"],
    join=[("field_label", "nf_fields.label")],
    limit=3,
)

• Each join entry is (col1, col2) or (col1, col2, jointype): col2 is qualified "table.column" and names the table being joined; col1 belongs to the primary table, or to a previously joined table if qualified (enabling chains). jointype is "inner" (default), "left", "right" or "full".
• One resolution rule everywhere: query keys, projection entries, sort entries, and $col names split at the first . — a joined-table prefix means that table's column (further dots are a path within it); anything else is a primary-table name with dots keeping their jsonb/array path meaning. Result dictionaries use projection entries verbatim as keys.
• The parser is join-aware (v2): _parse_dict/_parse_special thread a join context and resolve each name against its owning table, emitting qualified identifiers directly. So $or/$and/$not clauses span tables, and $col and $raw reach columns of different tables (names in both resolve by the same rule; filter_sql_injection resolves qualified pieces against the validated join specification and each table's own whitelist). (v1 split the query per table and qualified afterwards, which structurally couldn't express a clause mixing tables.)
• Sorting accepts joined columns by the same rule, ascending or (col, -1) descending; the default sort is the primary table's.
• Hot path untouched: join=None dispatches to exactly the previous code before anything else happens.
• Uncached: joined counts run SELECT COUNT(*) every time — no quick_count, no counts-table writes (tested).
• RIGHT/FULL joins surface rows whose primary columns are NULL — project a joined column to identify them (documented, tested).
• Remaining restrictions raise ValueError: split_ors/one_per/raw/groupby with join, dictionary projections, duplicate/self joins, the old quadruple join format.

$col — compare two columns

{"col1": {"$col": "col2"}} → "col1" = "col2"; also usable as the operand of any infix operator, with array-slicer support. The name resolves exactly like a query key, so under a join it can name any table's column. Independent of joins for the same-table case.

Removes join_search

The 2023-era method required tuple keys, had no count/left-join/validation, and generated invalid SQL whenever two tables were constrained. No callers in lmfdb, seminars, psetpartners, or lmfdb-lite.

Docs and tests

• QueryLanguage.md: Joined queries and $col sections (single resolution rule, jointypes, NULL-row semantics), and the $raw section's "Details to be added" is finally written.
• 21 tests in tests/test_joins.py — projections, constraint placement, jsonb paths on joined tables, LEFT/RIGHT/FULL, chained joins, joined-column sorts, cross-table $col, $or/$not across tables, info paging, streaming, uncached counts, lucky/lookup, error cases — cross-table $raw, plus 4 $col tests in test_search.py, unit tests for the joined filter_sql_injection, and 2 devmirror tests cross-checked row-by-row against nf_fields.
• Review round: joined projections and sorts accept dotted paths (nf_fields.class_group.0) via a shared _column_composable; analyze gains join= and slow-query logs for joined searches append join=[...] so their replication commands run; an accidental deletion of config.ini is restored (paths in projections outside joins remain slicer-only — pre-existing, guarded by a test).
• Suite: 554 passed / 26 skipped, ruff clean, all 23 devmirror tests pass; all CI checks green including the LMFDB-under-Sage job.

🤖 Generated with Claude Code