fix: json_each on SQLite generates invalid SQL - #169
Conversation
json_each and JSON_TABLE are table-valued functions, so the alias a
comprehension binds names a row rather than the array's element:
approvals.exists(a, a == "alice")
-> EXISTS (SELECT 1 FROM json_each(approvals) AS a WHERE a = ?)
SQLite rejects that with "no such column: a", and MySQL the same way, after
the converter has reported the conversion a success. The failure appears at
query time rather than at conversion time, which is the awkward part: nothing
in the caller's path says the SQL is wrong.
A reference to an iteration variable is now written by the dialect, which is
the only thing that knows what its own comprehension source bound the alias
to. Dialect.WriteIterVarRef defaults to the bare alias -- correct for
BigQuery, DuckDB, Postgres and Spark, where UNNEST and EXPLODE bind the
element itself -- and SQLite and MySQL qualify it, unconditionally, because
both of their sources are the table-valued form.
The converter tracks which names are iteration variables so that only those
are routed that way, scoped to the comprehension and restored afterwards,
since comprehensions nest and an inner one may reuse a name. The filter
comprehension writes the variable into its projection ahead of the FROM
clause, so the binding is taken at the top of each visitor rather than after
the source is written.
The shared SQLite expectations in testcases/comprehension_tests.go encoded the
old output and are updated. sqlite_iteration_variable_test.go is new and
executes what it converts against an in-memory database, which is the check
this bug asks for: the previous expectations were SQL that parsed and could
not run.
Fixes SPANDigital#168
json_each on MySQL and SQLite returns a table, not a row
json_each on MySQL and SQLite returns a table, not a rowjson_each on MySQL and SQLite generates invalid SQL
json_each on MySQL and SQLite generates invalid SQLjson_each on SQLite generates invalid SQL
|
@richardwooding SQLite and MySQL have broken implementations of JSON decoding; this should fix them. |
|
Thanks @scottlaird I will get this PR in |
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
|
Thanks @scottlaird for the excellent fix and the thorough write-up — the executable SQLite test that actually runs the generated SQL is exactly the right kind of regression coverage for this, and including the MySQL JSON_TABLE half saves us a follow-up. I pushed a small lint suppression for the intentional SQL concatenation in the new test. The remaining red checks are pre-existing dependency vulnerabilities failing on main (being fixed in #170) and the benchmark-storage step that can't run from forks. Merging! |
…177) * test: execute MySQL JSON comprehension SQL against a real MySQL Mirrors TestSQLiteComprehensionRunsAgainstSQLite for the JSON_TABLE half of #169: converts exists/all/exists_one over a JSON array column and runs the generated SQL in a MySQL 8 testcontainer, asserting the returned rows. Closes #173 * test: run MySQL integration tests against 8.4 LTS MySQL 8.0 (EOL April 2026) silently returns no rows for a correlated JSON_TABLE inside EXISTS when executed as a prepared statement — the same query works with literals on 8.0 and works in every form on newer versions, so the generated SQL is correct and 8.0 is the outlier. * fix: MySQL comprehension exists/all silently matched nothing on 8.x The MySQL 8.x optimizer transforms a correlated EXISTS into a semijoin and loses the correlation to a JSON_TABLE source, so the subquery runs without error and matches nothing (verified on 8.0 and 8.4; works on 9.x, or on 8.x with optimizer_switch='semijoin=off'). COUNT comparisons are never transformed, so the MySQL dialect now writes (SELECT COUNT(*) FROM ...) > 0 / = 0 for exists/all, the same shape exists_one already used. Adds Dialect.WriteComprehensionExists/WriteComprehensionNotExists so each dialect owns the quantifier wrapper; all other dialects keep the EXISTS forms. Adds MySQL expectations to the shared comprehension testcases, which previously had none.
Ports two fixes from the Go cel2sql (SPANDigital/cel2sql#169, #177), where they were found by executing generated comprehension SQL against real databases: - SQLite json_each and MySQL JSON_TABLE are table-valued, so a bare reference to the iteration variable was 'no such column'; DuckDB's FROM UNNEST(arr) AS a binds a to a STRUCT row, making a = 'x' a cast error. A new Dialect.write_comprehension_source hook lets each dialect bind the value to the variable (derived-table rename for json_each/JSON_TABLE, AS _t(var) for DuckDB). - MySQL 8.x transforms a correlated EXISTS into a semijoin and loses the correlation to JSON_TABLE, so exists()/all() executed without error and silently matched nothing (works from 9.x; verified against MySQL 8.4.11 and 9.7.1). New write_comprehension_exists / write_comprehension_not_exists hooks let MySQL emit COUNT comparisons, which are never transformed. Adds integration tests that execute exists/all/exists_one against PostgreSQL, DuckDB, SQLite, and MySQL.
Fixes #168.
SQLIte's json_each and MySQL's JSON_TABLE are table-valued functions, and
cel2sqlgenerates invalid SQL for them, breaking the ability to do comprehensions across JSON fields.For example, for SQLite
approval.exists(a, a == "alice")generates this SQL:However, for SQLite rejects this with
no such column: aat query time. The correct SQL would beSince JSON handling isn't standardized in SQL, each dialect support is slightly different:
UNNESTandEXPLODEare row-valued and work as-is.json_eachandJSON_TABLEare table-valued and do not work.This PR allows dialects to generate the correct syntax for JSON comprehensions and fixes SQLite and MySQL.
Scope
all,exists,exists_one,filterandmapall go through the same source writer, so all five were affected.Tests
sqlite_iteration_variable_test.goconverts and then executes against an in-memory SQLite database, asserting the rows that come back. That is the check this bug asks for: the previous expectations were SQL that parsed and could not run. Againstmainit fails withno such column: a.The shared SQLite expectations in
testcases/comprehension_tests.goencoded the old output and are updated; they are the same five comprehensions.go test ./...is otherwise unchanged frommainon this machine — the 27 failures before and after are the testcontainers suites, which need Docker.Note on the MySQL change
I included MySQL because
WriteUnnestthere emitsJSON_TABLE(… COLUMNS(value TEXT PATH '$')), which is also table-valued, although the exact structure is slightly different. I could not execute against MySQL here, so that half is untested. I can split it out into its own PR if you'd prefer.