Skip to content

fix: json_each on SQLite generates invalid SQL - #169

Merged
richardwooding merged 2 commits into
SPANDigital:mainfrom
scottlaird:fix/json-each-iteration-variable
Aug 22, 2026
Merged

fix: json_each on SQLite generates invalid SQL#169
richardwooding merged 2 commits into
SPANDigital:mainfrom
scottlaird:fix/json-each-iteration-variable

Conversation

@scottlaird

@scottlaird scottlaird commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Fixes #168.

SQLIte's json_each and MySQL's JSON_TABLE are table-valued functions, and cel2sql generates invalid SQL for them, breaking the ability to do comprehensions across JSON fields.

For example, for SQLite approval.exists(a, a == "alice") generates this SQL:

EXISTS (SELECT 1 FROM json_each(approvals) AS a WHERE a = ?)

However, for SQLite rejects this with no such column: a at query time. The correct SQL would be

EXISTS (SELECT 1 FROM json_each(approvals) AS a WHERE a.value = ?)

Since JSON handling isn't standardized in SQL, each dialect support is slightly different:

  • BigQuery, DuckDB, PostgreSQL, Spark: UNNEST and EXPLODE are row-valued and work as-is.
  • SQLite and MySQL: json_each and JSON_TABLE are table-valued and do not work.

This PR allows dialects to generate the correct syntax for JSON comprehensions and fixes SQLite and MySQL.

Scope

all, exists, exists_one, filter and map all go through the same source writer, so all five were affected.

Tests

sqlite_iteration_variable_test.go converts and then executes against an in-memory SQLite database, asserting the rows that come back. That is the check this bug asks for: the previous expectations were SQL that parsed and could not run. Against main it fails with no such column: a.

The shared SQLite expectations in testcases/comprehension_tests.go encoded the old output and are updated; they are the same five comprehensions.

go test ./... is otherwise unchanged from main on this machine — the 27 failures before and after are the testcontainers suites, which need Docker.

Note on the MySQL change

I included MySQL because WriteUnnest there emits JSON_TABLE(… COLUMNS(value TEXT PATH '$')), which is also table-valued, although the exact structure is slightly different. I could not execute against MySQL here, so that half is untested. I can split it out into its own PR if you'd prefer.

json_each and JSON_TABLE are table-valued functions, so the alias a
comprehension binds names a row rather than the array's element:

  approvals.exists(a, a == "alice")
    -> EXISTS (SELECT 1 FROM json_each(approvals) AS a WHERE a = ?)

SQLite rejects that with "no such column: a", and MySQL the same way, after
the converter has reported the conversion a success. The failure appears at
query time rather than at conversion time, which is the awkward part: nothing
in the caller's path says the SQL is wrong.

A reference to an iteration variable is now written by the dialect, which is
the only thing that knows what its own comprehension source bound the alias
to. Dialect.WriteIterVarRef defaults to the bare alias -- correct for
BigQuery, DuckDB, Postgres and Spark, where UNNEST and EXPLODE bind the
element itself -- and SQLite and MySQL qualify it, unconditionally, because
both of their sources are the table-valued form.

The converter tracks which names are iteration variables so that only those
are routed that way, scoped to the comprehension and restored afterwards,
since comprehensions nest and an inner one may reuse a name. The filter
comprehension writes the variable into its projection ahead of the FROM
clause, so the binding is taken at the top of each visitor rather than after
the source is written.

The shared SQLite expectations in testcases/comprehension_tests.go encoded the
old output and are updated. sqlite_iteration_variable_test.go is new and
executes what it converts against an in-memory database, which is the check
this bug asks for: the previous expectations were SQL that parsed and could
not run.

Fixes SPANDigital#168
@scottlaird scottlaird changed the title fix: write a comprehension's iteration variable through the dialect fix: json_each on MySQL and SQLite returns a table, not a row Aug 22, 2026
@scottlaird scottlaird changed the title fix: json_each on MySQL and SQLite returns a table, not a row fix: json_each on MySQL and SQLite generates invalid SQL Aug 22, 2026
@scottlaird scottlaird changed the title fix: json_each on MySQL and SQLite generates invalid SQL fix: json_each on SQLite generates invalid SQL Aug 22, 2026
@scottlaird

Copy link
Copy Markdown
Contributor Author

@richardwooding SQLite and MySQL have broken implementations of JSON decoding; this should fix them.

@richardwooding

Copy link
Copy Markdown
Contributor

Thanks @scottlaird I will get this PR in

@codecov-commenter

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@richardwooding

Copy link
Copy Markdown
Contributor

Thanks @scottlaird for the excellent fix and the thorough write-up — the executable SQLite test that actually runs the generated SQL is exactly the right kind of regression coverage for this, and including the MySQL JSON_TABLE half saves us a follow-up. I pushed a small lint suppression for the intentional SQL concatenation in the new test. The remaining red checks are pre-existing dependency vulnerabilities failing on main (being fixed in #170) and the benchmark-storage step that can't run from forks. Merging!

@richardwooding
richardwooding merged commit 818461e into SPANDigital:main Aug 22, 2026
6 of 9 checks passed
richardwooding added a commit that referenced this pull request Aug 22, 2026
…177)

* test: execute MySQL JSON comprehension SQL against a real MySQL

Mirrors TestSQLiteComprehensionRunsAgainstSQLite for the JSON_TABLE half
of #169: converts exists/all/exists_one over a JSON array column and runs
the generated SQL in a MySQL 8 testcontainer, asserting the returned rows.

Closes #173

* test: run MySQL integration tests against 8.4 LTS

MySQL 8.0 (EOL April 2026) silently returns no rows for a correlated
JSON_TABLE inside EXISTS when executed as a prepared statement — the
same query works with literals on 8.0 and works in every form on newer
versions, so the generated SQL is correct and 8.0 is the outlier.

* fix: MySQL comprehension exists/all silently matched nothing on 8.x

The MySQL 8.x optimizer transforms a correlated EXISTS into a semijoin
and loses the correlation to a JSON_TABLE source, so the subquery runs
without error and matches nothing (verified on 8.0 and 8.4; works on
9.x, or on 8.x with optimizer_switch='semijoin=off'). COUNT comparisons
are never transformed, so the MySQL dialect now writes
(SELECT COUNT(*) FROM ...) > 0 / = 0 for exists/all, the same shape
exists_one already used.

Adds Dialect.WriteComprehensionExists/WriteComprehensionNotExists so
each dialect owns the quantifier wrapper; all other dialects keep the
EXISTS forms. Adds MySQL expectations to the shared comprehension
testcases, which previously had none.
richardwooding added a commit to SPANDigital/pycel2sql that referenced this pull request Aug 22, 2026
Ports two fixes from the Go cel2sql (SPANDigital/cel2sql#169, #177),
where they were found by executing generated comprehension SQL against
real databases:

- SQLite json_each and MySQL JSON_TABLE are table-valued, so a bare
  reference to the iteration variable was 'no such column'; DuckDB's
  FROM UNNEST(arr) AS a binds a to a STRUCT row, making a = 'x' a cast
  error. A new Dialect.write_comprehension_source hook lets each
  dialect bind the value to the variable (derived-table rename for
  json_each/JSON_TABLE, AS _t(var) for DuckDB).

- MySQL 8.x transforms a correlated EXISTS into a semijoin and loses
  the correlation to JSON_TABLE, so exists()/all() executed without
  error and silently matched nothing (works from 9.x; verified against
  MySQL 8.4.11 and 9.7.1). New write_comprehension_exists /
  write_comprehension_not_exists hooks let MySQL emit COUNT
  comparisons, which are never transformed.

Adds integration tests that execute exists/all/exists_one against
PostgreSQL, DuckDB, SQLite, and MySQL.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cel2sql generates invalid SQLite for JSON columns

3 participants