Translate string.Join/Concat with ordering on SQLite#38344
Conversation
SQLite 3.44.0 added support for ORDER BY inside aggregate functions (group_concat(value, separator ORDER BY ...)). Translate string.Join and string.Concat over an ordered source instead of falling back to client evaluation. - Add SqliteAggregateFunctionExpression carrying the orderings - Render the ORDER BY inside the function in SqliteQuerySqlGenerator - Handle the new expression in SqliteSqlNullabilityProcessor - Enable the Join_with_ordering test Fixes dotnet#32201
There was a problem hiding this comment.
Pull request overview
This PR enables translation of string.Join/string.Concat over an ordered grouping in the SQLite provider by emitting SQLite’s group_concat(... ORDER BY ...) syntax (SQLite >= 3.44.0), avoiding client evaluation for these queries.
Changes:
- Introduces
SqliteAggregateFunctionExpressionto represent aggregate function invocations that carryORDER BYinformation. - Updates SQLite string-aggregate translation to allow ordered inputs and updates SQL generation to render
ORDER BYinside the aggregate function parentheses. - Enables the
Join_with_orderingSQLite functional test to assert server translation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/EFCore.Sqlite.FunctionalTests/Query/Translations/StringTranslationsSqliteTest.cs | Updates expected SQL for ordered string aggregation using group_concat(... ORDER BY ...). |
| src/EFCore.Sqlite.Core/Query/Internal/Translators/SqliteStringAggregateMethodTranslator.cs | Emits the new aggregate expression when orderings are present instead of refusing translation. |
| src/EFCore.Sqlite.Core/Query/Internal/SqliteSqlNullabilityProcessor.cs | Adds nullability-processing support for the new custom SQL expression node. |
| src/EFCore.Sqlite.Core/Query/Internal/SqliteQuerySqlGenerator.cs | Adds SQL generation for aggregate ORDER BY inside function parentheses. |
| src/EFCore.Sqlite.Core/Query/Internal/SqlExpressions/SqliteAggregateFunctionExpression.cs | New SQL expression type carrying aggregate function name, arguments, and orderings. |
- Refuse translation when the SQLite version is below 3.44 (ServerVersion check), falling back to client evaluation as before - Refuse ORDER BY combined with DISTINCT, which SQLite restricts to ordering by the distinct value itself - Skip Join_with_ordering on SQLite versions below 3.44
|
Good catches all three addressed in b89fb2b: version-gated to 3.44 (falls back to client eval below that), DISTINCT + ORDER BY now refused, and the test is gated the same way. |
|
Add a test for DISTINCT + ORDER BY only when ordering by the distinct column |
SQLite's group_concat() accepts only a single argument when DISTINCT is used,
so it cannot be combined with the separator that string.Join/Concat always
supply ("DISTINCT aggregates must have exactly one argument"), independent of
ordering. Refuse translation for DISTINCT (the query then falls back to APPLY,
which SQLite doesn't support) rather than emit SQL that fails at execution time.
- Refuse translation whenever IsDistinct (previously only refused together with
an ORDER BY)
- Add Join_with_distinct asserting it reports ApplyNotSupported
|
Looked into the distinct + ordering case. Two things block it. First, EF raises |
@dotnet-policy-service agree |
|
Thanks for the review and merge! |
Translates
string.Join/string.Concatover an ordered grouping to SQLite'sgroup_concatwith anORDER BYclause, supported since SQLite 3.44.0. Previously such queries fell back to client evaluation.Changes
SqliteAggregateFunctionExpression(carries the orderings), mirroringSqlServerAggregateFunctionExpressionSqliteStringAggregateMethodTranslator: stop refusing translation when an ordering is present; emit the new expression (plaingroup_concatwhen there's no ordering — SQL unchanged)SqliteQuerySqlGenerator: rendergroup_concat(value, separator ORDER BY ...). Note SQLite places the ordering inside the parentheses, unlike SQL Server'sWITHIN GROUPSqliteSqlNullabilityProcessor: handle the new expressionJoin_with_orderinginStringTranslationsSqliteTestNotes
group_concat(string aggregation);ORDER BYis meaningless forMin/Max.group_concat(DISTINCT x ORDER BY y)is restricted in SQLite (DISTINCT + ORDER BY only when ordering by the distinct column); no base test covers this combination.Fixes #32201