feat: Add `ALL` and `ANY` operators accepting array, subquery, or table parameters #1886

ShreckYe · 2023-11-16T11:06:25Z

ALL and ANY are 2 SQL operators to perform a comparison between a single column value and a range of other values.

According to java - PreparedStatement IN clause alternatives? - Stack Overflow, using = ANY(?) and PreparedStatement.setArray if supported can be more efficient than using IN (inList in Exposed) when passing lists of different sizes because a single cached prepared statement can be reused. This is the major motive of this PR.

Since according to references online they are called operators not functions, and CustomOperator in Exposed has 2 operands, I am not sure whether it is better to extend Op or CustomFunction, and so I have implemented both and they are both tested to work. Please help remove the less appropriate one if this PR is accepted.

…e simple tests which are not run yet

…rray of strings) to work on PostgreSQL (`postgres` and `postgresNG`) Cherry-picked from: 7dd846b

…and refactor to simplify the `AllAnyOp` implementation As tested, it works for H2 as well in addition to PostgreSQL.

…lines

e5l · 2023-12-06T12:16:10Z

@bog-walk, could you please check?

e5l · 2023-12-06T12:16:50Z

Hey @ShreckYe, thanks for the PR. Could you please run apiDump task and commit the changes?

bog-walk

Hi @ShreckYe and thanks for working on this PR. These operators will be a great addition to the API, so I've requested some changes to ensure that this implementation is extensible enough to be adapted later.

Using these operators with an array argument is very niched to PostgreSQL, and the standard way of using ANY|ALL is with a subquery on the right hand side (supported by all except SQLite). This will be the expectation and natural progression if these operators are supported.
MySQL also has a very niched use case, accepting a table on the right hand side. We need to be able to add that functionality if users ask for it, especially if we're adding the PostgreSQL array.

The most ideal way to keep your functionality and future use cases possible would be to implement the feature in a way similar to SubQueryOp, but this would mean creating a class and a function for every single comparison operator that can precede ANY|ALL and the DSL would become more verbose.

I think your way of incorporating the existing comparison operators reads better, so I would suggest extending Op and creating a sealed class that allows us to build on it later. I've included this class and how to use it in my comments below.

bog-walk · 2023-12-06T21:10:02Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/ColumnType.kt

+object UntypedAndUnsizedArrayColumnType : ColumnType() {
+    override fun sqlType(): String =
+        currentDialect.dataTypeProvider.untypedAndUnsizedArrayType()
+}


I appreciate the need for this and for the scope of this PR it's ok. It does have some problems, for example:

If a logger is enabled, the array object value is not processed in the output string, which isn't helpful to users wanting to see the values.

It won't work with edge cases like if a user wants to check if a datetime column value is in an array of LocalDateTimes.

But we need to flesh out the ArrayColumnType more thoroughly anyway, so I can work on these problems after this PR is merged.

I didn't remove this because the implemented Ops still depend on it. As you commented this will be fleshed out, so to not keep these new definitions in the PR, one way I can think of is to replace its usage with a temporary inlined anonymous ColumnType with its sqlType being the empty string or just "ARRAY" and the tests still pass.

Sure, that sounds like a good temporary option to me.

OK, so shall I remove it and adopt this approach in the next commit?

bog-walk · 2023-12-06T21:11:25Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/Op.kt

+/** Returns the specified [value] as an array query parameter. */
+fun <T> arrayParam(value: Array<T>): Expression<Array<T>> = QueryParameter(value, UntypedAndUnsizedArrayColumnType)


Please remove this as it isn't necessary for the Op implementation. An arrayParam() will be introduced to the API when the ArrayColumnType is fully supported for storing arrays.

bog-walk · 2023-12-06T21:23:45Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/SQLExpressionBuilder.kt

+/** Returns this array of data wrapped in the `ALL` operator. */
+fun <T> Array<T>.allOp(): Op<T> = AllAnyOp("ALL", this)
+
+/** Returns this array of data wrapped in the `ANY` operator. The name is explicitly distinguished from [Array.any]. */
+fun <T> Array<T>.anyOp(): Op<T> = AllAnyOp("ANY", this)


Having these be extension functions makes it difficult to extend the functionality to, for example, a subquery. It also doesn't read like a natural SQL query: WHERE { Users.id eq arrayOf(1, 2).anyOp() }.

Please replace them with anyFrom which takes an array argument, that way we can make an override for subqueries and both will read well:

fun <T> anyFrom(array: Array<T>): AnyFromArrayOp<T> = AnyFromArrayOp(array) fun <T> allFrom(array: Array<T>): AllFromArrayOp<T> = AllFromArrayOp(array) // use WHERE { Users.id eq anyFrom(arrayOf(1, 2)) } WHERE { Users.id eq anyFrom(Users.selectAll()) }

Also, please place the functions in the subsection titled "// Array Comparisons".

bog-walk · 2023-12-06T21:27:05Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/SQLExpressionBuilder.kt

+/** Checks if this expression is equal to any element from [array].
+ * This is a more efficient alternative to [ISqlExpressionBuilder.inList] on PostgreSQL and H2. */
+infix fun <T> ExpressionWithColumnType<T>.eqAny(array: Array<T>): Op<Boolean> =
+    this eq array.anyOp() // TODO or `array.anyFunction()`


This won't be necessary once the functions above are switched to take an argument instead of the more verbose extension function.

bog-walk · 2023-12-06T21:35:33Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/ops/AllAnyOps.kt

+class AllAnyOp<T>(val opName: String, val array: Array<T>) : Op<T>() {
+
+    override fun toQueryBuilder(queryBuilder: QueryBuilder) = queryBuilder {


My suggestion would be to replace this with a sealed class, something like this:

sealed class AnyOrAllFromBaseOp<T>( val isAny: Boolean, val subSearch: Any ) : Op<T>() { override fun toQueryBuilder(queryBuilder: QueryBuilder): Unit = queryBuilder { if (isAny) { +"ANY (" } else { +"ALL (" } when (subSearch) { is Array<*> -> registerArgument(UntypedAndUnsizedArrayColumnType, subSearch) } +")" } } class AnyFromArrayOp<T>(array: Array<T>) : AnyOrAllFromBaseOp<T>(isAny = true, subSearch = array) class AllFromArrayOp<T>(array: Array<T>) : AnyOrAllFromBaseOp<T>(isAny = false, subSearch = array)

With that, we'd be able to easily go in an extend support to subqueries, etc. by just adding a branch to the when block and some more subclasses.

I refactored opName into isAny as you suggested but there might still be other row and array comparison operators, for example SOME though it's only an alias of ANY. (see PostgreSQL docs and MySQL docs) If it's necessary to leave room for them (though I haven't find many cases supporting this), this class can be renamed to RowAndArrayComparisonOp as how PostgreSQL calls them and the old opName approach can be adopted.

Also I added a SubSearch type parameter instead of using Any for better type safety and implemented registerArgument in subclasses for better extensibility from my point of view. If this is unnecessary I can change them to what you suggested.

bog-walk · 2023-12-06T21:37:28Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/vendors/DataTypeProvider.kt

+    /** Data type for arrays with no specified size or element type, used only as types of [QueryParameter]s for PostgreSQL and H2.
+     * An array with no element type cannot be used for storing arrays in a column in either PostgreSQL or H2. */
+    abstract fun untypedAndUnsizedArrayType(): String
+


Let's go by the majority here, since only 2 dialects are implemented. Please swap this with the following and remove all exceptions from unsupported dialect files:

open fun untypedAndUnsizedArrayType(): String = throw UnsupportedByDialectException("This vendor does not support array data type", currentDialect)

bog-walk · 2023-12-06T21:40:04Z

exposed-tests/src/test/kotlin/org/jetbrains/exposed/sql/tests/shared/dml/SelectTests.kt

@@ -211,6 +213,55 @@ class SelectTests : DatabaseTestsBase() {
        }
    }

+    val testDBsSupportingArrays =
+        listOf(TestDB.POSTGRESQL, TestDB.POSTGRESQLNG, TestDB.H2, TestDB.H2_MYSQL, TestDB.H2_MARIADB, TestDB.H2_PSQL, TestDB.H2_ORACLE, TestDB.H2_SQLSERVER)


TestDB.allH2TestDB is available to replace listing all H2 modes.

bog-walk · 2023-12-06T21:50:03Z

exposed-tests/src/test/kotlin/org/jetbrains/exposed/sql/tests/shared/dml/SelectTests.kt

+    fun testEqAny(eqOp: Column<String>.() -> Op<Boolean>) {
+        withDb(testDBsSupportingArrays) {
+            withCitiesAndUsers { _, users, _ ->
+                val r = users.select { users.id.eqOp() }.orderBy(users.name).toList()
+
+                assertEquals(2, r.size)
+                assertEquals("Alex", r[0][users.name])
+                assertEquals("Andrey", r[1][users.name])
+            }


I appreciate that this helps reduce redundancy, especially since you implemented the feature using both Op and Function. After refactoring, please use this (or whatever code you're testing) directly in each unit test instead. That way it will be more clear what the DSL we're testing actually looks like when used.

The tests that would be great to see are:

eq anyFrom

One other comparison op: the example below is good -> greaterEq anyFrom

neq anyFrom

eq anyFrom(emptyArray())

Major changes: 1. support the `ALL` and `ANY` operators taking subquery arguments and table arguments, and the `IN` operator taking table arguments 1. remove some no longer needed code, simplify code, and rename some functions 1. add related tests 1. run `apiDump` The operators (`IN`, `ANY`, and `ALL`) with table arguments are only supported by PostgreSQL and H2. MySQL claims supporting this but it does not work with the JDBC driver as tested. Those with subquery arguments are not supported by SQLite.

ShreckYe · 2023-12-18T14:26:27Z

@bog-walk thank you for your comments. I have modified the sources as you suggested and requested. For some of them I still have some questions which I leave in their respective reply comments.

ShreckYe · 2023-12-18T14:52:17Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/SQLExpressionBuilder.kt

+/** Returns this subquery wrapped in the `ANY` operator. This function is not supported by the SQLite dialect. */
+fun <T> anyFrom(subQuery: Query): Op<T> = AllAnyFromSubQueryOp(true, subQuery)


The ANY and ALL operators with subquery arguments are not supported by SQLite as is also commented in the commit message. Actually ANY and ALL are not keywords of SQLite at all according to this SO question.

@ShreckYe I forgot to mention something. Please take AbstractQuery<*> as the argument so we don't restrict it to just Query. It should be possible to use a SetOperation for example as a subquery in the SQL.

ShreckYe · 2023-12-18T15:01:18Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/SQLExpressionBuilder.kt

+/** Returns this table wrapped in the `ANY` operator. This function is only supported by PostgreSQL and H2 dialects. */
+fun <T> anyFrom(table: Table): Op<T> = AllAnyFromTableOp(true, table)


Interestingly, The operators with table arguments are supported by PostgreSQL and H2 as tested though they haven't clearly documented about this, but the tests fail on MySQL who claims supporting it. I think it's probably because it's not supported by its JDBC connector. I verified by copying the SQL statement to an online MySQL executor and it works.

Yes, there is currently an issue with the mysql8 docker setup in our tests, so the test failing is a false negative.

OK, so shall I allow the tests to run on MySQL related dialects and just let them fail, or skip the tests as how they are now?

bog-walk

@ShreckYe Thanks for all the work so far, it's looking great. I've left some comments and requested a few small edits.

Also, please address the detekt edit requested in the failed action.

bog-walk · 2023-12-18T16:21:42Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/ColumnType.kt

+object UntypedAndUnsizedArrayColumnType : ColumnType() {
+    override fun sqlType(): String =
+        currentDialect.dataTypeProvider.untypedAndUnsizedArrayType()
+}


Sure, that sounds like a good temporary option to me.

bog-walk · 2023-12-18T16:25:07Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/SQLExpressionBuilder.kt

+/** Returns this table wrapped in the `ANY` operator. This function is only supported by PostgreSQL and H2 dialects. */
+fun <T> anyFrom(table: Table): Op<T> = AllAnyFromTableOp(true, table)


Yes, there is currently an issue with the mysql8 docker setup in our tests, so the test failing is a false negative.

bog-walk · 2023-12-18T16:28:00Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/SQLExpressionBuilder.kt

+// TODO Currently these functions delegate to `anyFrom`. Should the actual `SOME` SQL operator be supported?
+
+/** An alias for [anyFrom]. */
+fun <T> someFrom(subQuery: Query): Op<T> = anyFrom(subQuery)
+
+/** An alias for [anyFrom]. */
+fun <T> someFrom(array: Array<T>): Op<T> = anyFrom(array)
+
+/** An alias for [anyFrom]. */
+fun <T> someFrom(table: Table): Op<T> = anyFrom(table)
+


@ShreckYe There's already quite a bit in this PR. Please remove these aliases and then SOME can be introduced in another PR once this one is finalized. It won't take much tweaking with the current setup.

bog-walk · 2023-12-18T16:29:07Z

exposed-core/src/main/kotlin/org/jetbrains/exposed/sql/SQLExpressionBuilder.kt

+    /** Checks if this expression is equal to any element from the column of [table] with only a single column. This function is only supported by PostgreSQL and H2 dialects. */
+    infix fun <T> ExpressionWithColumnType<T>.notInTable(table: Table): InTableOp = InTableOp(this, table, false)


A "not" is probably missing from the KDocs here.

ShreckYe · 2023-12-19T14:49:00Z

@ShreckYe Thanks for all the work so far, it's looking great. I've left some comments and requested a few small edits.

Also, please address the detekt edit requested in the failed action.

Sorry I don't know how to fix the detekt failure. I don't see a blank line at the the location it mentions and my detekt plugin keeps crashing. I downloaded the detekt command line client and ran auto correction but nothing changed. Please help me check if it persists.

ShreckYe · 2023-12-19T14:57:51Z

OK I got the plugin to work on another computer and ran "AutoCorrect by detekt Rules" which removes Line 236 of the mentioned file.

e5l · 2024-01-03T12:34:21Z

Hey @ShreckYe, thanks for the PR!
LGTM. Merged

ShreckYe added 7 commits November 14, 2023 22:40

Initially implement allOp and anyOp for Iterable<T> and add som…

6aeed36

…e simple tests which are not run yet

Pass arrays instead of lists/iterables, getting testEqAny (for an a…

be49f65

…rray of strings) to work on PostgreSQL (`postgres` and `postgresNG`) Cherry-picked from: 7dd846b

Add an alternative custom function implementation of ALL and ANY …

2c549b1

…and refactor to simplify the `AllAnyOp` implementation As tested, it works for H2 as well in addition to PostgreSQL.

Improve code, add comments, and skip tests on unsupported dialects

020313c

Add a KDoc line

6076057

Add some more KDocs, add an eqAny shortcut, and fix a typo

7d0ef40

Describe return types explicitly to conform to the contributing guide…

c03aa1f

…lines

e5l requested a review from bog-walk December 6, 2023 12:15

bog-walk self-assigned this Dec 6, 2023

bog-walk requested changes Dec 6, 2023

View reviewed changes

ShreckYe added 3 commits December 11, 2023 10:06

Run apiDump again

93253a3

Complete the remaining tests

7157813

ShreckYe commented Dec 18, 2023

View reviewed changes

bog-walk requested changes Dec 18, 2023

View reviewed changes

Fix code issues as requested by @bog-walk

eeec230

AutoCorrect by detekt rules

30d4b85

bog-walk approved these changes Jan 2, 2024

View reviewed changes

bog-walk changed the title ~~feat: Add the ALL and ANY operators accepting array parameters with an added untyped and unsized array type for PostgreSQL and H2~~ feat: Add ALL and ANY operators accepting array, subquery, or table parameters Jan 2, 2024

bog-walk requested a review from e5l January 2, 2024 22:55

e5l approved these changes Jan 3, 2024

View reviewed changes

e5l merged commit 98df06d into JetBrains:main Jan 3, 2024
3 checks passed

bog-walk mentioned this pull request Feb 11, 2024

feat: EXPOSED-290 Support ANY and ALL operators using array column expressions #1988

Merged

bog-walk mentioned this pull request Feb 14, 2024

fix!: EXPOSED-288 Extend ANY and ALL operators to use ArrayColumnType #1992

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add `ALL` and `ANY` operators accepting array, subquery, or table parameters #1886

feat: Add `ALL` and `ANY` operators accepting array, subquery, or table parameters #1886

ShreckYe commented Nov 16, 2023

e5l commented Dec 6, 2023

e5l commented Dec 6, 2023

bog-walk left a comment •

edited

Loading

bog-walk Dec 6, 2023

ShreckYe Dec 18, 2023

bog-walk Dec 18, 2023

ShreckYe Dec 19, 2023 •

edited

Loading

bog-walk Dec 6, 2023

bog-walk Dec 6, 2023 •

edited

Loading

bog-walk Dec 6, 2023

bog-walk Dec 6, 2023

ShreckYe Dec 18, 2023 •

edited

Loading

bog-walk Dec 6, 2023

bog-walk Dec 6, 2023

bog-walk Dec 6, 2023

ShreckYe commented Dec 18, 2023

ShreckYe Dec 18, 2023

bog-walk Dec 18, 2023

ShreckYe Dec 18, 2023 •

edited

Loading

bog-walk Dec 18, 2023

ShreckYe Dec 19, 2023

bog-walk left a comment •

edited

Loading

bog-walk Dec 18, 2023

bog-walk Dec 18, 2023

bog-walk Dec 18, 2023

bog-walk Dec 18, 2023

ShreckYe commented Dec 19, 2023

ShreckYe commented Dec 19, 2023 •

edited

Loading

e5l commented Jan 3, 2024 •

edited

Loading

		/** Returns the specified [value] as an array query parameter. */
		fun <T> arrayParam(value: Array<T>): Expression<Array<T>> = QueryParameter(value, UntypedAndUnsizedArrayColumnType)

		class AllAnyOp<T>(val opName: String, val array: Array<T>) : Op<T>() {

		override fun toQueryBuilder(queryBuilder: QueryBuilder) = queryBuilder {

		/** Returns this subquery wrapped in the `ANY` operator. This function is not supported by the SQLite dialect. */
		fun <T> anyFrom(subQuery: Query): Op<T> = AllAnyFromSubQueryOp(true, subQuery)

		/** Returns this table wrapped in the `ANY` operator. This function is only supported by PostgreSQL and H2 dialects. */
		fun <T> anyFrom(table: Table): Op<T> = AllAnyFromTableOp(true, table)

		/** Checks if this expression is equal to any element from the column of [table] with only a single column. This function is only supported by PostgreSQL and H2 dialects. */
		infix fun <T> ExpressionWithColumnType<T>.notInTable(table: Table): InTableOp = InTableOp(this, table, false)

feat: Add ALL and ANY operators accepting array, subquery, or table parameters #1886

feat: Add ALL and ANY operators accepting array, subquery, or table parameters #1886

Conversation

ShreckYe commented Nov 16, 2023

e5l commented Dec 6, 2023

e5l commented Dec 6, 2023

bog-walk left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreckYe Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bog-walk Dec 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreckYe Dec 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreckYe commented Dec 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreckYe Dec 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bog-walk left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreckYe commented Dec 19, 2023

ShreckYe commented Dec 19, 2023 • edited Loading

e5l commented Jan 3, 2024 • edited Loading

feat: Add `ALL` and `ANY` operators accepting array, subquery, or table parameters #1886

feat: Add `ALL` and `ANY` operators accepting array, subquery, or table parameters #1886

bog-walk left a comment •

edited

Loading

ShreckYe Dec 19, 2023 •

edited

Loading

bog-walk Dec 6, 2023 •

edited

Loading

ShreckYe Dec 18, 2023 •

edited

Loading

ShreckYe Dec 18, 2023 •

edited

Loading

bog-walk left a comment •

edited

Loading

ShreckYe commented Dec 19, 2023 •

edited

Loading

e5l commented Jan 3, 2024 •

edited

Loading