[SPARK-47102][SQL] Add the COLLATION_ENABLED config flag
#45285
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MaxGekk
requested changes
Feb 27, 2024
mihailom-db
force-pushed
the
SPARK-47102
branch
from
0a3a336
to
16b468e
Compare
dbatomic
approved these changes
Feb 27, 2024
cloud-fan
reviewed
Feb 27, 2024
cloud-fan
reviewed
Feb 27, 2024
...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala
Outdated
Show resolved
Hide resolved
cloud-fan
reviewed
Feb 27, 2024
sql/core/src/test/scala/org/apache/spark/sql/CollationSuite.scala
Outdated
Show resolved
Hide resolved
cloud-fan
reviewed
Feb 27, 2024
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala
Outdated
Show resolved
Hide resolved
mkaravel
reviewed
Feb 27, 2024
sql/core/src/test/scala/org/apache/spark/sql/CollationSuite.scala
Outdated
Show resolved
Hide resolved
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala
Outdated
Show resolved
Hide resolved
...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala
Show resolved
Hide resolved
MaxGekk
requested changes
Feb 27, 2024
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
Outdated
Show resolved
Hide resolved
...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala
Outdated
Show resolved
Hide resolved
mihailom-db
force-pushed
the
SPARK-47102
branch
from
164dd52
to
a9f2c19
Compare
cloud-fan
reviewed
Feb 28, 2024
cloud-fan
reviewed
Feb 28, 2024
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
Outdated
Show resolved
Hide resolved
cloud-fan
reviewed
Feb 28, 2024
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
Show resolved
Hide resolved
cloud-fan
reviewed
Feb 28, 2024
sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala
Show resolved
Hide resolved
cloud-fan
reviewed
Feb 28, 2024
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
Outdated
Show resolved
Hide resolved
MaxGekk
reviewed
Feb 28, 2024
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryCompilationErrorsSuite.scala
Outdated
Show resolved
Hide resolved
cloud-fan
reviewed
Mar 1, 2024
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
Outdated
Show resolved
Hide resolved
cloud-fan
reviewed
Mar 1, 2024
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
Outdated
Show resolved
Hide resolved
dbatomic
reviewed
Mar 1, 2024
...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collationExpressions.scala
Outdated
Show resolved
Hide resolved
cloud-fan
approved these changes
Mar 4, 2024
MaxGekk
reviewed
Mar 4, 2024
There was a problem hiding this comment.
introduces new error class COLLATION_SUPPORT_NOT_ENABLED to appropriately report error
@mihailom-db Could update PR's description according to your recent changes. In particular, change the name of the error class.
Also, please add Closes #45218 to PR's description to close the PR automatically when this PR be merged.
|
@mihailom-db The test failure is related to your changes: Please, fix it. |
MaxGekk
changed the title
COLLATION_ENABLED config flag
|
The failed GA Run / Build modules: pyspark-connect has been passed already a couple comments before. Highly likely, it is not related to the changes. |
MaxGekk
approved these changes
Mar 5, 2024
|
+1, LGTM. Merging to master. |
jpcorreia99
pushed a commit
to jpcorreia99/spark
that referenced
this pull request
Mar 12, 2024
### What changes were proposed in this pull request? This PR adds `COLLATION_ENABLED` config to `SQLConf` and introduces new error class `UNSUPPORTED_FEATURE.COLLATION` to appropriately report error on usage of feature under development. Closes apache#45218 ### Why are the changes needed? We want to make collations configurable on this flag. These changes disable usage of `collate` and `collation` functions, along with any `COLLATE` syntax when the flag is set to false. By default, the flag is set to false. ### Does this PR introduce _any_ user-facing change? Yes. It introduces new error along with an appropriate message. ### How was this patch tested? ``` ./build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.sql.errors.QueryCompilationErrorsSuite test ./build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.sql.catalyst.expressions.CollationExpressionSuite test ./build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.sql.CollationSuite test ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45285 from mihailom-db/SPARK-47102. Authored-by: Mihailo Milosevic <mihailo.milosevic@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds
COLLATION_ENABLEDconfig toSQLConfand introduces new error classUNSUPPORTED_FEATURE.COLLATIONto appropriately report error on usage of feature under development.Closes #45218
Why are the changes needed?
We want to make collations configurable on this flag. These changes disable usage of
collateandcollationfunctions, along with anyCOLLATEsyntax when the flag is set to false. By default, the flag is set to false.Does this PR introduce any user-facing change?
Yes. It introduces new error along with an appropriate message.
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?
No.