[SPARK-37133][SQL] Add a config to optionally enforce ANSI reserved keywords #34403
Conversation
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
Outdated
Show resolved
Hide resolved
|
Kubernetes integration test starting |
| @@ -1728,7 +1728,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg | |||
| } | |||
|
|
|||
| override def visitCurrentLike(ctx: CurrentLikeContext): Expression = withOrigin(ctx) { | |||
| if (conf.ansiEnabled) { | |||
| if (conf.ansiEnabled && conf.enforceReservedKeywords) { | |||
There was a problem hiding this comment.
We only need conf.enforceReservedKeywords since it implies ANSI is enabled.
|
Kubernetes integration test status failure |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #144663 has finished for PR 34403 at commit
|
|
Test build #144667 has finished for PR 34403 at commit
|
|
the last commit just updates the doc. I'm merging it to master, thanks for the review! |
What changes were proposed in this pull request?
This PR adds a new config to optionally enforce the ANSI reserved keywords in the parser. The default value is true, so we by default still enforce it and there is no behavior change.
Why are the changes needed?
In Spark 3.2, the ANSI mode is GA. We want more people to try and use the ANSI mode, to find data issues as early as possible and get better data quality. However, the reserved keywords thing is a big stopper for many users that want to try ANSI mode. They have to update the SQL queries to pass the parser, which is nothing about data quality but just trouble.
With a new config to allow users to not enforce reserved keywords, I think we can get better adoption of the ANSI mode.
Does this PR introduce any user-facing change?
no
How was this patch tested?
updated tests.