New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-29530][SQL][2.4] Make SQLConf in SQL parse process thread safe #26240
Conversation
@gatorsmile Open pr for branch-2.4. |
ok to test |
Could you please add some unit test? |
This pr is to fix problem comment in #26187 (comment) |
Test build #112591 has finished for PR 26240 at commit
|
@gatorsmile @HyukjinKwon @uncleGen
|
Test build #112601 has finished for PR 26240 at commit
|
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM. Thank you, @AngersZhuuuu and @HyukjinKwon .
Merged to branch-2.4
### What changes were proposed in this pull request? SparkSession.sql() method parse process not under current sparksession's conf, so some configuration about parser is not valid in multi-thread situation. In this pr, we add a SQLConf parameter to AbstractSqlParser and initial it with SessionState's conf. Then for each SparkSession's parser process. It will use's it's own SessionState's SQLConf and to be thread safe ### Why are the changes needed? Fix bug ### Does this PR introduce any user-facing change? NO ### How was this patch tested? NO Closes #26240 from AngersZhuuuu/SPARK-29530-V2.4. Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
What changes were proposed in this pull request?
SparkSession.sql() method parse process not under current sparksession's conf, so some configuration about parser is not valid in multi-thread situation.
In this pr, we add a SQLConf parameter to AbstractSqlParser and initial it with SessionState's conf.
Then for each SparkSession's parser process. It will use's it's own SessionState's SQLConf and to be thread safe
Why are the changes needed?
Fix bug
Does this PR introduce any user-facing change?
NO
How was this patch tested?
NO