-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-12926][SQL] SQLContext to display warning message when non-sql configs are being set #10849
Conversation
Thanks for working this! I'm worried that throwing an exception could break existing applications (i.e. I know for a fact that spark-sql-perf uses the conf for its own options). What about changing this to a warning? |
Testing: ``` scala> sqlContext.sql("SET spark.shuffle.memoryFraction=0.4") 16/01/21 08:26:47 WARN SQLConf: Attempt to set non-Spark SQL config in SQLConf: key = spark.shuffle.memoryFraction, value = 0.4 res2: org.apache.spark.sql.DataFrame = [key: string, value: string] scala> sqlContext.getConf("spark.shuffle.memoryFraction") res3: String = 0.4 ```
@marmbrus : Makes sense. I have updated the diff with your suggestion. |
ping !! |
ok to test |
Test build #50008 has finished for PR 10849 at commit
|
Fixed scala style test |
Test build #50082 has finished for PR 10849 at commit
|
ping !! |
Please update the title and description (these become the commit message when merging). |
@marmbrus : Updated the title and description. |
Thanks, merging to master. |
Thanks, merged to master. |
Users unknowingly try to set core Spark configs in SQLContext but later realise that it didn't work. eg. sqlContext.sql("SET spark.shuffle.memoryFraction=0.4"). This PR adds a warning message when such operations are done.