[SPARK-13487][SQL] User-facing RuntimeConfig interface #11378
Conversation
Remove SparkSession unit test fix compilation more tests and methods
|
cc @marmbrus |
|
Test build #52003 has finished for PR 11378 at commit
|
|
LGTM. Merging to master. |
| * | ||
| * @since 2.0.0 | ||
| */ | ||
| def set(key: String, value: Long): RuntimeConfig |
There was a problem hiding this comment.
We don't have getBoolean and getLong, which means we don't really store config values as Boolean or Long. So why not we implement these 2 set in the base class, i.e. turn Boolean or Long to string and call the abstract set?The get returns String anyway.
There was a problem hiding this comment.
Are there any possibilities that we will implement the boolean and long set specially? i.e. not just turn it to string and set.
There was a problem hiding this comment.
Because it is a lot more annoying to have to type
.set("key", "1.0")
than just
.set("key", 1.0)
On Thursday, February 25, 2016, Wenchen Fan notifications@github.com
wrote:
In sql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala
#11378 (comment):
- */
- def set(key: String, value: String): RuntimeConfig
- /**
- * Sets the given Spark runtime configuration property.
- * @SInCE 2.0.0
- */
- def set(key: String, value: Boolean): RuntimeConfig
- /**
- * Sets the given Spark runtime configuration property.
- * @SInCE 2.0.0
- */
- def set(key: String, value: Long): RuntimeConfig
We don't have getBoolean and getLong, which means we don't really store
config values as Boolean or Long. So why not we implement these 2 set in
the base class, i.e. turn Boolean or Long to string and call the abstract
set?The get returns String anyway.—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/11378/files#r54211858.
There was a problem hiding this comment.
yea, I understand this 2 sets are useful, but I don't understand why we mark them abstract. It's just a toString and call the basic set, we can implement it in this base class.
There was a problem hiding this comment.
ah ok.
it is just a lot cleaner here for this file to not have any implementation
## What changes were proposed in this pull request? `RuntimeConfig` is the new user-facing API in 2.0 added in #11378. Until now, however, it's been dead code. This patch uses `RuntimeConfig` in `SessionState` and exposes that through the `SparkSession`. ## How was this patch tested? New test in `SQLContextSuite`. Author: Andrew Or <andrew@databricks.com> Closes #12669 from andrewor14/use-runtime-conf.
4 participants
What changes were proposed in this pull request?
This patch creates the public API for runtime configuration and an implementation for it. The public runtime configuration includes configs for existing SQL, as well as Hadoop Configuration.
This new interface is currently dead code. It will be added to SQLContext and a session entry point to Spark when we add that.
How was this patch tested?
a new unit test suite