SPARK-8949 - Print warnings when using preferred locations feature #7874
Conversation
| @@ -121,6 +122,7 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli | |||
| @DeveloperApi | |||
| def this(config: SparkConf, preferredNodeLocationData: Map[String, Set[SplitInfo]]) = { | |||
There was a problem hiding this comment.
We should probably mark this constructor as @Deprecated as well.
|
@JoshRosen @sryza Done. |
|
jenkins, test this please |
|
Jenkins, this is ok to test. |
|
Should I rebase this to the master to solve the build fail? |
|
The tests are in an extraordinary flaky state now, so it's unlikely to be caused by your patch, especially since Jenkins tests merge commits. Let's just retest and see if the problem re-occurs. |
|
Jenkins, retest this please. |
|
Test build #40527 has finished for PR 7874 at commit
|
| @@ -118,9 +118,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli | |||
| * Can be generated using [[org.apache.spark.scheduler.InputFormatInfo.computePreferredLocations]] | |||
| * from a list of input files or InputFormats for the application. | |||
| */ | |||
| @Deprecated("Passing in preferred locations has no effect at all, see SPARK-8949") | |||
There was a problem hiding this comment.
this should be
@deprecated(("Passing in preferred locations has no effect at all, see SPARK-8949", "1.5.0")
| @@ -118,9 +118,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli | |||
| * Can be generated using [[org.apache.spark.scheduler.InputFormatInfo.computePreferredLocations]] | |||
| * from a list of input files or InputFormats for the application. | |||
| */ | |||
| @Deprecated("Passing in preferred locations has no effect at all, see SPARK-8949", "1.5.0") | |||
There was a problem hiding this comment.
This is the java.lang annotation. Does it work with Scala? I thought we used @deprecated for this -- better for consistency even if both work
There was a problem hiding this comment.
Confirmed in https://issues.apache.org/jira/browse/SPARK-10099 @darkjh do you mind updating?
There was a problem hiding this comment.
Thanks @srowen I wasn't aware of the different decorators. It's done now.
|
Test build #1667 has finished for PR 7874 at commit
|
Add warnings according to SPARK-8949 in `SparkContext` - warnings in scaladoc - log warnings when preferred locations feature is used through `SparkContext`'s constructor However I didn't found any documentation reference of this feature. Please direct me if you know any reference to this feature. Author: Han JU <ju.han.felix@gmail.com> Closes #7874 from darkjh/SPARK-8949. (cherry picked from commit 3d16a54) Signed-off-by: Sean Owen <sowen@cloudera.com>
|
Test build #41231 timed out for PR 7874 at commit |
Add warnings according to SPARK-8949 in
SparkContextSparkContext's constructorHowever I didn't found any documentation reference of this feature. Please direct me if you know any reference to this feature.