[SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy #27668

beliefer · 2020-02-21T12:24:44Z

What changes were proposed in this pull request?

1.Add version information to the configuration of Deploy.
2.Update the docs of Deploy.

I sorted out some information show below.

Item name	Since version	JIRA ID	Commit ID	Note
spark.deploy.recoveryMode	0.8.1	None	`d66c01f`#diff-29dffdccd5a7f4c8b496c293e87c8668
spark.deploy.recoveryMode.factory	1.2.0	SPARK-1830	`deefd9d`#diff-29dffdccd5a7f4c8b496c293e87c8668	This configuration appears in branch-1.3, but the version number in the pom.xml file corresponding to the commit is 1.2.0-SNAPSHOT
spark.deploy.recoveryDirectory	0.8.1	None	`d66c01f`#diff-29dffdccd5a7f4c8b496c293e87c8668
spark.deploy.zookeeper.url	0.8.1	None	`d66c01f`#diff-4457313ca662a1cd60197122d924585c
spark.deploy.zookeeper.dir	0.8.1	None	`d66c01f`#diff-a84228cb45c7d5bd93305a1f5bf720b6
spark.deploy.retainedApplications	0.8.0	None	`46eecd1`#diff-29dffdccd5a7f4c8b496c293e87c8668
spark.deploy.retainedDrivers	1.1.0	None	`7446f5f`#diff-29dffdccd5a7f4c8b496c293e87c8668
spark.dead.worker.persistence	0.8.0	None	`46eecd1`#diff-29dffdccd5a7f4c8b496c293e87c8668
spark.deploy.maxExecutorRetries	1.6.3	SPARK-16956	`ace458f`#diff-29dffdccd5a7f4c8b496c293e87c8668
spark.deploy.spreadOut	0.6.1	None	`bb2b9ff`#diff-0e7ae91819fc8f7b47b0f97be7116325
spark.deploy.defaultCores	0.9.0	None	`d8bcc8e`#diff-29dffdccd5a7f4c8b496c293e87c8668

Why are the changes needed?

Supplemental configuration version information.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Exists UT

beliefer · 2020-02-21T12:27:17Z

cc @HyukjinKwon I think we should also add version info for other modules.

SparkQA · 2020-02-21T12:47:56Z

Test build #118791 has finished for PR 27668 at commit 2eec92e.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2020-02-21T14:43:15Z

retest this please

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

SparkQA · 2020-02-21T17:29:35Z

Test build #118795 has finished for PR 27668 at commit 2eec92e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-22T03:20:13Z

Test build #118806 has finished for PR 27668 at commit 4e7ff54.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2020-02-22T04:05:27Z

@HyukjinKwon spark.deploy.recoveryMode.factory appears in branch-1.3, but the version number in the pom.xml file corresponding to the commit is 1.2.0-SNAPSHOT. Is the since version of this configuration 1.2.0 or 1.3.0?

SparkQA · 2020-02-22T05:40:48Z

Test build #118810 has finished for PR 27668 at commit ee3cdd2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-22T06:59:07Z

Test build #118811 has finished for PR 27668 at commit bd6416b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

HyukjinKwon

We can remove the commit logs in the comments at the codes (sorry I meant to add a Github comment on this PR for reviewers). Looks fine to me. I will take a close look and merge if it's not merged in few days.

beliefer · 2020-02-24T04:42:41Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala


 private[spark] object Deploy {
  val RECOVERY_MODE = ConfigBuilder("spark.deploy.recoveryMode")
+    .version("0.8.1")


No JIRA ID, commit ID: d66c01f

beliefer · 2020-02-24T04:43:48Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createWithDefault("NONE")

  val RECOVERY_MODE_FACTORY = ConfigBuilder("spark.deploy.recoveryMode.factory")
+    .version("1.2.0")


SPARK-1830 commit ID: deefd9d
This configuration appears in branch-1.3, but the version number in the pom.xml file corresponding to the commit is 1.2.0-SNAPSHOT

beliefer · 2020-02-24T04:45:54Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createWithDefault("")

  val RECOVERY_DIRECTORY = ConfigBuilder("spark.deploy.recoveryDirectory")
+    .version("0.8.1")


No JIRA ID, commit ID: d66c01f

beliefer · 2020-02-24T04:46:02Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

  val ZOOKEEPER_URL = ConfigBuilder("spark.deploy.zookeeper.url")
    .doc(s"When `${RECOVERY_MODE.key}` is set to ZOOKEEPER, this " +
      "configuration is used to set the zookeeper URL to connect to.")
+    .version("0.8.1")


No JIRA ID, commit ID: d66c01f

beliefer · 2020-02-24T04:46:08Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createOptional

  val ZOOKEEPER_DIRECTORY = ConfigBuilder("spark.deploy.zookeeper.dir")
+    .version("0.8.1")


No JIRA ID, commit ID: d66c01f

beliefer · 2020-02-24T04:46:16Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createOptional

  val RETAINED_APPLICATIONS = ConfigBuilder("spark.deploy.retainedApplications")
+    .version("0.8.0")


No JIRA ID, commit ID: 46eecd1

beliefer · 2020-02-24T04:46:22Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createWithDefault(200)

  val RETAINED_DRIVERS = ConfigBuilder("spark.deploy.retainedDrivers")
+    .version("1.1.0")


No JIRA ID, commit ID: 7446f5f

beliefer · 2020-02-24T04:46:29Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createWithDefault(200)

  val REAPER_ITERATIONS = ConfigBuilder("spark.dead.worker.persistence")
+    .version("0.8.0")


No JIRA ID, commit ID: 46eecd1

beliefer · 2020-02-24T04:46:35Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createWithDefault(15)

  val MAX_EXECUTOR_RETRIES = ConfigBuilder("spark.deploy.maxExecutorRetries")
+    .version("1.6.3")


SPARK-16956, commit ID: ace458f

beliefer · 2020-02-24T04:46:41Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createWithDefault(10)

  val SPREAD_OUT_APPS = ConfigBuilder("spark.deploy.spreadOut")
+    .version("0.6.1")


No JIRA ID, commit ID: bb2b9ff

beliefer · 2020-02-24T04:46:48Z

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

    .createWithDefault(true)

  val DEFAULT_CORES = ConfigBuilder("spark.deploy.defaultCores")
+    .version("0.9.0")


No JIRA ID, commit ID: d8bcc8e

beliefer · 2020-02-24T04:50:06Z

@HyukjinKwon I have removed the comments and add GitHub comments for each item.

SparkQA · 2020-02-24T07:25:12Z

Test build #118843 has finished for PR 27668 at commit 5891872.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-02-25T02:38:55Z

Merged to master.

beliefer · 2020-02-25T03:38:17Z

@HyukjinKwon Thanks for your help! @srowen Thanks for your review.

beliefer added 7 commits February 15, 2020 14:46

Add version to config

ff402f7

Support SET -v

d216d43

Optimize code

d55a88d

Optimize code

4da0b91

Version -> Since Version

6d8eb75

Optimize code

5712703

Add version info for deploy.

2eec92e

srowen reviewed Feb 21, 2020

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala Show resolved Hide resolved

Merge branch 'master' into add-version-to-deploy-config

4e7ff54

beliefer changed the title ~~[WIP][SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy~~ [SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy Feb 22, 2020

beliefer added 2 commits February 22, 2020 11:31

Add JIRA ID and commit ID.

ee3cdd2

Update comment.

bd6416b

HyukjinKwon reviewed Feb 24, 2020

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala Outdated Show resolved Hide resolved

HyukjinKwon approved these changes Feb 24, 2020

View reviewed changes

Remove comments no need

5891872

beliefer commented Feb 24, 2020

View reviewed changes

HyukjinKwon closed this in 7911de9 Feb 25, 2020

beliefer deleted the add-version-to-deploy-config branch April 23, 2024 06:38

[SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy #27668

[SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy #27668

Uh oh!

Conversation

beliefer commented Feb 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

beliefer commented Feb 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Feb 21, 2020

Uh oh!

beliefer commented Feb 21, 2020

Uh oh!

Uh oh!

SparkQA commented Feb 21, 2020

Uh oh!

SparkQA commented Feb 22, 2020

Uh oh!

beliefer commented Feb 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Feb 22, 2020

Uh oh!

SparkQA commented Feb 22, 2020

Uh oh!

Uh oh!

HyukjinKwon left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer Feb 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beliefer commented Feb 24, 2020

beliefer commented Feb 21, 2020 •

edited

Loading

beliefer commented Feb 21, 2020 •

edited

Loading

beliefer commented Feb 22, 2020 •

edited

Loading

HyukjinKwon left a comment •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading

beliefer Feb 24, 2020 •

edited

Loading