Skip to content

[SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy #27668

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from
Closed

[SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy #27668

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ff402f7
Add version to config
beliefer Feb 15, 2020
d216d43
Support SET -v
beliefer Feb 15, 2020
d55a88d
Optimize code
beliefer Feb 16, 2020
4da0b91
Optimize code
beliefer Feb 17, 2020
6d8eb75
Version -> Since Version
beliefer Feb 18, 2020
5712703
Optimize code
beliefer Feb 19, 2020
2eec92e
Add version info for deploy.
beliefer Feb 21, 2020
4e7ff54
Merge branch 'master' into add-version-to-deploy-config
beliefer Feb 22, 2020
ee3cdd2
Add JIRA ID and commit ID.
beliefer Feb 22, 2020
bd6416b
Update comment.
beliefer Feb 22, 2020
5891872
Remove comments no need
beliefer Feb 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions core/src/main/scala/org/apache/spark/internal/config/Deploy.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,48 +19,59 @@ package org.apache.spark.internal.config

private[spark] object Deploy {
val RECOVERY_MODE = ConfigBuilder("spark.deploy.recoveryMode")
.version("0.8.1")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: d66c01f

.stringConf
.createWithDefault("NONE")

val RECOVERY_MODE_FACTORY = ConfigBuilder("spark.deploy.recoveryMode.factory")
.version("1.2.0")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-1830 commit ID: deefd9d
This configuration appears in branch-1.3, but the version number in the pom.xml file corresponding to the commit is 1.2.0-SNAPSHOT

.stringConf
.createWithDefault("")

val RECOVERY_DIRECTORY = ConfigBuilder("spark.deploy.recoveryDirectory")
.version("0.8.1")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: d66c01f

.stringConf
.createWithDefault("")

val ZOOKEEPER_URL = ConfigBuilder("spark.deploy.zookeeper.url")
.doc(s"When `${RECOVERY_MODE.key}` is set to ZOOKEEPER, this " +
"configuration is used to set the zookeeper URL to connect to.")
.version("0.8.1")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: d66c01f

.stringConf
.createOptional

val ZOOKEEPER_DIRECTORY = ConfigBuilder("spark.deploy.zookeeper.dir")
.version("0.8.1")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: d66c01f

.stringConf
.createOptional

val RETAINED_APPLICATIONS = ConfigBuilder("spark.deploy.retainedApplications")
.version("0.8.0")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: 46eecd1

.intConf
.createWithDefault(200)

val RETAINED_DRIVERS = ConfigBuilder("spark.deploy.retainedDrivers")
.version("1.1.0")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: 7446f5f

.intConf
.createWithDefault(200)

val REAPER_ITERATIONS = ConfigBuilder("spark.dead.worker.persistence")
.version("0.8.0")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: 46eecd1

.intConf
.createWithDefault(15)

val MAX_EXECUTOR_RETRIES = ConfigBuilder("spark.deploy.maxExecutorRetries")
.version("1.6.3")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-16956, commit ID: ace458f

.intConf
.createWithDefault(10)

val SPREAD_OUT_APPS = ConfigBuilder("spark.deploy.spreadOut")
.version("0.6.1")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: bb2b9ff

.booleanConf
.createWithDefault(true)

val DEFAULT_CORES = ConfigBuilder("spark.deploy.defaultCores")
.version("0.9.0")
Copy link
Contributor Author

@beliefer beliefer Feb 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No JIRA ID, commit ID: d8bcc8e

.intConf
.createWithDefault(Int.MaxValue)

Expand Down
5 changes: 4 additions & 1 deletion docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2597,22 +2597,25 @@ Spark subsystems.
### Deploy

<table class="table">
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.deploy.recoveryMode</code></td>
<td>NONE</td>
<td>The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches.
This is only applicable for cluster mode when running with Standalone or Mesos.</td>
<td>0.8.1</td>
</tr>
<tr>
<td><code>spark.deploy.zookeeper.url</code></td>
<td>None</td>
<td>When `spark.deploy.recoveryMode` is set to ZOOKEEPER, this configuration is used to set the zookeeper URL to connect to.</td>
<td>0.8.1</td>
</tr>
<tr>
<td><code>spark.deploy.zookeeper.dir</code></td>
<td>None</td>
<td>When `spark.deploy.recoveryMode` is set to ZOOKEEPER, this configuration is used to set the zookeeper directory to store recovery state.</td>
<td>0.8.1</td>
</tr>
</table>

Expand Down