[SPARK-31269][DOC] Supplement version for configuration only appear in configuration doc#28035
Closed
beliefer wants to merge 2 commits intoapache:masterfrom
Closed
[SPARK-31269][DOC] Supplement version for configuration only appear in configuration doc#28035beliefer wants to merge 2 commits intoapache:masterfrom
beliefer wants to merge 2 commits intoapache:masterfrom
Conversation
beliefer
commented
Mar 26, 2020
| <td> | ||
| The name of your application. This will appear in the UI and in log data. | ||
| </td> | ||
| <td>0.9.0</td> |
Contributor
Author
There was a problem hiding this comment.
No JIRA ID, commit ID: 994f080#diff-529fc5c06b9731c1fbda6f3db60b16aa
beliefer
commented
Mar 26, 2020
| <code>spark.driver.resource.{resourceName}.discoveryScript</code> | ||
| for the driver to find the resource on startup. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-27760, commit ID: d30284b#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 26, 2020
| name and an array of addresses. For a client-submitted driver, discovery script must assign | ||
| different resource addresses to this driver comparing to other drivers on the same host. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-27488, commit ID: 74e5e41#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 26, 2020
| the Kubernetes device plugin naming convention. (e.g. For GPUs on Kubernetes | ||
| this config would be set to nvidia.com or amd.com) | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-27362, commit ID: 1277f8f#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 26, 2020
| <code>spark.executor.resource.{resourceName}.discoveryScript</code> | ||
| for the executor to find the resource on startup. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-27760, commit ID: d30284b#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 26, 2020
| write to STDOUT a JSON string in the format of the ResourceInformation class. This has a | ||
| name and an array of addresses. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-27024, commit ID: db2e3c4#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 26, 2020
| the Kubernetes device plugin naming convention. (e.g. For GPUs on Kubernetes | ||
| this config would be set to nvidia.com or amd.com) | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-27362, commit ID: 1277f8f#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 26, 2020
| <em>Note:</em> This will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or | ||
| LOCAL_DIRS (YARN) environment variables set by the cluster manager. | ||
| </td> | ||
| <td>0.5.0</td> |
Contributor
Author
There was a problem hiding this comment.
No JIRA ID, commit ID: 0e93891#diff-17fd275d280b667722664ed833c6402a
beliefer
commented
Mar 26, 2020
| <td> | ||
| Logs the effective SparkConf as INFO when a SparkContext is started. | ||
| </td> | ||
| <td>0.9.0</td> |
Contributor
Author
There was a problem hiding this comment.
No JIRA ID, commit ID: d8bcc8e#diff-364713d7776956cb8b0a771e9b62f82d
beliefer
commented
Mar 26, 2020
| The cluster manager to connect to. See the list of | ||
| <a href="submitting-applications.html#master-urls"> allowed master URL's</a>. | ||
| </td> | ||
| <td>0.9.0</td> |
Contributor
Author
beliefer
commented
Mar 26, 2020
| Instead, please set this through the <code>--driver-java-options</code> command line option or in | ||
| your default properties file. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-23472, commit ID: f830005#diff-a78ecfc6a89edfaf0b60a5eaa0381970
beliefer
commented
Mar 26, 2020
| verbose gc logging to a file named for the executor ID of the app in /tmp, pass a 'value' of: | ||
| <code>-verbose:gc -Xloggc:/tmp/{{APP_ID}}-{{EXECUTOR_ID}}.gc</code> | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-23472, commit ID: f830005#diff-a78ecfc6a89edfaf0b60a5eaa0381970
beliefer
commented
Mar 26, 2020
| Add the environment variable specified by <code>EnvironmentVariableName</code> to the Executor | ||
| process. The user can specify multiple of these to set multiple environment variables. | ||
| </td> | ||
| <td>0.9.0</td> |
Contributor
Author
There was a problem hiding this comment.
No JIRA ID, commit ID: 642029e#diff-529fc5c06b9731c1fbda6f3db60b16aa
beliefer
commented
Mar 26, 2020
| passing a profiler class in as a parameter to the <code>SparkContext</code> constructor. | ||
| </td> | ||
| <td></td> | ||
| <td>1.2.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-3478, commit ID: 1aa549b#diff-d6fe2792e44f6babc94aabfefc8b9bce
beliefer
commented
Mar 26, 2020
| by <code>pstats.Stats()</code>. If this is specified, the profile result will not be displayed | ||
| automatically. | ||
| </td> | ||
| <td>1.2.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-3478, commit ID: 1aa549b#diff-d6fe2792e44f6babc94aabfefc8b9bce
beliefer
commented
Mar 26, 2020
| (e.g. <code>512m</code>, <code>2g</code>). | ||
| If the memory used during aggregation goes above this amount, it will spill the data into disks. | ||
| </td> | ||
| <td>1.1.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-2538, commit ID: 14174ab#diff-d6fe2792e44f6babc94aabfefc8b9bce
beliefer
commented
Mar 26, 2020
| repositories given by the command-line option <code>--repositories</code>. For more details, see | ||
| <a href="submitting-applications.html#advanced-dependency-management">Advanced Dependency Management</a>. | ||
| </td> | ||
| <td>1.5.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-9263, commit ID: 3433571#diff-63a5d817d2d45ae24de577f6a1bd80f9
beliefer
commented
Mar 26, 2020
| Comma-separated list of groupId:artifactId, to exclude while resolving the dependencies | ||
| provided in <code>spark.jars.packages</code> to avoid dependency conflicts. | ||
| </td> | ||
| <td>1.5.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-9263, commit ID: 3433571#diff-63a5d817d2d45ae24de577f6a1bd80f9
beliefer
commented
Mar 26, 2020
| <code>spark.jars.packages</code>. This will override the Ivy property <code>ivy.default.ivy.user.dir</code> | ||
| which defaults to ~/.ivy2. | ||
| </td> | ||
| <td>1.3.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-5341, commit ID: 3b7acd2#diff-63a5d817d2d45ae24de577f6a1bd80f9
beliefer
commented
Mar 26, 2020
| artifact server like Artifactory. Details on the settings file format can be | ||
| found at <a href="http://ant.apache.org/ivy/history/latest-milestone/settings.html">Settings Files</a> | ||
| </td> | ||
| <td>2.2.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-17568, commit ID: 3bc2eff#diff-4d2ab44195558d5a9d5f15b8803ef39d
beliefer
commented
Mar 26, 2020
| Comma-separated list of additional remote repositories to search for the maven coordinates | ||
| given with <code>--packages</code> or <code>spark.jars.packages</code>. | ||
| </td> | ||
| <td>2.3.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-21403, commit ID: d8257b9#diff-4d2ab44195558d5a9d5f15b8803ef39d
beliefer
commented
Mar 26, 2020
| set to a non-zero value. This retry logic helps stabilize large shuffles in the face of long GC | ||
| pauses or transient network connectivity issues. | ||
| </td> | ||
| <td>1.2.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-4188, commit ID: c1ea5c5#diff-d2ce9b38bdc38ca9d7119f9c2cf79907
beliefer
commented
Mar 26, 2020
| large clusters. For clusters with many hard disks and few hosts, this may result in insufficient | ||
| concurrency to saturate all disks, and so users may consider increasing this value. | ||
| </td> | ||
| <td>1.2.1</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-4740, commit ID: 441ec34#diff-d2ce9b38bdc38ca9d7119f9c2cf79907
beliefer
commented
Mar 26, 2020
| block transfer. For environments where off-heap memory is tightly limited, users may wish to | ||
| turn this off to force all allocations from Netty to be on-heap. | ||
| </td> | ||
| <td>1.2.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-4188, commit ID: c1ea5c5#diff-d2ce9b38bdc38ca9d7119f9c2cf79907
beliefer
commented
Mar 26, 2020
| (Netty only) How long to wait between retries of fetches. The maximum delay caused by retrying | ||
| is 15 seconds by default, calculated as <code>maxRetries * retryWait</code>. | ||
| </td> | ||
| <td>1.2.1</td> |
Contributor
Author
There was a problem hiding this comment.
No JIRA ID, commit ID: 5e5d8f4#diff-d2ce9b38bdc38ca9d7119f9c2cf79907
|
Test build #120421 has finished for PR 28035 at commit
|
beliefer
commented
Mar 26, 2020
| application (see <code>spark.shuffle.service.enabled</code> option below). If set below 1, | ||
| will fallback to OS default defined by Netty's <code>io.netty.util.NetUtil#SOMAXCONN</code>. | ||
| </td> | ||
| <td>1.1.1</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-2468, commit ID: 66b4c81#diff-bdee8e601924d41e93baa7287189e878
beliefer
commented
Mar 26, 2020
| <td> | ||
| Cache entries limited to the specified memory footprint, in bytes unless otherwise specified. | ||
| </td> | ||
| <td>2.3.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-21501, commit ID: 1662e93#diff-97d5edc927a83a678e013ae00343df94
beliefer
commented
Mar 26, 2020
| disabled in order to use Spark local directories that reside on NFS filesystems (see | ||
| <a href="https://issues.apache.org/jira/browse/SPARK-6313">SPARK-6313</a> for more details). | ||
| </td> | ||
| <td>1.2.2</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-6313, commit ID: a2a94a1#diff-d239aee594001f8391676e1047a0381e
beliefer
commented
Mar 26, 2020
| Whether to overwrite files added through SparkContext.addFile() when the target file exists and | ||
| its contents do not match those of the source. | ||
| </td> | ||
| <td>1.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
No JIRA ID, commit ID: 84670f2#diff-d239aee594001f8391676e1047a0381e
beliefer
commented
Mar 26, 2020
| are not affected by these issues.</td> | ||
| are not affected by these issues. | ||
| </td> | ||
| <td>1.0.3</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-2546, commit ID: 6d8f1dd#diff-83eb37f7b0ebed3c14ccb7bff0d577c2
beliefer
commented
Mar 26, 2020
| This setting is ignored for jobs generated through Spark Streaming's StreamingContext, since data may | ||
| need to be rewritten to pre-existing output directories during checkpoint recovery. | ||
| </td> | ||
| <td>1.0.1</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-1677, commit ID: 8100cbd#diff-f70e97c099b5eac05c75288cb215e080
beliefer
commented
Mar 26, 2020
| Version 2 may have better performance, but version 1 may handle failures better in certain situations, | ||
| as per <a href="https://issues.apache.org/jira/browse/MAPREDUCE-4815">MAPREDUCE-4815</a>. | ||
| </td> | ||
| <td>2.2.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-20107, commit ID: edc87d7#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 26, 2020
| connections arrives in a short period of time. | ||
| </td> | ||
| <td></td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-27868, commit ID: 09ed64d#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 26, 2020
| turn this off to force all allocations to be on-heap. | ||
| </td> | ||
| <td></td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-24920, commit ID: e103c4a#diff-0ac65da2bc6b083fb861fe410c7688c2
beliefer
commented
Mar 26, 2020
| to port + maxRetries. | ||
| </td> | ||
| <td></td> | ||
| <td>1.1.1</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-3565, commit ID: 32f2222#diff-d239aee594001f8391676e1047a0381e
beliefer
commented
Mar 26, 2020
| you can set larger value. | ||
| </td> | ||
| <td></td> | ||
| <td>1.1.1</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-2677, commit ID: bd3ce2f#diff-f748e95f2aa97ed715afa53ddeeac9de
beliefer
commented
Mar 26, 2020
| that register to the listener bus. Consider increasing value, if the listener events corresponding | ||
| to shared queue are dropped. Increasing this value may result in the driver using more memory. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-28574, commit ID: c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509
beliefer
commented
Mar 26, 2020
| Consider increasing value, if the listener events corresponding to appStatus queue are dropped. | ||
| Increasing this value may result in the driver using more memory. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-28574, commit ID: c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509
beliefer
commented
Mar 26, 2020
| executor management listeners. Consider increasing value if the listener events corresponding to | ||
| executorManagement queue are dropped. Increasing this value may result in the driver using more memory. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-28574, commit ID: c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509
beliefer
commented
Mar 27, 2020
| that write events to eventLogs. Consider increasing value if the listener events corresponding to eventLog queue | ||
| are dropped. Increasing this value may result in the driver using more memory. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-28574, commit ID: c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509
beliefer
commented
Mar 27, 2020
| Consider increasing value if the listener events corresponding to streams queue are dropped. Increasing | ||
| this value may result in the driver using more memory. | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-28574, commit ID: c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509
beliefer
commented
Mar 27, 2020
| in order to assign resource slots (e.g. a 0.2222 configuration, or 1/0.2222 slots will become | ||
| 4 tasks/resource, not 5). | ||
| </td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-27760, commit ID: d30284b#diff-76e731333fb756df3bff5ddb3b731c46
beliefer
commented
Mar 27, 2020
| <td> | ||
| Number of consecutive stage attempts allowed before a stage is aborted. | ||
| </td> | ||
| <td>2.2.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-13369, commit ID: 7b5d873#diff-6a9ff7fb74fd490a50462d45db2d5e11
beliefer
commented
Mar 27, 2020
| Fall back on <code>spark.rpc.io.serverThreads</code> | ||
| </td> | ||
| <td>Number of threads used in the server thread pool</td> | ||
| <td>1.6.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-10745, commit ID: 7c5b641#diff-d2ce9b38bdc38ca9d7119f9c2cf79907
beliefer
commented
Mar 27, 2020
| Fall back on <code>spark.rpc.io.clientThreads</code> | ||
| </td> | ||
| <td>Number of threads used in the client thread pool</td> | ||
| <td>1.6.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-10745, commit ID: 7c5b641#diff-d2ce9b38bdc38ca9d7119f9c2cf79907
beliefer
commented
Mar 27, 2020
| Fall back on <code>spark.rpc.netty.dispatcher.numThreads</code> | ||
| </td> | ||
| <td>Number of threads used in RPC message dispatcher thread pool</td> | ||
| <td>3.0.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-29398, commit ID: 2f0a38c#diff-a68a21481fea5053848ca666dd3201d8
beliefer
commented
Mar 27, 2020
| Executable for executing R scripts in client modes for driver. Ignored in cluster modes. | ||
| </td> | ||
| <td></td> | ||
| <td>1.5.3</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-10971, commit ID: 9695f45#diff-025470e1b7094d7cf4a78ea353fb3981
beliefer
commented
Mar 27, 2020
| <code>spark.r.shell.command</code> is used for sparkR shell while <code>spark.r.driver.command</code> is used for running R script. | ||
| </td> | ||
| <td></td> | ||
| <td>2.1.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-17178, commit ID: fa63479#diff-a78ecfc6a89edfaf0b60a5eaa0381970
beliefer
commented
Mar 27, 2020
| Checkpoint interval for graph and message in Pregel. It used to avoid stackOverflowError due to long lineage chains | ||
| after lots of iterations. The checkpoint is disabled by default. | ||
| </td> | ||
| <td>2.2.0</td> |
Contributor
Author
There was a problem hiding this comment.
SPARK-5484, commit ID: f971ce5#diff-e399679417ffa6eeedf26a7630baca16
HyukjinKwon
approved these changes
Mar 31, 2020
Member
|
Merged to master. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Apr 7, 2020
…n configuration doc ### What changes were proposed in this pull request? The `configuration.md` exists some config not organized by `ConfigEntry`. This PR supplements version for configuration only appear in configuration doc. I sorted out some information show below. Item name | Since version | JIRA ID | Commit ID | Note -- | -- | -- | -- | -- spark.app.name | 0.9.0 | None | 994f080#diff-529fc5c06b9731c1fbda6f3db60b16aa | spark.driver.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b#diff-76e731333fb756df3bff5ddb3b731c46 | spark.driver.resource.{resourceName}.discoveryScript | 3.0.0 | SPARK-27488 | 74e5e41#diff-76e731333fb756df3bff5ddb3b731c46 | spark.driver.resource.{resourceName}.vendor | 3.0.0 | SPARK-27362 | 1277f8f#diff-76e731333fb756df3bff5ddb3b731c46 | spark.executor.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b#diff-76e731333fb756df3bff5ddb3b731c46 | spark.executor.resource.{resourceType}.discoveryScript | 3.0.0 | SPARK-27024 | db2e3c4#diff-76e731333fb756df3bff5ddb3b731c46 | spark.executor.resource.{resourceName}.vendor | 3.0.0 | SPARK-27362 | 1277f8f#diff-76e731333fb756df3bff5ddb3b731c46 | spark.local.dir | 0.5.0 | None | 0e93891#diff-17fd275d280b667722664ed833c6402a | spark.logConf | 0.9.0 | None | d8bcc8e#diff-364713d7776956cb8b0a771e9b62f82d | spark.master | 0.9.0 | SPARK-544 | 2573add#diff-529fc5c06b9731c1fbda6f3db60b16aa | spark.driver.defaultJavaOptions | 3.0.0 | SPARK-23472 | f830005#diff-a78ecfc6a89edfaf0b60a5eaa0381970 | spark.executor.defaultJavaOptions | 3.0.0 | SPARK-23472 | f830005#diff-a78ecfc6a89edfaf0b60a5eaa0381970 | spark.executorEnv.[EnvironmentVariableName] | 0.9.0 | None | 642029e#diff-529fc5c06b9731c1fbda6f3db60b16aa | spark.python.profile | 1.2.0 | SPARK-3478 | 1aa549b#diff-d6fe2792e44f6babc94aabfefc8b9bce | spark.python.profile.dump | 1.2.0 | SPARK-3478 | 1aa549b#diff-d6fe2792e44f6babc94aabfefc8b9bce | spark.python.worker.memory | 1.1.0 | SPARK-2538 | 14174ab#diff-d6fe2792e44f6babc94aabfefc8b9bce | spark.jars.packages | 1.5.0 | SPARK-9263 | 3433571#diff-63a5d817d2d45ae24de577f6a1bd80f9 | spark.jars.excludes | 1.5.0 | SPARK-9263 | 3433571#diff-63a5d817d2d45ae24de577f6a1bd80f9 | spark.jars.ivy | 1.3.0 | SPARK-5341 | 3b7acd2#diff-63a5d817d2d45ae24de577f6a1bd80f9 | spark.jars.ivySettings | 2.2.0 | SPARK-17568 | 3bc2eff#diff-4d2ab44195558d5a9d5f15b8803ef39d | spark.jars.repositories | 2.3.0 | SPARK-21403 | d8257b9#diff-4d2ab44195558d5a9d5f15b8803ef39d | spark.shuffle.io.maxRetries | 1.2.0 | SPARK-4188 | c1ea5c5#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.shuffle.io.numConnectionsPerPeer | 1.2.1 | SPARK-4740 | 441ec34#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.shuffle.io.preferDirectBufs | 1.2.0 | SPARK-4188 | c1ea5c5#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.shuffle.io.retryWait | 1.2.1 | None | 5e5d8f4#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.shuffle.io.backLog | 1.1.1 | SPARK-2468 | 66b4c81#diff-bdee8e601924d41e93baa7287189e878 | spark.shuffle.service.index.cache.size | 2.3.0 | SPARK-21501 | 1662e93#diff-97d5edc927a83a678e013ae00343df94 | spark.shuffle.maxChunksBeingTransferred | 2.3.0 | SPARK-21175 | 799e131#diff-0ac65da2bc6b083fb861fe410c7688c2 | spark.sql.ui.retainedExecutions | 1.5.0 | SPARK-8861 and SPARK-8862 | ebc3aad#diff-81764e4d52817f83bdd5336ef1226bd9 | spark.streaming.ui.retainedBatches | 1.0.0 | SPARK-1386 | f36dc3f#diff-56b8d67d07284cfab165d5363bd3500e | spark.default.parallelism | 0.5.0 | None | e5c4cd8#diff-0544ebf7533fa70ff5103e0fe1f0b036 | spark.files.fetchTimeout | 1.0.0 | None | f6f9d02#diff-d239aee594001f8391676e1047a0381e | spark.files.useFetchCache | 1.2.2 | SPARK-6313 | a2a94a1#diff-d239aee594001f8391676e1047a0381e | spark.files.overwrite | 1.0.0 | None | 84670f2#diff-d239aee594001f8391676e1047a0381e | Exists in branch-1.0, but the version of pom is 0.9.0-incubating-SNAPSHOT spark.hadoop.cloneConf | 1.0.3 | SPARK-2546 | 6d8f1dd#diff-83eb37f7b0ebed3c14ccb7bff0d577c2 | spark.hadoop.validateOutputSpecs | 1.0.1 | SPARK-1677 | 8100cbd#diff-f70e97c099b5eac05c75288cb215e080 | spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version | 2.2.0 | SPARK-20107 | edc87d7#diff-76e731333fb756df3bff5ddb3b731c46 | spark.rpc.io.backLog | 3.0.0 | SPARK-27868 | 09ed64d#diff-76e731333fb756df3bff5ddb3b731c46 | spark.network.io.preferDirectBufs | 3.0.0 | SPARK-24920 | e103c4a#diff-0ac65da2bc6b083fb861fe410c7688c2 | spark.port.maxRetries | 1.1.1 | SPARK-3565 | 32f2222#diff-d239aee594001f8391676e1047a0381e | spark.core.connection.ack.wait.timeout | 1.1.1 | SPARK-2677 | bd3ce2f#diff-f748e95f2aa97ed715afa53ddeeac9de | spark.scheduler.listenerbus.eventqueue.shared.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.scheduler.listenerbus.eventqueue.appStatus.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.scheduler.listenerbus.eventqueue.executorManagement.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.scheduler.listenerbus.eventqueue.eventLog.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.scheduler.listenerbus.eventqueue.streams.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.task.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b#diff-76e731333fb756df3bff5ddb3b731c46 | spark.stage.maxConsecutiveAttempts | 2.2.0 | SPARK-13369 | 7b5d873#diff-6a9ff7fb74fd490a50462d45db2d5e11 | spark.{driver\|executor}.rpc.io.serverThreads | 1.6.0 | SPARK-10745 | 7c5b641#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.{driver\|executor}.rpc.io.clientThreads | 1.6.0 | SPARK-10745 | 7c5b641#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.{driver\|executor}.rpc.netty.dispatcher.numThreads | 3.0.0 | SPARK-29398 | 2f0a38c#diff-a68a21481fea5053848ca666dd3201d8 | spark.r.driver.command | 1.5.3 | SPARK-10971 | 9695f45#diff-025470e1b7094d7cf4a78ea353fb3981 | spark.r.shell.command | 2.1.0 | SPARK-17178 | fa63479#diff-a78ecfc6a89edfaf0b60a5eaa0381970 | spark.graphx.pregel.checkpointInterval | 2.2.0 | SPARK-5484 | f971ce5#diff-e399679417ffa6eeedf26a7630baca16 | ### Why are the changes needed? Supplemental configuration version information. ### Does this PR introduce any user-facing change? 'No'. ### How was this patch tested? Jenkins test Closes #28035 from beliefer/supplement-configuration-version. Authored-by: beliefer <beliefer@163.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
Member
|
Merged to branch-3.0 too. |
sjincho
pushed a commit
to sjincho/spark
that referenced
this pull request
Apr 15, 2020
…n configuration doc ### What changes were proposed in this pull request? The `configuration.md` exists some config not organized by `ConfigEntry`. This PR supplements version for configuration only appear in configuration doc. I sorted out some information show below. Item name | Since version | JIRA ID | Commit ID | Note -- | -- | -- | -- | -- spark.app.name | 0.9.0 | None | 994f080#diff-529fc5c06b9731c1fbda6f3db60b16aa | spark.driver.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b#diff-76e731333fb756df3bff5ddb3b731c46 | spark.driver.resource.{resourceName}.discoveryScript | 3.0.0 | SPARK-27488 | 74e5e41#diff-76e731333fb756df3bff5ddb3b731c46 | spark.driver.resource.{resourceName}.vendor | 3.0.0 | SPARK-27362 | 1277f8f#diff-76e731333fb756df3bff5ddb3b731c46 | spark.executor.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b#diff-76e731333fb756df3bff5ddb3b731c46 | spark.executor.resource.{resourceType}.discoveryScript | 3.0.0 | SPARK-27024 | db2e3c4#diff-76e731333fb756df3bff5ddb3b731c46 | spark.executor.resource.{resourceName}.vendor | 3.0.0 | SPARK-27362 | 1277f8f#diff-76e731333fb756df3bff5ddb3b731c46 | spark.local.dir | 0.5.0 | None | 0e93891#diff-17fd275d280b667722664ed833c6402a | spark.logConf | 0.9.0 | None | d8bcc8e#diff-364713d7776956cb8b0a771e9b62f82d | spark.master | 0.9.0 | SPARK-544 | 2573add#diff-529fc5c06b9731c1fbda6f3db60b16aa | spark.driver.defaultJavaOptions | 3.0.0 | SPARK-23472 | f830005#diff-a78ecfc6a89edfaf0b60a5eaa0381970 | spark.executor.defaultJavaOptions | 3.0.0 | SPARK-23472 | f830005#diff-a78ecfc6a89edfaf0b60a5eaa0381970 | spark.executorEnv.[EnvironmentVariableName] | 0.9.0 | None | 642029e#diff-529fc5c06b9731c1fbda6f3db60b16aa | spark.python.profile | 1.2.0 | SPARK-3478 | 1aa549b#diff-d6fe2792e44f6babc94aabfefc8b9bce | spark.python.profile.dump | 1.2.0 | SPARK-3478 | 1aa549b#diff-d6fe2792e44f6babc94aabfefc8b9bce | spark.python.worker.memory | 1.1.0 | SPARK-2538 | 14174ab#diff-d6fe2792e44f6babc94aabfefc8b9bce | spark.jars.packages | 1.5.0 | SPARK-9263 | 3433571#diff-63a5d817d2d45ae24de577f6a1bd80f9 | spark.jars.excludes | 1.5.0 | SPARK-9263 | 3433571#diff-63a5d817d2d45ae24de577f6a1bd80f9 | spark.jars.ivy | 1.3.0 | SPARK-5341 | 3b7acd2#diff-63a5d817d2d45ae24de577f6a1bd80f9 | spark.jars.ivySettings | 2.2.0 | SPARK-17568 | 3bc2eff#diff-4d2ab44195558d5a9d5f15b8803ef39d | spark.jars.repositories | 2.3.0 | SPARK-21403 | d8257b9#diff-4d2ab44195558d5a9d5f15b8803ef39d | spark.shuffle.io.maxRetries | 1.2.0 | SPARK-4188 | c1ea5c5#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.shuffle.io.numConnectionsPerPeer | 1.2.1 | SPARK-4740 | 441ec34#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.shuffle.io.preferDirectBufs | 1.2.0 | SPARK-4188 | c1ea5c5#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.shuffle.io.retryWait | 1.2.1 | None | 5e5d8f4#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.shuffle.io.backLog | 1.1.1 | SPARK-2468 | 66b4c81#diff-bdee8e601924d41e93baa7287189e878 | spark.shuffle.service.index.cache.size | 2.3.0 | SPARK-21501 | 1662e93#diff-97d5edc927a83a678e013ae00343df94 | spark.shuffle.maxChunksBeingTransferred | 2.3.0 | SPARK-21175 | 799e131#diff-0ac65da2bc6b083fb861fe410c7688c2 | spark.sql.ui.retainedExecutions | 1.5.0 | SPARK-8861 and SPARK-8862 | ebc3aad#diff-81764e4d52817f83bdd5336ef1226bd9 | spark.streaming.ui.retainedBatches | 1.0.0 | SPARK-1386 | f36dc3f#diff-56b8d67d07284cfab165d5363bd3500e | spark.default.parallelism | 0.5.0 | None | e5c4cd8#diff-0544ebf7533fa70ff5103e0fe1f0b036 | spark.files.fetchTimeout | 1.0.0 | None | f6f9d02#diff-d239aee594001f8391676e1047a0381e | spark.files.useFetchCache | 1.2.2 | SPARK-6313 | a2a94a1#diff-d239aee594001f8391676e1047a0381e | spark.files.overwrite | 1.0.0 | None | 84670f2#diff-d239aee594001f8391676e1047a0381e | Exists in branch-1.0, but the version of pom is 0.9.0-incubating-SNAPSHOT spark.hadoop.cloneConf | 1.0.3 | SPARK-2546 | 6d8f1dd#diff-83eb37f7b0ebed3c14ccb7bff0d577c2 | spark.hadoop.validateOutputSpecs | 1.0.1 | SPARK-1677 | 8100cbd#diff-f70e97c099b5eac05c75288cb215e080 | spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version | 2.2.0 | SPARK-20107 | edc87d7#diff-76e731333fb756df3bff5ddb3b731c46 | spark.rpc.io.backLog | 3.0.0 | SPARK-27868 | 09ed64d#diff-76e731333fb756df3bff5ddb3b731c46 | spark.network.io.preferDirectBufs | 3.0.0 | SPARK-24920 | e103c4a#diff-0ac65da2bc6b083fb861fe410c7688c2 | spark.port.maxRetries | 1.1.1 | SPARK-3565 | 32f2222#diff-d239aee594001f8391676e1047a0381e | spark.core.connection.ack.wait.timeout | 1.1.1 | SPARK-2677 | bd3ce2f#diff-f748e95f2aa97ed715afa53ddeeac9de | spark.scheduler.listenerbus.eventqueue.shared.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.scheduler.listenerbus.eventqueue.appStatus.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.scheduler.listenerbus.eventqueue.executorManagement.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.scheduler.listenerbus.eventqueue.eventLog.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.scheduler.listenerbus.eventqueue.streams.capacity | 3.0.0 | SPARK-28574 | c212c9d#diff-eb519ad78cc3cf0b95839cc37413b509 | spark.task.resource.{resourceName}.amount | 3.0.0 | SPARK-27760 | d30284b#diff-76e731333fb756df3bff5ddb3b731c46 | spark.stage.maxConsecutiveAttempts | 2.2.0 | SPARK-13369 | 7b5d873#diff-6a9ff7fb74fd490a50462d45db2d5e11 | spark.{driver\|executor}.rpc.io.serverThreads | 1.6.0 | SPARK-10745 | 7c5b641#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.{driver\|executor}.rpc.io.clientThreads | 1.6.0 | SPARK-10745 | 7c5b641#diff-d2ce9b38bdc38ca9d7119f9c2cf79907 | spark.{driver\|executor}.rpc.netty.dispatcher.numThreads | 3.0.0 | SPARK-29398 | 2f0a38c#diff-a68a21481fea5053848ca666dd3201d8 | spark.r.driver.command | 1.5.3 | SPARK-10971 | 9695f45#diff-025470e1b7094d7cf4a78ea353fb3981 | spark.r.shell.command | 2.1.0 | SPARK-17178 | fa63479#diff-a78ecfc6a89edfaf0b60a5eaa0381970 | spark.graphx.pregel.checkpointInterval | 2.2.0 | SPARK-5484 | f971ce5#diff-e399679417ffa6eeedf26a7630baca16 | ### Why are the changes needed? Supplemental configuration version information. ### Does this PR introduce any user-facing change? 'No'. ### How was this patch tested? Jenkins test Closes apache#28035 from beliefer/supplement-configuration-version. Authored-by: beliefer <beliefer@163.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The
configuration.mdexists some config not organized byConfigEntry.This PR supplements version for configuration only appear in configuration doc.
I sorted out some information show below.
Why are the changes needed?
Supplemental configuration version information.
Does this PR introduce any user-facing change?
'No'.
How was this patch tested?
Jenkins test