[SPARK-31118][K8S][DOC] Add version information to the configuration of K8S #27875
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
beliefer
commented
Mar 11, 2020
| @@ -33,19 +33,22 @@ private[spark] object Config extends Logging { | |||
| "to .kube/config under your home directory. If not specified then your current " + | |||
| "context is used. You can always override specific aspects of the config file " + | |||
| "provided configuration using other Spark on K8S configuration options.") | |||
| .version("3.0.0") | |||
There was a problem hiding this comment.
SPARK-25887， commit ID: c542c24#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_DRIVER_MASTER_URL = | ||
| ConfigBuilder("spark.kubernetes.driver.master") | ||
| .doc("The internal Kubernetes master (API server) address " + | ||
| "to be used for driver to request executors.") | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-30371, commit ID: f14061c#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createWithDefault(KUBERNETES_MASTER_INTERNAL_URL) | ||
|
|
||
| val KUBERNETES_NAMESPACE = | ||
| ConfigBuilder("spark.kubernetes.namespace") | ||
| .doc("The namespace that will be used for running the driver and executor pods.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -54,22 +57,26 @@ private[spark] object Config extends Logging { | |||
| .doc("Container image to use for Spark containers. Individual container types " + | |||
| "(e.g. driver or executor) can also be configured to use different images if desired, " + | |||
| "by setting the container type-specific image name.") | |||
| .version("2.3.0") | |||
There was a problem hiding this comment.
SPARK-22994, commit ID: b94debd#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val DRIVER_CONTAINER_IMAGE = | ||
| ConfigBuilder("spark.kubernetes.driver.container.image") | ||
| .doc("Container image to use for the driver.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-22807, commit ID: fb3636b#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .fallbackConf(CONTAINER_IMAGE) | ||
|
|
||
| val EXECUTOR_CONTAINER_IMAGE = | ||
| ConfigBuilder("spark.kubernetes.executor.container.image") | ||
| .doc("Container image to use for the executors.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-22807, commit ID: fb3636b#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .fallbackConf(CONTAINER_IMAGE) | ||
|
|
||
| val CONTAINER_IMAGE_PULL_POLICY = | ||
| ConfigBuilder("spark.kubernetes.container.image.pullPolicy") | ||
| .doc("Kubernetes image pull policy. Valid values are Always, Never, and IfNotPresent.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-22807, commit ID: fb3636b#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -78,6 +85,7 @@ private[spark] object Config extends Logging { | |||
| ConfigBuilder("spark.kubernetes.container.image.pullSecrets") | |||
| .doc("Comma separated list of the Kubernetes secrets used " + | |||
| "to access private image registries.") | |||
| .version("2.4.0") | |||
There was a problem hiding this comment.
SPARK-23668, commit ID: cccaaa1#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -95,24 +103,28 @@ private[spark] object Config extends Logging { | |||
| val SUBMISSION_CLIENT_REQUEST_TIMEOUT = | |||
| ConfigBuilder("spark.kubernetes.submission.requestTimeout") | |||
| .doc("request timeout to be used in milliseconds for starting the driver") | |||
| .version("3.0.0") | |||
There was a problem hiding this comment.
SPARK-27023, commit ID: e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .intConf | ||
| .createWithDefault(10000) | ||
|
|
||
| val SUBMISSION_CLIENT_CONNECTION_TIMEOUT = | ||
| ConfigBuilder("spark.kubernetes.submission.connectionTimeout") | ||
| .doc("connection timeout to be used in milliseconds for starting the driver") | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-27023, commit ID: e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .intConf | ||
| .createWithDefault(10000) | ||
|
|
||
| val DRIVER_CLIENT_REQUEST_TIMEOUT = | ||
| ConfigBuilder("spark.kubernetes.driver.requestTimeout") | ||
| .doc("request timeout to be used in milliseconds for driver to request executors") | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-27023, commit ID: e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .intConf | ||
| .createWithDefault(10000) | ||
|
|
||
| val DRIVER_CLIENT_CONNECTION_TIMEOUT = | ||
| ConfigBuilder("spark.kubernetes.driver.connectionTimeout") | ||
| .doc("connection timeout to be used in milliseconds for driver to request executors") | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-27023, commit ID: e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -122,82 +134,95 @@ private[spark] object Config extends Logging { | |||
| "this service account when requesting executor pods from the API server. If specific " + | |||
| "credentials are given for the driver pod to use, the driver will favor " + | |||
| "using those credentials instead.") | |||
| .version("2.3.0") | |||
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME = | ||
| ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName") | ||
| .doc("Service account that is used when running the executor pod." + | ||
| "If this parameter is not setup, the fallback logic will use the driver's service account.") | ||
| .version("3.1.0") |
There was a problem hiding this comment.
SPARK-30122, commit ID: f9f06ee#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_DRIVER_LIMIT_CORES = | ||
| ConfigBuilder("spark.kubernetes.driver.limit.cores") | ||
| .doc("Specify the hard cpu limit for the driver pod") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-22646, commit ID: 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_DRIVER_REQUEST_CORES = | ||
| ConfigBuilder("spark.kubernetes.driver.request.cores") | ||
| .doc("Specify the cpu request for the driver pod") | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-27754, commit ID: 1a8c093#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_DRIVER_SUBMIT_CHECK = | ||
| ConfigBuilder("spark.kubernetes.submitInDriver") | ||
| .internal() | ||
| .version("2.4.0") |
There was a problem hiding this comment.
SPARK-22839, commit ID: f15906d#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .booleanConf | ||
| .createWithDefault(false) | ||
|
|
||
| val KUBERNETES_EXECUTOR_LIMIT_CORES = | ||
| ConfigBuilder("spark.kubernetes.executor.limit.cores") | ||
| .doc("Specify the hard cpu limit for each executor pod") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_EXECUTOR_SCHEDULER_NAME = | ||
| ConfigBuilder("spark.kubernetes.executor.scheduler.name") | ||
| .doc("Specify the scheduler name for each executor pod") | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-29436, commit ID: f800fa3#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_EXECUTOR_REQUEST_CORES = | ||
| ConfigBuilder("spark.kubernetes.executor.request.cores") | ||
| .doc("Specify the cpu request for each executor pod") | ||
| .version("2.4.0") |
There was a problem hiding this comment.
SPARK-23285, commit ID: fe2b7a4#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_DRIVER_POD_NAME = | ||
| ConfigBuilder("spark.kubernetes.driver.pod.name") | ||
| .doc("Name of the driver pod.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_EXECUTOR_POD_NAME_PREFIX = | ||
| ConfigBuilder("spark.kubernetes.executor.podNamePrefix") | ||
| .doc("Prefix to use in front of the executor pod names.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .internal() | ||
| .stringConf | ||
| .createOptional | ||
|
|
||
| val KUBERNETES_ALLOCATION_BATCH_SIZE = | ||
| ConfigBuilder("spark.kubernetes.allocation.batch.size") | ||
| .doc("Number of pods to launch at once in each round of executor allocation.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .intConf | ||
| .checkValue(value => value > 0, "Allocation batch size should be a positive integer") | ||
| .createWithDefault(5) | ||
|
|
||
| val KUBERNETES_ALLOCATION_BATCH_DELAY = | ||
| ConfigBuilder("spark.kubernetes.allocation.batch.delay") | ||
| .doc("Time to wait between each round of executor allocation.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -206,6 +231,7 @@ private[spark] object Config extends Logging { | |||
| ConfigBuilder("spark.kubernetes.executor.lostCheck.maxAttempts") | |||
| .doc("Maximum number of attempts allowed for checking the reason of an executor loss " + | |||
| "before it is assumed that the executor failed.") | |||
| .version("2.3.0") | |||
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| // For testing only. | ||
| val KUBERNETES_DRIVER_POD_NAME_PREFIX = | ||
| ConfigBuilder("spark.kubernetes.driver.resourceNamePrefix") | ||
| .internal() | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-25876, commit ID: 6be272b#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -215,12 +241,14 @@ private[spark] object Config extends Logging { | |||
| ConfigBuilder("spark.kubernetes.submission.waitAppCompletion") | |||
| .doc("In cluster mode, whether to wait for the application to finish before exiting the " + | |||
| "launcher process.") | |||
| .version("2.3.0") | |||
There was a problem hiding this comment.
SPARK-22646, commit ID: 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| .booleanConf | ||
| .createWithDefault(true) | ||
|
|
||
| val REPORT_INTERVAL = | ||
| ConfigBuilder("spark.kubernetes.report.interval") | ||
| .doc("Interval between reports of the current app status in cluster mode.") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-22646, commit ID: 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -229,6 +257,7 @@ private[spark] object Config extends Logging { | |||
| ConfigBuilder("spark.kubernetes.executor.apiPollingInterval") | |||
| .doc("Interval between polls against the Kubernetes API server to inspect the " + | |||
| "state of executors.") | |||
| .version("2.4.0") | |||
There was a problem hiding this comment.
SPARK-24248, commit ID: 270a9a3#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -775,6 +806,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add the annotation specified by <code>AnnotationName</code> to the driver pod. | |||
| For example, <code>spark.kubernetes.driver.annotation.something=true</code>. | |||
| </td> | |||
| <td>2.3.0</td> | |||
There was a problem hiding this comment.
SPARK-22646, commit ID: 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -785,6 +817,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Note that Spark also adds its own labels to the executor pod | |||
| for bookkeeping purposes. | |||
| </td> | |||
| <td>2.3.0</td> | |||
There was a problem hiding this comment.
SPARK-22646, commit ID: 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -793,6 +826,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add the annotation specified by <code>AnnotationName</code> to the executor pods. | |||
| For example, <code>spark.kubernetes.executor.annotation.something=true</code>. | |||
| </td> | |||
| <td>2.3.0</td> | |||
There was a problem hiding this comment.
SPARK-22646, commit ID: 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -871,6 +913,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| will result in the driver pod and executors having a node selector with key <code>identifier</code> and value | |||
| <code>myIdentifier</code>. Multiple node selector keys can be added by setting multiple configurations with this prefix. | |||
| </td> | |||
| <td>2.3.0</td> | |||
There was a problem hiding this comment.
SPARK-18278, commit ID: e9b2070#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -879,6 +922,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add the environment variable specified by <code>EnvironmentVariableName</code> to | |||
| the Driver process. The user can specify multiple of these to set multiple environment variables. | |||
| </td> | |||
| <td>2.3.0</td> | |||
There was a problem hiding this comment.
SPARK-22646, commit ID: 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -887,6 +931,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add the <a href="https://kubernetes.io/docs/concepts/configuration/secret/">Kubernetes Secret</a> named <code>SecretName</code> to the driver pod on the path specified in the value. For example, | |||
| <code>spark.kubernetes.driver.secrets.spark-secret=/etc/secrets</code>. | |||
| </td> | |||
| <td>2.3.0</td> | |||
There was a problem hiding this comment.
SPARK-22757, commit ID: 171f6dd#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -895,6 +940,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add the <a href="https://kubernetes.io/docs/concepts/configuration/secret/">Kubernetes Secret</a> named <code>SecretName</code> to the executor pod on the path specified in the value. For example, | |||
| <code>spark.kubernetes.executor.secrets.spark-secret=/etc/secrets</code>. | |||
| </td> | |||
| <td>2.3.0</td> | |||
There was a problem hiding this comment.
SPARK-22757, commit ID: 171f6dd#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -903,6 +949,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add as an environment variable to the driver container with name EnvName (case sensitive), the value referenced by key <code> key </code> in the data of the referenced <a href="https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables">Kubernetes Secret</a>. For example, | |||
| <code>spark.kubernetes.driver.secretKeyRef.ENV_VAR=spark-secret:key</code>. | |||
| </td> | |||
| <td>2.4.0</td> | |||
There was a problem hiding this comment.
SPARK-24232, commit ID: 21e1fc7#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -911,6 +958,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add as an environment variable to the executor container with name EnvName (case sensitive), the value referenced by key <code> key </code> in the data of the referenced <a href="https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables">Kubernetes Secret</a>. For example, | |||
| <code>spark.kubernetes.executor.secrets.ENV_VAR=spark-secret:key</code>. | |||
| </td> | |||
| <td>2.4.0</td> | |||
There was a problem hiding this comment.
SPARK-24232, commit ID: 21e1fc7#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -919,6 +967,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add the <a href="https://kubernetes.io/docs/concepts/storage/volumes/">Kubernetes Volume</a> named <code>VolumeName</code> of the <code>VolumeType</code> type to the driver pod on the path specified in the value. For example, | |||
| <code>spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint</code>. | |||
| </td> | |||
| <td>2.4.0</td> | |||
There was a problem hiding this comment.
SPARK-23529, commit ID: 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -935,6 +985,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Specify if the mounted volume is read only or not. For example, | |||
| <code>spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false</code>. | |||
| </td> | |||
| <td>2.4.0</td> | |||
There was a problem hiding this comment.
SPARK-23529, commit ID: 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -951,6 +1003,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Add the <a href="https://kubernetes.io/docs/concepts/storage/volumes/">Kubernetes Volume</a> named <code>VolumeName</code> of the <code>VolumeType</code> type to the executor pod on the path specified in the value. For example, | |||
| <code>spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint</code>. | |||
| </td> | |||
| <td>2.4.0</td> | |||
There was a problem hiding this comment.
SPARK-23529, commit ID: 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -967,6 +1021,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Specify if the mounted volume is read only or not. For example, | |||
| <code>spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false</code>. | |||
| </td> | |||
| <td>2.4.0</td> | |||
There was a problem hiding this comment.
SPARK-23529, commit ID: 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -927,6 +976,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Specifies a <a href="https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath">subpath</a> to be mounted from the volume into the driver pod. | |||
| <code>spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint</code>. | |||
| </td> | |||
| <td>3.0.0</td> | |||
There was a problem hiding this comment.
SPARK-25960, commit ID: 3df307a#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -959,6 +1012,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Specifies a <a href="https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath">subpath</a> to be mounted from the volume into the executor pod. | |||
| <code>spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint</code>. | |||
| </td> | |||
| <td>3.0.0</td> | |||
There was a problem hiding this comment.
SPARK-25960, commit ID: 3df307a#diff-6e882d5561424e7e6651eb46f10104b8
beliefer
commented
Mar 11, 2020
| @@ -943,6 +994,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Configure <a href="https://kubernetes.io/docs/concepts/storage/volumes/">Kubernetes Volume</a> options passed to the Kubernetes with <code>OptionName</code> as key having specified value, must conform with Kubernetes option format. For example, | |||
| <code>spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim</code>. | |||
| </td> | |||
| <td>2.4.0</td> | |||
There was a problem hiding this comment.
SPARK-23529, commit ID: 5ff1b9b#diff-b5527f236b253e0d9f5db5164bdb43e9
beliefer
commented
Mar 11, 2020
| @@ -975,6 +1030,7 @@ See the [configuration page](configuration.html) for information on Spark config | |||
| Configure <a href="https://kubernetes.io/docs/concepts/storage/volumes/">Kubernetes Volume</a> options passed to the Kubernetes with <code>OptionName</code> as key having specified value. For example, | |||
| <code>spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim</code>. | |||
| </td> | |||
| <td>2.4.0</td> | |||
There was a problem hiding this comment.
SPARK-23529, commit ID: 5ff1b9b#diff-b5527f236b253e0d9f5db5164bdb43e9
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
retest this please |
|
Test build #119673 has finished for PR 27875 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
HyukjinKwon
approved these changes
Mar 12, 2020
|
Merged to master and brnach-3.0. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Apr 7, 2020
…of K8S Add version information to the configuration of `K8S`. I sorted out some information show below. Item name | Since version | JIRA ID | Commit ID | Note -- | -- | -- | -- | -- spark.kubernetes.context | 3.0.0 | SPARK-25887 | c542c24#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.master | 3.0.0 | SPARK-30371 | f14061c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.namespace | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.container.image | 2.3.0 | SPARK-22994 | b94debd#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.container.image | 2.3.0 | SPARK-22807 | fb3636b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.container.image | 2.3.0 | SPARK-22807 | fb3636b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.container.image.pullPolicy | 2.3.0 | SPARK-22807 | fb3636b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.container.image.pullSecrets | 2.4.0 | SPARK-23668 | cccaaa1#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.submission.requestTimeout | 3.0.0 | SPARK-27023 | e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.submission.connectionTimeout | 3.0.0 | SPARK-27023 | e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.requestTimeout | 3.0.0 | SPARK-27023 | e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.connectionTimeout | 3.0.0 | SPARK-27023 | e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8 | KUBERNETES_AUTH_DRIVER_CONF_PREFIX.serviceAccountName | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName | 3.1.0 | SPARK-30122 | f9f06ee#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.executor spark.kubernetes.driver.limit.cores | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.request.cores | 3.0.0 | SPARK-27754 | 1a8c093#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.submitInDriver | 2.4.0 | SPARK-22839 | f15906d#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.limit.cores | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.scheduler.name | 3.0.0 | SPARK-29436 | f800fa3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.request.cores | 2.4.0 | SPARK-23285 | fe2b7a4#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.pod.name | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.resourceNamePrefix | 3.0.0 | SPARK-25876 | 6be272b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.podNamePrefix | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.allocation.batch.size | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.allocation.batch.delay | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.lostCheck.maxAttempts | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.submission.waitAppCompletion | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.report.interval | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.apiPollingInterval | 2.4.0 | SPARK-24248 | 270a9a3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.eventProcessingInterval | 2.4.0 | SPARK-24248 | 270a9a3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.memoryOverheadFactor | 2.4.0 | SPARK-23984 | 1a644af#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.pyspark.pythonVersion | 2.4.0 | SPARK-23984 | a791c29#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.kerberos.krb5.path | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.kerberos.krb5.configMapName | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.hadoop.configMapName | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.kerberos.tokenSecret.name | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.kerberos.tokenSecret.itemKey | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.resource.type | 2.4.1 | SPARK-25021 | 9031c78#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.local.dirs.tmpfs | 3.0.0 | SPARK-25262 | da6fa38#diff-6e882d5561424e7e6651eb46f10104b8 | It exists in branch-3.0, but in pom.xml it is 2.4.0-snapshot spark.kubernetes.driver.podTemplateFile | 3.0.0 | SPARK-24434 | f6cc354#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.podTemplateFile | 3.0.0 | SPARK-24434 | f6cc354#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.podTemplateContainerName | 3.0.0 | SPARK-24434 | f6cc354#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.podTemplateContainerName | 3.0.0 | SPARK-24434 | f6cc354#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.deleteOnTermination | 3.0.0 | SPARK-25515 | 0c2935b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.dynamicAllocation.deleteGracePeriod | 3.0.0 | SPARK-28487 | 0343854#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.appKillPodDeletionGracePeriod | 3.0.0 | SPARK-24793 | 05168e7#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.file.upload.path | 3.0.0 | SPARK-23153 | 5e74570#diff-6e882d5561424e7e6651eb46f10104b8 | The following appears in the document | | | | spark.kubernetes.authenticate.submission.caCertFile | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.submission.clientKeyFile | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.submission.clientCertFile | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.submission.oauthToken | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.submission.oauthTokenFile | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.caCertFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.clientKeyFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.clientCertFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.oauthToken | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.oauthTokenFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.mounted.caCertFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.mounted.clientKeyFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.mounted.clientCertFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.mounted.oauthTokenFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.caCertFile | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.clientKeyFile | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.clientCertFile | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.oauthToken | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.oauthTokenFile | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.label.[LabelName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.annotation.[AnnotationName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.label.[LabelName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.annotation.[AnnotationName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.node.selector.[labelKey] | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driverEnv.[EnvironmentVariableName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.secrets.[SecretName] | 2.3.0 | SPARK-22757 | 171f6dd#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.secrets.[SecretName] | 2.3.0 | SPARK-22757 | 171f6dd#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.secretKeyRef.[EnvName] | 2.4.0 | SPARK-24232 | 21e1fc7#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.secretKeyRef.[EnvName] | 2.4.0 | SPARK-24232 | 21e1fc7#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPath | 3.0.0 | SPARK-25960 | 3df307a#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnly | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName] | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-b5527f236b253e0d9f5db5164bdb43e9 | spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.path | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.subPath | 3.0.0 | SPARK-25960 | 3df307a#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.readOnly | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].options.[OptionName] | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-b5527f236b253e0d9f5db5164bdb43e9 | Supplemental configuration version information. 'No' Exists UT Closes #27875 from beliefer/add-version-to-k8s-config. Authored-by: beliefer <beliefer@163.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
sjincho
pushed a commit
to sjincho/spark
that referenced
this pull request
Apr 15, 2020
…of K8S ### What changes were proposed in this pull request? Add version information to the configuration of `K8S`. I sorted out some information show below. Item name | Since version | JIRA ID | Commit ID | Note -- | -- | -- | -- | -- spark.kubernetes.context | 3.0.0 | SPARK-25887 | c542c24#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.master | 3.0.0 | SPARK-30371 | f14061c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.namespace | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.container.image | 2.3.0 | SPARK-22994 | b94debd#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.container.image | 2.3.0 | SPARK-22807 | fb3636b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.container.image | 2.3.0 | SPARK-22807 | fb3636b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.container.image.pullPolicy | 2.3.0 | SPARK-22807 | fb3636b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.container.image.pullSecrets | 2.4.0 | SPARK-23668 | cccaaa1#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.submission.requestTimeout | 3.0.0 | SPARK-27023 | e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.submission.connectionTimeout | 3.0.0 | SPARK-27023 | e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.requestTimeout | 3.0.0 | SPARK-27023 | e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.connectionTimeout | 3.0.0 | SPARK-27023 | e9e8bb3#diff-6e882d5561424e7e6651eb46f10104b8 | KUBERNETES_AUTH_DRIVER_CONF_PREFIX.serviceAccountName | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName | 3.1.0 | SPARK-30122 | f9f06ee#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.executor spark.kubernetes.driver.limit.cores | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.request.cores | 3.0.0 | SPARK-27754 | 1a8c093#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.submitInDriver | 2.4.0 | SPARK-22839 | f15906d#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.limit.cores | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.scheduler.name | 3.0.0 | SPARK-29436 | f800fa3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.request.cores | 2.4.0 | SPARK-23285 | fe2b7a4#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.pod.name | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.resourceNamePrefix | 3.0.0 | SPARK-25876 | 6be272b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.podNamePrefix | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.allocation.batch.size | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.allocation.batch.delay | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.lostCheck.maxAttempts | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.submission.waitAppCompletion | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.report.interval | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.apiPollingInterval | 2.4.0 | SPARK-24248 | 270a9a3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.eventProcessingInterval | 2.4.0 | SPARK-24248 | 270a9a3#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.memoryOverheadFactor | 2.4.0 | SPARK-23984 | 1a644af#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.pyspark.pythonVersion | 2.4.0 | SPARK-23984 | a791c29#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.kerberos.krb5.path | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.kerberos.krb5.configMapName | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.hadoop.configMapName | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.kerberos.tokenSecret.name | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.kerberos.tokenSecret.itemKey | 3.0.0 | SPARK-23257 | 6c9c84f#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.resource.type | 2.4.1 | SPARK-25021 | 9031c78#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.local.dirs.tmpfs | 3.0.0 | SPARK-25262 | da6fa38#diff-6e882d5561424e7e6651eb46f10104b8 | It exists in branch-3.0, but in pom.xml it is 2.4.0-snapshot spark.kubernetes.driver.podTemplateFile | 3.0.0 | SPARK-24434 | f6cc354#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.podTemplateFile | 3.0.0 | SPARK-24434 | f6cc354#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.podTemplateContainerName | 3.0.0 | SPARK-24434 | f6cc354#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.podTemplateContainerName | 3.0.0 | SPARK-24434 | f6cc354#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.deleteOnTermination | 3.0.0 | SPARK-25515 | 0c2935b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.dynamicAllocation.deleteGracePeriod | 3.0.0 | SPARK-28487 | 0343854#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.appKillPodDeletionGracePeriod | 3.0.0 | SPARK-24793 | 05168e7#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.file.upload.path | 3.0.0 | SPARK-23153 | 5e74570#diff-6e882d5561424e7e6651eb46f10104b8 | The following appears in the document | | | | spark.kubernetes.authenticate.submission.caCertFile | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.submission.clientKeyFile | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.submission.clientCertFile | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.submission.oauthToken | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.submission.oauthTokenFile | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.caCertFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.clientKeyFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.clientCertFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.oauthToken | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.oauthTokenFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.mounted.caCertFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.mounted.clientKeyFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.mounted.clientCertFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.driver.mounted.oauthTokenFile | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.caCertFile | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.clientKeyFile | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.clientCertFile | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.oauthToken | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.authenticate.oauthTokenFile | 2.4.0 | SPARK-23146 | 571a6f0#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.label.[LabelName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.annotation.[AnnotationName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.label.[LabelName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.annotation.[AnnotationName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.node.selector.[labelKey] | 2.3.0 | SPARK-18278 | e9b2070#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driverEnv.[EnvironmentVariableName] | 2.3.0 | SPARK-22646 | 3f4060c#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.secrets.[SecretName] | 2.3.0 | SPARK-22757 | 171f6dd#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.secrets.[SecretName] | 2.3.0 | SPARK-22757 | 171f6dd#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.secretKeyRef.[EnvName] | 2.4.0 | SPARK-24232 | 21e1fc7#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.secretKeyRef.[EnvName] | 2.4.0 | SPARK-24232 | 21e1fc7#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPath | 3.0.0 | SPARK-25960 | 3df307a#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnly | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName] | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-b5527f236b253e0d9f5db5164bdb43e9 | spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.path | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.subPath | 3.0.0 | SPARK-25960 | 3df307a#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.readOnly | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-6e882d5561424e7e6651eb46f10104b8 | spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].options.[OptionName] | 2.4.0 | SPARK-23529 | 5ff1b9b#diff-b5527f236b253e0d9f5db5164bdb43e9 | ### Why are the changes needed? Supplemental configuration version information. ### Does this PR introduce any user-facing change? 'No' ### How was this patch tested? Exists UT Closes apache#27875 from beliefer/add-version-to-k8s-config. Authored-by: beliefer <beliefer@163.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add version information to the configuration of
K8S.I sorted out some information show below.
Why are the changes needed?
Supplemental configuration version information.
Does this PR introduce any user-facing change?
'No'
How was this patch tested?
Exists UT