Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[KYUUBI #5654] Introduce new config to forcibly rewrite pod name #5672

Closed
wants to merge 1 commit into from
Closed

[KYUUBI #5654] Introduce new config to forcibly rewrite pod name #5672

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions docs/configuration/settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,20 +309,22 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co

### Kubernetes

| Key | Default | Meaning | Type | Since |
|-----------------------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.kubernetes.authenticate.caCertFile | <undefined> | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.clientCertFile | <undefined> | Path to the client cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.clientKeyFile | <undefined> | Path to the client key file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.oauthToken | <undefined> | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.oauthTokenFile | <undefined> | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.context | <undefined> | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
| kyuubi.kubernetes.context.allow.list || The allowed kubernetes context list, if it is empty, there is no kubernetes context limitation. | set | 1.8.0 |
| kyuubi.kubernetes.master.address | <undefined> | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
| kyuubi.kubernetes.namespace.allow.list || The allowed kubernetes namespace list, if it is empty, there is no kubernetes namespace limitation. | set | 1.8.0 |
| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M | The period for which the Kyuubi server retains application information after the application terminates. | duration | 1.7.1 |
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |
| Key | Default | Meaning | Type | Since |
|----------------------------------------------------------------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.kubernetes.authenticate.caCertFile | <undefined> | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.clientCertFile | <undefined> | Path to the client cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.clientKeyFile | <undefined> | Path to the client key file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.oauthToken | <undefined> | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.oauthTokenFile | <undefined> | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.context | <undefined> | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
| kyuubi.kubernetes.context.allow.list || The allowed kubernetes context list, if it is empty, there is no kubernetes context limitation. | set | 1.8.0 |
| kyuubi.kubernetes.master.address | <undefined> | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
| kyuubi.kubernetes.namespace.allow.list || The allowed kubernetes namespace list, if it is empty, there is no kubernetes namespace limitation. | set | 1.8.0 |
| kyuubi.kubernetes.spark.forciblyRewriteDriverPodName.enabled | false | Whether to forcibly rewrite Spark driver pod name with 'kyuubi-<uuid>-driver'. If disabled, Kyuubi will try to preserve the application name while satisfying K8s' pod name policy, but some vendors may have stricter pod name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
| kyuubi.kubernetes.spark.forciblyRewriteExecutorPodNamePrefix.enabled | false | Whether to forcibly rewrite Spark executor pod name prefix with 'kyuubi-<uuid>'. If disabled, Kyuubi will try to preserve the application name while satisfying K8s' pod name policy, but some vendors may have stricter Pod name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M | The period for which the Kyuubi server retains application information after the application terminates. | duration | 1.7.1 |
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |

### Lineage

Expand Down
20 changes: 20 additions & 0 deletions kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3153,4 +3153,24 @@ object KyuubiConf {
.serverOnly
.intConf
.createOptional

val KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME: ConfigEntry[Boolean] =
buildConf("kyuubi.kubernetes.spark.forciblyRewriteDriverPodName.enabled")
.doc("Whether to forcibly rewrite Spark driver pod name with 'kyuubi-<uuid>-driver'. " +
"If disabled, Kyuubi will try to preserve the application name while satisfying K8s' " +
"pod name policy, but some vendors may have stricter pod name policies, thus the " +
"generated name may become illegal.")
.version("1.8.1")
.booleanConf
.createWithDefault(false)

val KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX: ConfigEntry[Boolean] =
buildConf("kyuubi.kubernetes.spark.forciblyRewriteExecutorPodNamePrefix.enabled")
.doc("Whether to forcibly rewrite Spark executor pod name prefix with 'kyuubi-<uuid>'. " +
"If disabled, Kyuubi will try to preserve the application name while satisfying K8s' " +
"pod name policy, but some vendors may have stricter Pod name policies, thus the " +
"generated name may become illegal.")
.version("1.8.1")
.booleanConf
.createWithDefault(false)
}
16 changes: 14 additions & 2 deletions kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import org.apache.hadoop.security.UserGroupInformation

import org.apache.kyuubi._
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf._
import org.apache.kyuubi.engine.{ApplicationManagerInfo, KyuubiApplicationManager, ProcBuilder}
import org.apache.kyuubi.engine.KubernetesApplicationOperation.{KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT}
import org.apache.kyuubi.engine.ProcBuilder.KYUUBI_ENGINE_LOG_PATH_KEY
Expand Down Expand Up @@ -229,17 +230,28 @@ class SparkProcessBuilder(
kubernetesNamespace())
}

private val forciblyRewriteDriverPodName: Boolean =
conf.get(KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME)
private val forciblyRewriteExecPodNamePrefix: Boolean =
conf.get(KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX)

def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
val appName = conf.getOrElse(APP_KEY, "spark")
val map = mutable.Map.newBuilder[String, String]
if (clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
if (!conf.contains(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)) {
val prefix = KubernetesUtils.generateExecutorPodNamePrefix(appName, engineRefId)
val prefix = KubernetesUtils.generateExecutorPodNamePrefix(
appName,
engineRefId,
forciblyRewriteExecPodNamePrefix)
map += (KUBERNETES_EXECUTOR_POD_NAME_PREFIX -> prefix)
}
if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
if (!conf.contains(KUBERNETES_DRIVER_POD_NAME)) {
val name = KubernetesUtils.generateDriverPodName(appName, engineRefId)
val name = KubernetesUtils.generateDriverPodName(
appName,
engineRefId,
forciblyRewriteDriverPodName)
map += (KUBERNETES_DRIVER_POD_NAME -> name)
}
}
Expand Down
24 changes: 15 additions & 9 deletions kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,21 +130,27 @@ object KubernetesUtils extends Logging {
.replaceAll("^[0-9]", "x")
}

def generateDriverPodName(appName: String, engineRefId: String): String = {
val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, engineRefId)}-driver"
if (resolvedResourceName.length <= DRIVER_POD_NAME_MAX_LENGTH) {
resolvedResourceName
} else {
def generateDriverPodName(
appName: String,
engineRefId: String,
forciblyRewrite: Boolean): String = {
lazy val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, engineRefId)}-driver"
if (forciblyRewrite || resolvedResourceName.length > DRIVER_POD_NAME_MAX_LENGTH) {
s"kyuubi-$engineRefId-driver"
} else {
resolvedResourceName
}
}

def generateExecutorPodNamePrefix(appName: String, engineRefId: String): String = {
def generateExecutorPodNamePrefix(
appName: String,
engineRefId: String,
forciblyRewrite: Boolean): String = {
val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, engineRefId)}"
if (resolvedResourceName.length <= EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH) {
resolvedResourceName
} else {
if (forciblyRewrite || resolvedResourceName.length > EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH) {
s"kyuubi-$engineRefId"
} else {
resolvedResourceName
}
}
}
21 changes: 20 additions & 1 deletion kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import org.scalatestplus.mockito.MockitoSugar

import org.apache.kyuubi._
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf.{ENGINE_LOG_TIMEOUT, ENGINE_SPARK_MAIN_RESOURCE}
import org.apache.kyuubi.config.KyuubiConf.{ENGINE_LOG_TIMEOUT, ENGINE_SPARK_MAIN_RESOURCE, KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME, KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX}
import org.apache.kyuubi.engine.ProcBuilder.KYUUBI_ENGINE_LOG_PATH_KEY
import org.apache.kyuubi.engine.spark.SparkProcessBuilder._
import org.apache.kyuubi.ha.HighAvailabilityConf
Expand Down Expand Up @@ -336,6 +336,15 @@ class SparkProcessBuilderSuite extends KerberizedTestHelper with MockitoSugar {
val conf4 = Map(APP_KEY -> chineseAppName)
val driverPodName4 = processBuilder.appendPodNameConf(conf4).get(KUBERNETES_DRIVER_POD_NAME)
assert(driverPodName4 === Some(s"kyuubi-test-$engineRefId-driver"))
val newProcessBuilder = new SparkProcessBuilder(
"kyuubi",
conf.set(MASTER_KEY, "k8s://internal").set(DEPLOY_MODE_KEY, "cluster").set(
KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME,
true),
engineRefId)
val conf5 = Map(APP_KEY -> "test-forcibly-rewrite-app")
val driverPodName5 = newProcessBuilder.appendPodNameConf(conf5).get(KUBERNETES_DRIVER_POD_NAME)
assert(driverPodName5 === Some(s"kyuubi-$engineRefId-driver"))
}

test("[KYUUBI #5165] Test SparkProcessBuilder#appendExecutorPodPrefix") {
Expand Down Expand Up @@ -363,6 +372,16 @@ class SparkProcessBuilderSuite extends KerberizedTestHelper with MockitoSugar {
val execPodNamePrefix3 = processBuilder
.appendPodNameConf(conf3).get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)
assert(execPodNamePrefix3 === Some(s"kyuubi-$engineRefId"))
val newProcessBuilder = new SparkProcessBuilder(
"kyuubi",
conf.set(MASTER_KEY, "k8s://internal").set(DEPLOY_MODE_KEY, "cluster").set(
KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX,
true),
engineRefId)
val conf5 = Map(APP_KEY -> "test-forcibly-rewrite-app")
val execPodNamePrefix4 = newProcessBuilder
.appendPodNameConf(conf5).get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)
assert(execPodNamePrefix4 === Some(s"kyuubi-$engineRefId"))
}

test("extract spark core scala version") {
Expand Down
Loading