Skip to content

Commit

Permalink
[KYUUBI #2719] [SUB-TASK][KPIP-4] Support internal rest request authe…
Browse files Browse the repository at this point in the history
…ntication to enable http request redirection across kyuubi instances

### _Why are the changes needed?_

Support internal rest request authentication to enable redirect http request across kyuubi instances

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2719 from turboFei/internal_access.

Closes #2719

b9e150e [Fei Wang] revert config key change
aa37872 [Fei Wang] make it internal
ce96d92 [Fei Wang] comments
6fe8523 [Fei Wang] Support internal rest request authentication to enable redirect http request across kyuubi instances

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
  • Loading branch information
turboFei committed May 23, 2022
1 parent 9578475 commit f1cf95f
Show file tree
Hide file tree
Showing 14 changed files with 152 additions and 37 deletions.
8 changes: 0 additions & 8 deletions docs/deployment/settings.md
Expand Up @@ -220,13 +220,6 @@ Key | Default | Meaning | Type | Since
<code>kyuubi.engine.pool.name</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>engine-pool</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The name of engine pool.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.engine.pool.size</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>-1</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The size of engine pool. Note that, if the size is less than 1, the engine pool will not be enabled; otherwise, the size of the engine pool will be min(this, kyuubi.engine.pool.size.threshold).</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
<code>kyuubi.engine.pool.size.threshold</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>9</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>This parameter is introduced as a server-side parameter, and controls the upper limit of the engine pool.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
<code>kyuubi.engine.security.crypto.cipher</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>AES/CBC/PKCS5PADDING</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The cipher transformation to use for encrypting engine access token.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.engine.security.crypto.ivLength</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>16</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Initial vector length, in bytes.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.engine.security.crypto.keyAlgorithm</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>AES</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The algorithm for generated secret keys.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.engine.security.crypto.keyLength</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>128</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The length in bits of the encryption key to generate. Valid values are 128, 192 and 256</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.engine.security.enabled</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Whether to enable the internal secure access between Kyuubi server and engine.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.engine.security.secret.provider</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>org.apache.kyuubi.service.authentication.ZooKeeperEngineSecuritySecretProviderImpl</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The class used to manage the engine security secret. This class must be a subclass of EngineSecuritySecretProvider.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.engine.security.token.max.lifetime</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT10M</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The max lifetime of the token used for secure access between Kyuubi server and engine.</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.engine.session.initialize.sql</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>SemiColon-separated list of SQL statements to be initialized in the newly created engine session before queries. This configuration can not be used in JDBC url due to the limitation of Beeline/JDBC driver.</div>|<div style='width: 30pt'>seq</div>|<div style='width: 20pt'>1.3.0</div>
<code>kyuubi.engine.share.level</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>USER</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Engines will be shared in different levels, available configs are: <ul> <li>CONNECTION: engine will not be shared but only used by the current client connection</li> <li>USER: engine will be shared by all sessions created by a unique username, see also kyuubi.engine.share.level.subdomain</li> <li>GROUP: engine will be shared by all sessions created by all users belong to the same primary group name. The engine will be launched by the group name as the effective username, so here the group name is kind of special user who is able to visit the compute resources/data of a team. It follows the [Hadoop GroupsMapping](https://reurl.cc/xE61Y5) to map user to a primary group. If the primary group is not found, it fallback to the USER level. <li>SERVER: the App will be shared by Kyuubi servers</li></ul></div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.2.0</div>
<code>kyuubi.engine.share.level.sub.domain</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) - Using kyuubi.engine.share.level.subdomain instead</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.2.0</div>
Expand Down Expand Up @@ -292,7 +285,6 @@ Key | Default | Meaning | Type | Since
<code>kyuubi.ha.zookeeper.connection.retry.policy</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>EXPONENTIAL_BACKOFF</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The retry policy for connecting to the zookeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul></div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.0.0</div>
<code>kyuubi.ha.zookeeper.connection.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>15000</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The timeout(ms) of creating the connection to the zookeeper ensemble</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.0.0</div>
<code>kyuubi.ha.zookeeper.engine.auth.type</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>NONE</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The type of zookeeper authentication for engine, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul></div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.3.2</div>
<code>kyuubi.ha.zookeeper.engine.secure.secret.node</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The zk node contains the secret that used for internal secure between Kyuubi server and Kyuubi engine, please make sure that it is only visible for Kyuubi.</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.5.0</div>
<code>kyuubi.ha.zookeeper.namespace</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>kyuubi</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The root directory for the service to deploy its instance uri</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.0.0</div>
<code>kyuubi.ha.zookeeper.node.creation.timeout</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT2M</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Timeout for creating zookeeper node</div>|<div style='width: 30pt'>duration</div>|<div style='width: 20pt'>1.2.0</div>
<code>kyuubi.ha.zookeeper.publish.configs</code>|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>When set to true, publish Kerberos configs to Zookeeper.Note that the Hive driver needs to be greater than 1.3 or 2.0 or apply HIVE-11581 patch.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.4.0</div>
Expand Down
Expand Up @@ -1250,21 +1250,26 @@ object KyuubiConf {

val ENGINE_SECURITY_ENABLED: ConfigEntry[Boolean] =
buildConf("kyuubi.engine.security.enabled")
.doc("Whether to enable the internal secure access between Kyuubi server and engine.")
.internal
.doc("Whether to enable the internal secure access. Before 1.6.0, it is used for the secure" +
" access between kyuubi server and kyuubi engine. Since 1.6.0, kyuubi supports internal" +
" secure across kyuubi server instances.")
.version("1.5.0")
.booleanConf
.createWithDefault(false)

val ENGINE_SECURITY_TOKEN_MAX_LIFETIME: ConfigEntry[Long] =
buildConf("kyuubi.engine.security.token.max.lifetime")
.doc("The max lifetime of the token used for secure access between Kyuubi server and engine.")
.internal
.doc("The max lifetime of the token used for internal secure access.")
.version("1.5.0")
.timeConf
.createWithDefault(Duration.ofMinutes(10).toMillis)

val ENGINE_SECURITY_SECRET_PROVIDER: ConfigEntry[String] =
buildConf("kyuubi.engine.security.secret.provider")
.doc("The class used to manage the engine security secret. This class must be a " +
.internal
.doc("The class used to manage the internal security secret. This class must be a " +
"subclass of EngineSecuritySecretProvider.")
.version("1.5.0")
.stringConf
Expand All @@ -1273,6 +1278,7 @@ object KyuubiConf {

val ENGINE_SECURITY_CRYPTO_KEY_LENGTH: ConfigEntry[Int] =
buildConf("kyuubi.engine.security.crypto.keyLength")
.internal
.doc("The length in bits of the encryption key to generate. " +
"Valid values are 128, 192 and 256")
.version("1.5.0")
Expand All @@ -1282,21 +1288,24 @@ object KyuubiConf {

val ENGINE_SECURITY_CRYPTO_IV_LENGTH: ConfigEntry[Int] =
buildConf("kyuubi.engine.security.crypto.ivLength")
.internal
.doc("Initial vector length, in bytes.")
.version("1.5.0")
.intConf
.createWithDefault(16)

val ENGINE_SECURITY_CRYPTO_KEY_ALGORITHM: ConfigEntry[String] =
buildConf("kyuubi.engine.security.crypto.keyAlgorithm")
.internal
.doc("The algorithm for generated secret keys.")
.version("1.5.0")
.stringConf
.createWithDefault("AES")

val ENGINE_SECURITY_CRYPTO_CIPHER_TRANSFORMATION: ConfigEntry[String] =
buildConf("kyuubi.engine.security.crypto.cipher")
.doc("The cipher transformation to use for encrypting engine access token.")
.internal
.doc("The cipher transformation to use for encrypting internal access token.")
.version("1.5.0")
.stringConf
.createWithDefault("AES/CBC/PKCS5PADDING")
Expand Down
Expand Up @@ -19,6 +19,6 @@ package org.apache.kyuubi.service.authentication

class EngineSecureAuthenticationProviderImpl extends PasswdAuthenticationProvider {
override def authenticate(user: String, password: String): Unit = {
EngineSecurityAccessor.get().authToken(password)
InternalSecurityAccessor.get().authToken(password)
}
}
Expand Up @@ -24,7 +24,7 @@ import org.apache.kyuubi.{KyuubiSQLException, Logging}
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf._

class EngineSecurityAccessor(conf: KyuubiConf, val isServer: Boolean) {
class InternalSecurityAccessor(conf: KyuubiConf, val isServer: Boolean) {
val cryptoKeyLengthBytes = conf.get(ENGINE_SECURITY_CRYPTO_KEY_LENGTH) / java.lang.Byte.SIZE
val cryptoIvLength = conf.get(ENGINE_SECURITY_CRYPTO_IV_LENGTH)
val cryptoKeyAlgorithm = conf.get(ENGINE_SECURITY_CRYPTO_KEY_ALGORITHM)
Expand Down Expand Up @@ -109,16 +109,16 @@ class EngineSecurityAccessor(conf: KyuubiConf, val isServer: Boolean) {
}
}

object EngineSecurityAccessor extends Logging {
@volatile private var _engineSecurityAccessor: EngineSecurityAccessor = _
object InternalSecurityAccessor extends Logging {
@volatile private var _engineSecurityAccessor: InternalSecurityAccessor = _

def initialize(conf: KyuubiConf, isServer: Boolean): Unit = {
if (_engineSecurityAccessor == null) {
_engineSecurityAccessor = new EngineSecurityAccessor(conf, isServer)
_engineSecurityAccessor = new InternalSecurityAccessor(conf, isServer)
}
}

def get(): EngineSecurityAccessor = {
def get(): InternalSecurityAccessor = {
_engineSecurityAccessor
}
}
Expand Up @@ -57,7 +57,7 @@ class KyuubiAuthenticationFactory(conf: KyuubiConf, isServer: Boolean = true) ex
}

if (conf.get(ENGINE_SECURITY_ENABLED)) {
EngineSecurityAccessor.initialize(conf, isServer)
InternalSecurityAccessor.initialize(conf, isServer)
}

private def getSaslProperties: java.util.Map[String, String] = {
Expand Down
Expand Up @@ -20,7 +20,7 @@ package org.apache.kyuubi.service.authentication
import org.apache.kyuubi.{KyuubiFunSuite, KyuubiSQLException}
import org.apache.kyuubi.config.KyuubiConf

class EngineSecurityAccessorSuite extends KyuubiFunSuite {
class InternalSecurityAccessorSuite extends KyuubiFunSuite {
private val conf = KyuubiConf()
conf.set(
KyuubiConf.ENGINE_SECURITY_SECRET_PROVIDER,
Expand All @@ -31,7 +31,7 @@ class EngineSecurityAccessorSuite extends KyuubiFunSuite {
val newConf = conf.clone
newConf.set(KyuubiConf.ENGINE_SECURITY_CRYPTO_CIPHER_TRANSFORMATION, cipher)

val secureAccessor = new EngineSecurityAccessor(newConf, true)
val secureAccessor = new InternalSecurityAccessor(newConf, true)
val value = "tokenToEncrypt"
val encryptedValue = secureAccessor.encrypt(value)
assert(secureAccessor.decrypt(encryptedValue) === value)
Expand All @@ -40,7 +40,7 @@ class EngineSecurityAccessorSuite extends KyuubiFunSuite {
secureAccessor.authToken(token)
intercept[KyuubiSQLException](secureAccessor.authToken("invalidToken"))

val engineSecureAccessor = new EngineSecurityAccessor(newConf, false)
val engineSecureAccessor = new InternalSecurityAccessor(newConf, false)
engineSecureAccessor.authToken(token)
}
}
Expand Down
Expand Up @@ -153,8 +153,9 @@ object HighAvailabilityConf {

val HA_ZK_ENGINE_SECURE_SECRET_NODE: OptionalConfigEntry[String] =
buildConf("kyuubi.ha.zookeeper.engine.secure.secret.node")
.doc("The zk node contains the secret that used for internal secure between Kyuubi server " +
"and Kyuubi engine, please make sure that it is only visible for Kyuubi.")
.internal
.doc("The zk node contains the secret that used for internal secure, please make sure " +
"that it is only visible for Kyuubi.")
.version("1.5.0")
.stringConf
.createOptional
Expand Down
Expand Up @@ -20,5 +20,5 @@ package org.apache.kyuubi.server.http.authentication
object AuthSchemes extends Enumeration {
type AuthScheme = Value

val BASIC, NEGOTIATE = Value
val BASIC, NEGOTIATE, KYUUBI_INTERNAL = Value
}
Expand Up @@ -28,7 +28,7 @@ import org.apache.hadoop.security.authentication.client.AuthenticationException
import org.apache.kyuubi.Logging
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf.AUTHENTICATION_METHOD
import org.apache.kyuubi.service.authentication.AuthTypes
import org.apache.kyuubi.service.authentication.{AuthTypes, InternalSecurityAccessor}
import org.apache.kyuubi.service.authentication.AuthTypes.{KERBEROS, NOSASL}

class AuthenticationFilter(conf: KyuubiConf) extends Filter with Logging {
Expand Down Expand Up @@ -72,6 +72,10 @@ class AuthenticationFilter(conf: KyuubiConf) extends Filter with Logging {
val basicHandler = new BasicAuthenticationHandler(basicAuthType)
addAuthHandler(basicHandler)
}
if (InternalSecurityAccessor.get() != null) {
val internalHandler = new KyuubiInternalAuthenticationHandler
addAuthHandler(internalHandler)
}
super.init(filterConfig)
}

Expand Down
Expand Up @@ -26,7 +26,7 @@ import org.eclipse.jetty.server.handler.HandlerWrapper

import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf.{AUTHENTICATION_METHOD, ENGINE_SECURITY_ENABLED}
import org.apache.kyuubi.service.authentication.{AuthTypes, EngineSecurityAccessor}
import org.apache.kyuubi.service.authentication.{AuthTypes, InternalSecurityAccessor}
import org.apache.kyuubi.service.authentication.AuthTypes.KERBEROS

class KyuubiHttpAuthenticationFactory(conf: KyuubiConf) {
Expand All @@ -35,7 +35,7 @@ class KyuubiHttpAuthenticationFactory(conf: KyuubiConf) {
private val ugi = UserGroupInformation.getCurrentUser

if (conf.get(ENGINE_SECURITY_ENABLED)) {
EngineSecurityAccessor.initialize(conf, true)
InternalSecurityAccessor.initialize(conf, true)
}

private[kyuubi] val httpHandlerWrapperFactory =
Expand Down
@@ -0,0 +1,66 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.kyuubi.server.http.authentication

import java.nio.charset.Charset
import java.util.Base64
import javax.servlet.http.{HttpServletRequest, HttpServletResponse}

import org.apache.kyuubi.Logging
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.server.http.authentication.AuthSchemes.AuthScheme
import org.apache.kyuubi.service.authentication.InternalSecurityAccessor

class KyuubiInternalAuthenticationHandler extends AuthenticationHandler with Logging {
import AuthenticationHandler._

private var conf: KyuubiConf = _
override val authScheme: AuthScheme = AuthSchemes.KYUUBI_INTERNAL
private val internalSecurityAccessor = InternalSecurityAccessor.get()

override def init(conf: KyuubiConf): Unit = {
this.conf = conf
}

override def authenticationSupported: Boolean = {
internalSecurityAccessor != null
}

override def authenticate(
request: HttpServletRequest,
response: HttpServletResponse): String = {
var authUser: String = null
val authorization = getAuthorization(request)
val inputToken = Option(authorization).map(a => Base64.getDecoder.decode(a.getBytes()))
.getOrElse(Array.empty[Byte])
val creds = new String(inputToken, Charset.forName("UTF-8")).split(":")

if (creds.size < 2 || creds(0).trim.isEmpty || creds(1).trim.isEmpty) {
response.setHeader(WWW_AUTHENTICATE, authScheme.toString)
response.setStatus(HttpServletResponse.SC_UNAUTHORIZED)
} else {
val Seq(user, password) = creds.toSeq.take(2)
internalSecurityAccessor.authToken(password)
response.setStatus(HttpServletResponse.SC_OK)
authUser = user
}
authUser
}

override def destroy(): Unit = {}
}

0 comments on commit f1cf95f

Please sign in to comment.