[SPARK-27997][K8S] Add support for kubernetes OAuth Token refresh by haodemon · Pull Request #33675 · apache/spark

haodemon · 2021-08-07T19:52:11Z

What changes were proposed in this pull request?

This change allows a spark user to provide a class which implements fabric's OAuthTokenProvider to refresh tokens throughout the life of the spark app.

spark.kubernetes.authenticate.submission.oauthTokenProvider=<token>
spark.kubernetes.authenticate.driver.oauthTokenProvider=<token>
spark.kubernetes.authenticate.oauthTokenProvider=<token>

https://javadoc.io/doc/io.fabric8/kubernetes-client/5.6.0/io/fabric8/kubernetes/client/OAuthTokenProvider.html

Why are the changes needed?

Currently, while running spark on kubernetes, one should specify oauth tokens via config before starting an application.

spark.kubernetes.authenticate.submission.oauthToken=<token>
spark.kubernetes.authenticate.driver.oauthToken=<token>
spark.kubernetes.authenticate.oauthToken=<token>

The token has an expiration time (usually an hour, for GKE) and there is no way to update the token in the runtime. The spark app starts to throw exceptions.

io.fabric8.kubernetes.client.KubernetesClientException: Unauthorized
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:202)
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Does this PR introduce any user-facing change?

No, this is a new feature.

How was this patch tested?

A class which implements OAuthTokenProvider interface[0] was added into the classpath on driver node with no other spark options for tokens specified
It was also tested with expired tokens specified, and the token was updated via the user-provided class.

--conf spark.kubernetes.authenticate.submission.oauthToken=<expired>
--conf spark.kubernetes.authenticate.driver.oauthToken=<expired> 
--conf spark.kubernetes.authenticate.oauthToken=<expired>

There is no need to use any other token-related configuration options if this class is provided.

An example of the user-provided class for GKE
[0] https://gist.github.com/haodemon/5490fefdb258275c1f805d584319090b

import io.fabric8.kubernetes.client.OAuthTokenProvider

class OAuthGoogleTokenProvider extends OAuthTokenProvider {
  private val binary = "gcloud"
  private val args = "config config-helper --format=json"

  override def getToken: String = {
    val response = (binary + " " + args).!!
    val token = new ObjectMapper().readTree(response)
      .get("credential")
      .get("access_token")
    token.getTextValue
  }
}

dongjoon-hyun · 2021-08-08T06:06:06Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

+    ConfigBuilder("spark.kubernetes.client.oauth.token.provider.class")
+      .doc("A class that implements fabric's OAuthTokenProvider interface to " +
+        "provide a token refresh for long running jobs.")
+      .version("3.1.3")


Could you use 3.3.0 because the master branch is 3.3.0-SNAPSHOT?

Thank you for the review. Changed to 3.3.0

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

dongjoon-hyun

Thank you for making a PR, @haodemon .

Apache Spark mainly runs the unit tests on your GitHub Action. Could you enable the GitHub Action build_and_test job in your Spark fork?
- We also have a Jenkins sever. I'll trigger it though.
Apache Spark 3.3.0 will use 5.6.0, could you update the link to 5.6.0 link in the PR description?
- https://javadoc.io/doc/io.fabric8/kubernetes-client/4.12.0/io/fabric8/kubernetes/client/OAuthTokenProvider.html

dongjoon-hyun · 2021-08-08T06:11:22Z

ok to test

SparkQA · 2021-08-08T06:28:54Z

Test build #142193 has finished for PR 33675 at commit 7b0f6c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
.doc(\"A class that implements fabric's OAuthTokenProvider interface to \" +

SparkQA · 2021-08-08T06:59:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46705/

SparkQA · 2021-08-08T07:56:39Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46705/

SparkQA · 2021-08-08T13:21:56Z

Test build #142197 has finished for PR 33675 at commit a20229b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
.doc(\"A class that implements OAuthTokenProvider interface to \" +

SparkQA · 2021-08-08T13:48:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46709/

SparkQA · 2021-08-08T14:41:56Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46709/

haodemon · 2021-08-08T15:00:50Z

@dongjoon-hyun, answering inline.

Apache Spark mainly runs the unit tests on your GitHub Action. Could you enable the GitHub Action build_and_test job in your Spark fork?

We also have a Jenkins sever. I'll trigger it though.

I'm not sure why they didn't start before, but seems like now they're passing.

Apache Spark 3.3.0 will use 5.6.0, could you update the link to 5.6.0 link in the PR description?

https://javadoc.io/doc/io.fabric8/kubernetes-client/4.12.0/io/fabric8/kubernetes/client/OAuthTokenProvider.html

Updated, thank you.

dongjoon-hyun · 2021-08-11T00:47:10Z

...ubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala

-      .withOption(oauthTokenValue) {
+      .withOption(oauthTokenProviderInstance) {
+        (provider, configBuilder) => configBuilder.withOauthTokenProvider(provider)
+      }.withOption(oauthTokenValue) {


To be clear, if we have two configurations at the same time mistakingly, we invoke OAuthTokenProvider and override it with oauthTokenValue?

Could you check what happens when we call configBuilder.withOauthTokenProvider(provider) and configBuilder.withOauthToken(token) together?

The token that was specified in configBuilder.withOauthToken(token) will stay unused, and the class that implements OAuthTokenProvider will be used instead. I confirmed this via tests on my cluster by simultaneously:

1 . Adding a debugging log into my class and providing it via

--conf spark.kubernetes.client.oauth.token.provider.class <myClass>

Specifying tokens via

--conf spark.kubernetes.authenticate.submission.oauthToken=<token> --conf spark.kubernetes.authenticate.driver.oauthToken=<token> --conf spark.kubernetes.authenticate.oauthToken=<token>

My log:

[2021-08-11 12:51:58,012] DEBUG (OAuthGoogleTokenProvider.java:62) - Refreshing kubernetes oauth token via [/usr/lib/google-cloud-sdk/bin/gcloud, config, config-helper, --format=json] [2021-08-11 12:51:58,498] DEBUG (OAuthGoogleTokenProvider.java:73) - New token expiry time is 3511s

From the log it could be concluded that OAuthTokenProvider has a higher precedence over the token that was specified in configBuilder.withOauthToken(token). It could also be confirmed by looking at the fabric's code in
https://github.com/fabric8io/kubernetes-client/blob/74cc63df9b6333d083ee24a6ff3455eaad0a6da8/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/RequestConfig.java#L136

Thank you for checking. In that case, the current situation may cause a confusion to the Spark users. From Spark perspective, we have two options.

Option A: We can revise the description of this new configuration about that precedence.

Option B: We can ignore spark.kubernetes.client.oauth.token.provider.class when oauthToken exists from Spark side.

Which one do you prefer?

the current situation may cause a confusion to the Spark users.

Indeed. Thanks for making me realize this. The patch is missing a way for a Spark user to specify how they would like the Spark to authenticate in Kubernetes when running on a client or cluster mode. There is several options present for oauthToken:

spark.kubernetes.authenticate.submission.oauthToken spark.kubernetes.authenticate.driver.oauthToken spark.kubernetes.authenticate.oauthToken

And I think we need to have the same for token provider, like spark.kubernetes.authenticate.*.oauthTokenProvider.

This change would:

Make the new options consistent with the rest of kubernetes options.

Make existing options mutually exclusive. For every mode only one of the oauthToken, oauthTokenFile, oauthTokenProvider would be allowed.

If we try this, we won't have to add anything about precedence into the docs and there would be no need to ignore anything in the code.

@dongjoon-hyun, sorry for a lot of text.

I think either precedence or doing an assertion that only one of these is set is fine, but let's just pick one and do it so we can get this in.

I prepared the changes, so now we have:

• OAuthTokenProvider options are now consistent with the other two options – OAuthToken and OAuthTokenFile.
• Added an assertion that only one of the options is set (either OAuthToken, OAuthTokenFile or OAuthTokenProvider
• Added docs

I have tested this in client mode on Kubernetes.

dongjoon-hyun · 2021-08-11T00:52:46Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

  val CLIENT_CERT_FILE_CONF_SUFFIX = "clientCertFile"
  val CA_CERT_FILE_CONF_SUFFIX = "caCertFile"

+


nit. Shall we remove the redundant emtpy line?

dongjoon-hyun · 2021-08-11T03:20:26Z

BTW, I embedded your example into the PR description to provide it as a commit message, @haodemon . Please let me know if that is not what you want.

SparkQA · 2021-08-11T12:52:39Z

Test build #142330 has finished for PR 33675 at commit 5830123.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-11T13:07:19Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46838/

SparkQA · 2021-08-11T13:55:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46841/

SparkQA · 2021-08-11T14:51:30Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46841/

holdenk · 2021-09-13T19:05:25Z

Jenkins retest this please.
@haodemon have you run the K8s integration tests locally on your env?

SparkQA · 2021-09-13T19:26:14Z

Test build #143220 has finished for PR 33675 at commit 5830123.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-13T19:55:14Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47722/

SparkQA · 2021-09-13T20:02:38Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47722/

SparkQA · 2021-09-15T01:25:49Z

Test build #143289 has finished for PR 33675 at commit 56fc132.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-15T01:56:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47792/

SparkQA · 2021-09-15T04:34:13Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47796/

SparkQA · 2021-09-15T04:42:56Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47796/

SparkQA · 2021-10-07T12:50:51Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48455/

SparkQA · 2021-10-07T13:39:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48455/

SparkQA · 2021-10-07T13:49:40Z

Test build #143997 has started for PR 33675 at commit 58ce2f4.

SparkQA · 2021-10-28T01:29:07Z

Test build #144684 has finished for PR 33675 at commit b69ddc2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaModuleOptions
class HistogramPlotBase(NumericPlotBase):
class KdePlotBase(NumericPlotBase):
class IndexNameTypeHolder(object):
new_class = type(NameTypeHolder.short_name, (NameTypeHolder,),
new_class = param.type if isinstance(param, np.dtype) else param
class Database(NamedTuple):
class Table(NamedTuple):
class Column(NamedTuple):
class Function(NamedTuple):
class SparkUpgradeException(CapturedException):
protected class YarnSchedulerEndpoint(override val rpcEnv: RpcEnv)
public class ExpressionImplUtils
public final class TableIndex
public abstract class Filter implements Expression, Serializable
public class NumericHistogram
* The Coord class defines a histogram bin, which is just an (x,y) pair.
public static class Coord implements Comparable
public class ColumnarBatch implements AutoCloseable
public final class ColumnarBatchRow extends InternalRow
class IndexAlreadyExistsException(message: String, cause: Option[Throwable] = None)
class NoSuchIndexException(message: String, cause: Option[Throwable] = None)
trait FunctionExpressionBuilder
case class HistogramNumeric(
trait ExtractValue extends Expression with NullIntolerant
case class Sec(child: Expression)
case class Csc(child: Expression)
case class AesEncrypt(input: Expression, key: Expression, child: Expression)
case class AesDecrypt(input: Expression, key: Expression, child: Expression)
trait OperationHelper extends AliasHelper with PredicateHelper
case class AsOfJoin(
case class SetCatalogAndNamespace(child: LogicalPlan) extends UnaryCommand
case class CreateFunction(
case class CreateView(
case class CreateIndex(
case class SkewJoinChildWrapper(plan: SparkPlan) extends LeafExecNode
case class SetCatalogCommand(catalogName: String) extends LeafRunnableCommand
case class SetNamespaceCommand(namespace: Seq[String]) extends LeafRunnableCommand
case class ShowCatalogsCommand(pattern: Option[String]) extends LeafRunnableCommand
case class ShowCurrentNamespaceCommand() extends LeafRunnableCommand
case class WriterBucketSpec(
case class CreateIndexExec(
class SparkUDFExpressionBuilder extends FunctionExpressionBuilder

SparkQA · 2021-10-28T02:03:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49153/

SparkQA · 2021-10-28T02:48:32Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49153/

SparkQA · 2021-12-14T15:01:10Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50658/

SparkQA · 2021-12-14T15:40:45Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50658/

github-actions · 2022-03-25T00:16:03Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

tiagovrtr · 2022-04-28T21:35:01Z

@haodemon Can we reopen this and I'll help with anything needed to get the tests to pass?

dongjoon-hyun · 2022-04-28T21:51:48Z

This is reopened according to @tiagovrtr 's request.

haodemon · 2022-04-28T22:09:06Z

@haodemon Can we reopen this and I'll help with anything needed to get the tests to pass?

Thank you, @tiagovrtr. I'll take a look tomorrow and will try to update the PR. In any case, I'll post an update here.

AmplabJenkins · 2022-04-29T23:35:51Z

Can one of the admins verify this patch?

tiagovrtr · 2022-05-15T18:55:40Z

this patch seems only to bring the latest changes from master, anything else to do here?

tiagovrtr · 2022-06-04T00:43:29Z

Looping in @renato-farias as I'm leaving the @bbc and he'll very much welcome this fix in a future release 🙂

github-actions · 2022-09-30T00:36:27Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

dongjoon-hyun · 2023-08-10T15:30:42Z

Hi, All.
I'm going to take over this PR for Apache Spark 4.0.0 while keeping the authorship of @haodemon .

dongjoon-hyun · 2023-08-10T15:31:50Z

Here is a rebased PR.

[SPARK-27997][K8S] Support user-provided OAuth Token Providers #42432

[SPARK-27997][K8S] Add support for OAuth Token refresh

7b0f6c3

github-actions bot added the KUBERNETES label Aug 7, 2021

dongjoon-hyun reviewed Aug 8, 2021

View reviewed changes

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Aug 8, 2021

View reviewed changes

Changed doc and version=3.3.0 for KUBERNETES_CLIENT_OAUTH_TOKEN_PROVIDER

a20229b

dongjoon-hyun reviewed Aug 11, 2021

View reviewed changes

haodemon added 2 commits August 11, 2021 15:21

Merge branch 'apache:master' into SPARK-27997

5bca16f

Changed doc for token provider option

5830123

Merge branch 'apache:master' into SPARK-27997

56fc132

Merge branch 'apache:master' into SPARK-27997

b69ddc2

github-actions bot added the Stale label Mar 25, 2022

github-actions bot closed this Mar 26, 2022

dongjoon-hyun removed the Stale label Apr 28, 2022

dongjoon-hyun reopened this Apr 28, 2022

haodemon added 3 commits April 29, 2022 17:38

Merge branch 'master' into SPARK-27997

f28744b

Merge branch 'SPARK-27997' of github.com:haodemon/spark into SPARK-27997

8d44ca8

Merge branch 'master' into SPARK-27997

2035f5d

github-actions bot added the Stale label Sep 30, 2022

github-actions bot closed this Oct 1, 2022

		val CLIENT_CERT_FILE_CONF_SUFFIX = "clientCertFile"
		val CA_CERT_FILE_CONF_SUFFIX = "caCertFile"

Conversation

haodemon commented Aug 7, 2021 • edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 8, 2021

Uh oh!

SparkQA commented Aug 8, 2021

Uh oh!

SparkQA commented Aug 8, 2021

Uh oh!

SparkQA commented Aug 8, 2021

Uh oh!

SparkQA commented Aug 8, 2021

Uh oh!

SparkQA commented Aug 8, 2021

Uh oh!

SparkQA commented Aug 8, 2021

Uh oh!

haodemon commented Aug 8, 2021

Uh oh!

dongjoon-hyun Aug 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haodemon Aug 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 11, 2021

Uh oh!

SparkQA commented Aug 11, 2021

Uh oh!

SparkQA commented Aug 11, 2021

Uh oh!

SparkQA commented Aug 11, 2021

Uh oh!

SparkQA commented Aug 11, 2021

Uh oh!

holdenk commented Sep 13, 2021

Uh oh!

SparkQA commented Sep 13, 2021

Uh oh!

SparkQA commented Sep 13, 2021

Uh oh!

SparkQA commented Sep 13, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

haodemon commented Aug 7, 2021 •

edited by dongjoon-hyun

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun Aug 11, 2021 •

edited

Loading

haodemon Aug 11, 2021 •

edited

Loading

tiagovrtr commented Jun 4, 2022 •

edited

Loading