[SPARK-25825][K8S][WIP] Enable token renewal for both --keytab and tokenSecret

[ifilonenko]

What changes were proposed in this pull request?

Enabled token renewal when specifying --keytab or (spark.kubernetes.kerberos.tokenSecret.renewal + spark.kubernetes.kerberos.tokenSecret.name) for Kerberos on Kubernetes

How was this patch tested?

Unit and Integration tests

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4683/

[SparkQA]

Test build #98337 has finished for PR 22915 at commit 92b61e0.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4683/

[vanzin]

Hey, let's make a deal.

You point me at an existing service that does this, that anyone can download and use. It doesn't even need to be open source, it just needs to have a well defined interface that we can actually write code for or document. And then you write according to that service's interface.

Then I'll stop saying that this service does not exist.

It does not matter how many times you talk about this service if that service is not defined. You need to provide something that people can use, not give them hand-wavy explanations and leave it to them to figure out how to implement this service or where to find it.

So please, until this service actually exists in some form, please refrain from mentioning it in Spark's documentation and building things based on it.

[ifilonenko]

The problem is that such a service can run in a variety of ways, so I thought it was a matter of defining what the resulting secret would look like. We wrote an example external service in our deprecated-fork to give an example of how such a service would function like: apache-spark-on-k8s#453. In essence, using a service keytab it should aquire delegation tokens bounded to the job-users principle. and place the contents in the secret as a new data-item. For us internally, and other companies running their own unique external renewal services. we might have varying implementations, but I just want to have a well-defined spec of the resulting secret, so I am just experimenting with a WIP spec below.

However, it clearly seems necessary to define how such a service should function as well. Would that be sufficient? Sadly, that would still be a bit hand-wavy.

[vanzin]

I don't want us to support internal proprietary services. If a company has an internal proprietary service for this, they can also spend their own resources to customize Spark so it does what they need.

We (the Spark community) cannot maintain compatibility and make sure code keeps working with things we don't know exist. Which is why I've always complained when you mention this service.

it clearly seems necessary to define how such a service should function as well. that would still be a bit hand-wavy.

It can't be hand wavy. Otherwise if someone tells us their own custom implementation of that service doesn't work, whose fault it is?

Again, we just can't write code against something that does not exist.

[vanzin]

One idea here is to create a command line tool in Spark to implement this functionality. Users could use that tool to do what it is you're trying to achieve here. It can be k8s-specific - lots of features start that way.

Then there is a clear protocol between the running app and this external entity, and people can even create a simple service around this tool if they'd like. And you avoid the whole "who's actually using this stuff" question.

[skonto]

The API should be defined first but It would be good to have a reference implementation of this service targeting K8s, that could be run out of the box. End-user wants the whole thing most of the time. This should be part of Spark since this way we make sure we keep things updated all the time.
I like the server implementation in the fork and btw this functionality should be backported to the spark operator project as well. Actually an operator on K8s is for managing these kind of things and I am wondering if it should have been part of the Spark project, like the mesos dispatcher is. On the other hand, the unfortunate thing is that ideally resource managers should be separate projects but the separation never happened. I am not sure how a cli tool would help here, we need a pod service right?

[vanzin]

If there is an open source implementation that you want to add to Spark, by all means, do it!

The CLI tool was a suggestion to work around the fact that a bunch of code and documentation references this thing that doesn't exist. The CLI tool would make it exist.

[vanzin]

I'm not seeing progress here, and old-style token renewal is already implemented in the k8s backend. Anything on top would be a different change. So closing.