[SPARK-27022][DSTREAMS] Add kafka delegation token support.#23929
Closed
gaborgsomogyi wants to merge 3 commits intoapache:masterfrom
Closed
[SPARK-27022][DSTREAMS] Add kafka delegation token support.#23929gaborgsomogyi wants to merge 3 commits intoapache:masterfrom
gaborgsomogyi wants to merge 3 commits intoapache:masterfrom
Conversation
|
Test build #102916 has finished for PR 23929 at commit
|
Contributor
Author
|
cc @vanzin |
vanzin
reviewed
Mar 4, 2019
Contributor
vanzin
left a comment
vanzin
left a comment
There was a problem hiding this comment.
Is the any documentation to update? Better do that here if needed.
...kafka-0-10-token-provider/src/test/scala/org/apache/spark/kafka010/KafkaTokenUtilSuite.scala
Show resolved
Hide resolved
Contributor
Author
Since its a small change and don't expect heavy architectural changes absolutely make sense, adding... |
gaborgsomogyi
commented
Mar 5, 2019
|
Test build #103041 has finished for PR 23929 at commit
|
vanzin
reviewed
Mar 5, 2019
|
Test build #103139 has finished for PR 23929 at commit
|
Contributor
|
Merging to master. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
It adds Kafka delegation token support for DStreams. Please be aware as Kafka native sink is not available for DStreams this PR contains delegation token usage only on consumer side.
What this PR contains:
KafkaConfigUpdatermoved tokafka-0-10-token-providerKafkaSecurityHelperfunctionality moved intoKafkaTokenUtilHow was this patch tested?
Existing unit tests + on cluster.
Long running Kafka to file tests on 4 node cluster with randomly thrown artificial exceptions.
Test scenario:
Kafka broker settings:
After each 7.5 minutes new delegation token obtained from Kafka broker (10 min * 0.75).
When token expired after 10 minutes (Spark obtains new one and doesn't renew the old), the brokers expiring thread comes after each 5 minutes (invalidates expired tokens) and artificial exception has been thrown inside the Spark application (such case Spark closes connection), then the latest delegation token picked up correctly.
cd docs/
SKIP_API=1 jekyll build
Manual webpage check.