addAadSupportInSpark #32393
Merged
addAadSupportInSpark #32393
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
added
the
xinlian12
commented
Dec 1, 2022
...smos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/CosmosCatalogBase.scala
Show resolved
Hide resolved
|
API change check API changes are not detected in this pull request. |
xinlian12
commented
Dec 1, 2022
...smos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/CosmosCatalogBase.scala
Outdated
Show resolved
Hide resolved
...e-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/cosmosclient/ICosmosClient.scala
Outdated
Show resolved
Hide resolved
...e-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/cosmosclient/ICosmosClient.scala
Outdated
Show resolved
Hide resolved
...e-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/cosmosclient/ICosmosClient.scala
Outdated
Show resolved
Hide resolved
4 tasks
6 tasks
xinlian12
force-pushed
the
aadSupportInSpark
branch
from
5bb9fd1
to
3855036
Compare
xinlian12
force-pushed
the
aadSupportInSpark
branch
3 times, most recently
from
86cf08e
to
205be9c
Compare
xinlian12
commented
Jan 24, 2023
...smos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/CosmosClientCache.scala
Outdated
Show resolved
Hide resolved
xinlian12
commented
Jan 24, 2023
sdk/cosmos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/CosmosConfig.scala
Show resolved
Hide resolved
xinlian12
force-pushed
the
aadSupportInSpark
branch
from
c6da9af
to
482440c
Compare
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
LGTM - thanks! |
FabianMeiswinkel
approved these changes
Jan 31, 2023
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
added 2 commits
January 31, 2023 16:54
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
xinlian12
requested review from
hallipr,
weshaggard,
benbp and
JimSuplizio
as code owners
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run java - cosmos - spark |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Hi @xinlian12 , I've created a PR to bump the version. PR's merged. It should be fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add aad support in spark.
Why the need
You cannot use any Azure Cosmos DB data plane SDK to authenticate management operations with an Azure AD identity.
Azure Cosmos DB RBAC
Design
Cosmos Config
New config entry has been introduced which allows customer to provide service principle related config
spark.cosmos.auth.typespark.cosmos.account.subscriptionIdspark.cosmos.account.tenantIdspark.cosmos.account.resourceGroupNamespark.cosmos.account.azureEnvironmentspark.cosmos.auth.aad.clientIdspark.cosmos.auth.aad.clientSecretSpark catalog
Catalog is the API interfaces which allows customer to interact with the metadata store.
CosmosCatalogBaseandCosmosCatalogcontain all the related implementations, which includes most of the related management operations (like creating database, creating container etc). The idea here is to use ManagementSDK for the underlying operations if customer is using ServicePrinciple based AAD authentication, use CosmosDB Java SDK V4 if customer is using masterKey authentication.Throughput control
Query database/container throughput throughput AAD authentication is also not supported through Cosmos DB data plane SDK currently, so in order for throughput control to work correctly, it needs a way to query throughput successfully.
A new internal method is being introduced which accept a throughputQueryMono, which will be used in
ThroughputContainerController.Test
Currently spark CI pipeline only targets CosmosDB Emulator, however using CosmosDB Emulator for testing is challenging as:
So the following tests is used as gates:
basicScenarioAadnotebook which can be triggered through /azp run java - cosmos - sparkCosmosClientCacheITest,CosmosCatalogITest