-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Support AAD Passthrough for ADLS mounts #63
Comments
@stikkireddy - this is the suggestion I mentioned to you. Does this seem like a reasonable approach? Am thinking that the properties on |
@stuartleeks I like the approach, at this point in time I do not have visibility into how many people use this resource to provision mounts so I think it would be best to make these "breaking" changes asap as this is still a minor version of the provider and breaking changes are expected to happen. But the service_principal block seems to be a good approach to make the resource a bit cleaner |
Cool. Thanks for your response - I'm hoping to work on this tomorrow :-) |
Hey @stikkireddy I've coded this up and am trying to test it now. I've set this on the cluster I'm testing the AAD Passthrough mount with: spark_conf = {
"spark.databricks.passthrough.enabled": "true"
} When I run the test I get an error: I noticed that if I create a cluster via the UI I also get a Is there something that I'm missing? Any docs of setting up a cluster for passthrough with the API? Thanks! |
Hey @stuartleeks was on PTO till today, had sometime to look at this. Pass through is generally used in a shared "high concurrency" cluster and it seems that the following spark options are what creates these "high concurrency" clusters. You may need these enabled to enable passthrough or at least test passthrough with adls.
|
Standard clusters require "single_user_name" field populated in the cluster model in the post request for the passthrough to work appropriately |
@stikkireddy - I've added #71 to track adding |
@stuartleeks not sure what the status of this is? its closable correct? i believe we merged single_user_name. |
We hit issues automating mounts for AAD Pass through when running under a service principal account. |
@stuartleeks is this one still relevant? Will close if it's not. |
Hi, I stumbled upon this issue. We'd like to use the feature. Is there any workaround on how to make passthrough auth work? Can the Feature Request be re-opened? |
Is your feature request related to a problem? Please describe.
Currently ADLS mounts allow mounts to be created using service princpal details, but for some scenarios we want to be able to provision mounts using AAD Passthrough: https://docs.microsoft.com/en-us/azure/databricks/security/credential-passthrough/adls-passthrough#--mount-azure-data-lake-storage-to-dbfs-using-credential-passthrough
Current ADLS Gen 2 mount resource:
Describe the solution you'd like
Would like to be able to specify to use AAD Passthrough rather than passing
client_id
etcThe proposed change to the resource is shown below
Service principal:
AAD Passthrough:
The text was updated successfully, but these errors were encountered: