Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ISSUE] Databricks provider uses invalid access token #66

Closed
sdebruyn opened this issue May 26, 2020 · 7 comments · Fixed by #110
Closed

[ISSUE] Databricks provider uses invalid access token #66

sdebruyn opened this issue May 26, 2020 · 7 comments · Fixed by #110
Assignees
@stikkireddy
Labels
azure Occurring on Azure cloud bug Something isn't working
Milestone
v0.2.0

Comments

@sdebruyn
Copy link
Contributor

sdebruyn commented May 26, 2020 •
edited

Terraform Version

➜ terraform -v
Terraform v0.12.25
+ provider.azuread v0.9.0
+ provider.azurerm v2.11.0
+ provider.databricks (unversioned)
+ provider.http v1.2.0
+ provider.null v2.1.2
+ provider.random v2.2.1
+ provider.time v0.5.0

Current master branch

Affected Resource(s)

  • all resources

Terraform Configuration Files

https://github.com/datarootsio/terraform-module-azure-datalake/blob/a3c400b5bf40c2d64159bd703428c062f0174a23/databricks.tf

Debug Output

https://github.com/datarootsio/terraform-module-azure-datalake/runs/709963745?check_suite_focus=true

2020-05-26T16:40:31.5532525Z     command.go:172: Error: Response from server (403) <html>
2020-05-26T16:40:31.5532687Z     command.go:172: <head>
2020-05-26T16:40:31.5533058Z     command.go:172: <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
2020-05-26T16:40:31.5533244Z     command.go:172: <title>Error 403 Invalid access token.</title>
2020-05-26T16:40:31.5533404Z     command.go:172: </head>
2020-05-26T16:40:31.5533565Z     command.go:172: <body><h2>HTTP ERROR 403</h2>
2020-05-26T16:40:31.5533748Z     command.go:172: <p>Problem accessing /api/2.0/secrets/put. Reason:
2020-05-26T16:40:31.5533922Z     command.go:172: <pre>    Invalid access token.</pre></p>
2020-05-26T16:40:31.5534079Z     command.go:172: </body>
2020-05-26T16:40:31.5534228Z     command.go:172: </html>
2020-05-26T16:40:31.5534573Z     command.go:172: : invalid character '<' looking for beginning of value
2020-05-26T16:40:31.5534739Z     command.go:172: 
2020-05-26T16:40:31.5534912Z     command.go:172:   on databricks.tf line 73, in resource "databricks_secret" "cmdb_master":
2020-05-26T16:40:31.5535168Z     command.go:172:   73: resource "databricks_secret" "cmdb_master" {

Expected Behavior

Create a databricks_secret

Actual Behavior

See error output above

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. Use the module https://github.com/datarootsio/terraform-module-azure-datalake as described in the readme

Comments

At first sight I thought it was an issue with databricks_token but it does not seem to be directly related to that resource. It seems to be an issue with the token that this provider is using underneath to create the resources as this seems to happen with a databricks_secret that isn't using any access tokens explictly.
@sdebruyn sdebruyn mentioned this issue May 26, 2020
Merged
@sdebruyn
Copy link
Contributor Author

Note that this does not happen every time... it seems another issue where the API says it's okay, but you can't use it right away

@stikkireddy
Copy link
Contributor

@sdebruyn hmm a bit tricky to recreate this, we can use the retry functionality implemented a little while ago to potentially retry http calls with this but, I am not sure if that is ideal. I want to see if others have any thoughts on why this is happening or if it is reproducible. If you find a way to reproduce it 1/n times that would be great! Till then I would like to hold off on making changes to the token configure. We will switch to AAD tokens when they support secrets 👍 .

@stikkireddy stikkireddy mentioned this issue Jun 12, 2020
Closed
@sdebruyn
Copy link
Contributor Author

It doesn't happen on every run, but it did here: https://github.com/datarootsio/terraform-module-azure-datalake/commit/cc36e3c69acef26821ddbfb980b867a6d6241b82/checks

@mfolley
Copy link

mfolley commented Jun 16, 2020 •
edited

We consistently see this issue when our deployments take longer than 10 minutes, which is often the case when cluster creation is part of the operation.

I think I have tracked this down to the fact that the workspace access token created is hard-coded to live for 600 seconds: https://github.com/databrickslabs/terraform-provider-databricks/blob/master/databricks/azure_auth.go#L136 (not 60 mins as the comment suggests)

I believe this may also explain the behavior seen in #94 (though the cause of the timeout there looks like a separate issue).

@sdebruyn
Copy link
Contributor Author

That is a very good point. My deployments can take up to an hour because they involve connecting several components and some resources like Cosmos DB takes about 10 minutes for that resource alone.

Maybe it would be useful to have an expiration of 1 hour with automatic renewal before it expires?

@stikkireddy stikkireddy mentioned this issue Jun 16, 2020
Merged
@stikkireddy
Copy link
Contributor

proposed pull request is here at #110 please let me know if that seems appropriate fix for this.

@nfx
Copy link
Contributor

nfx commented Jun 22, 2020

probably we'd need to re-issue PAT's on 403 responses, when they expire

@nfx nfx added the bug Something isn't working label Jun 22, 2020
@nfx nfx added this to the v0.2.0 milestone Jun 22, 2020
@nfx nfx closed this as completed in #110 Jun 25, 2020
@nfx nfx added the azure Occurring on Azure cloud label Feb 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stikkireddy stikkireddy

Labels
azure Occurring on Azure cloud bug Something isn't working
Projects
None yet
Milestone
v0.2.0
Development

Successfully merging a pull request may close this issue.

Fix Azure SP Personal Access Token Duration databricks/terraform-provider-databricks
4 participants
@nfx @sdebruyn @mfolley @stikkireddy